Pseudo-SIERA Using BsR

There has been a lot of chatter in the sabermetric community about SIERA, the new DIPS-style ERA estimator using batted ball data developed by Eric Seidman and Matt Swartz for Baseball Prospectus. Before even commenting on the metric itself, I think it is worthwhile to applaud BP for their recent hires (including Swartz and Seidman, but also Tommy Bennett and Colin Wyers) that have reinvigorated the sabermetric aspect of their operation--and for their openness in sharing the development of SIERA in a five-part series.

SIERA is innovative in that it does not consider line drives (except to the extent that they also represent a plate appearance), citing the low correlation in year-to-year line drive frequencies; it considers groundball rate with a denominator of plate appearances rather than balls in play; it treats flyballs and pop-ups equally and as offsets to grounders in a (GB - FB - PU)/PA term; and it is based on a regression equation with quadratic and interactive terms.

It is the latter property of the metric which makes someone who approaches sabermetrics from my particular viewpoint and biases blanch a bit. My personal reaction to any sort of "ugly" regression equation is to wonder if there is a way to accomplish the same objective with similar accuracy while using a more intuitive (to me) model. Naturally, whenever run estimation is involved, that leads me to Base Runs, which is the most intuitive simple model of team run scoring (IMO--of course, this blog is always just my opinion and nothing more, but I'm being extra careful today).

So I quickly threw together a Base Run estimator, using only batted ball types (GB, FB, LD, and PU), walks, and strikeouts. I further divided batted balls into just two categories--groundballs (G) and non-grounders (nG = FB + PU + LD). This is not exactly the same as what SIERA does by using (GB - FB - PU)/PA, but it does embrace the not-so-fine breakdown of batted ball types that SIERA introduced.

I then estimated singles, doubles, triples, homers, and outs from G, nG, and K. To do this, I used the data published by Colin Wyers here on event rates by batted ball types , and figured a weighted average for non-grounders by using the GB, FB, PU, and LD data published by BP. This posed a bit of a problem, as Colin used Retrosheet definitions to derive his figures while BP uses a different data source; such is the nature of working with batted ball data without a DIY-approach.

From there, I ensured that estimated singles equaled actual singles for the 2009 major leagues (and doubles, etc.) and simply plugged everything into BsR:

eS = .2167(G) + .2092(nG)
eD = .0184(G) + .1022(nG)
eT = .001(G) + .0119(nG)
eHR = .0701(nG)
A = eS + eD + eT + W
B = (2*(eS + 2*eD+ 3*eT + 4*eHR) - (eS + eD + eT + eHR) - 4*eHR + .05*W)*.78
C = .7636(G) + .607(nG) + K
Pseudo-SIERA = (A*B/(B+C) + eHR)*.904/(C/3)

NOTE: As commentator Bryan points out, this formula is riddled with errors. I'll leave it here as to not hide my stupidity, but please see comment #4 for a corrected set of equations.

It is imperative to point out that what I did was clearly cheating from the standpoint of testing a formula--I calibrated it against the very dataset against which I am going to test it, rendering the test essentially useless. I did this simply for ease--my point in this entire exercise is not to supplant SIERA but just to raise the question of whether a different model can incorporate Seidman and Swartz's findings.

There is also a great deal of phony precision (i.e. ten-thousandths place decimals) on display, and a great deal of simplification that could be done to the terms (everything could be written in just four variables--G, nG, W, and K, but unless one is actually intending to use the formula on a regular basis doing so is a waste of time). Also, a better BsR B factor could have been used--I stuck with one I've used forever despite being less than optimal.

I then applied Pseudo-SIERA to all pitchers with >= 200 PA in 2009. The RMSE in estimating ERA for the real SIERA was 1.00; for Pseudo-SIERA, 1.06. Of course, as I already made clear, Pseudo-SIERA had the advantage of being calibrated specifically on this dataset.

Of course, the more telling test of a SIERA-type metric is how it does at predicting future ERA, something that I have obviously not tested here. There's really no need, at least with this implementation of the pseudo formula--I have no expectation that it would outperform SIERA. My goal here was just to incorporate a couple of ideas from Seidman and Swartz's work into a BsR model, and demonstrate that such a model has potential to be used in conjunction with those ideas. Nothing more.

On a final, unrelated note, I posted the first guest scoresheet contribution to Weekly Scoresheet yesterday. I have two more ready to go, and hopefully there will be more to come. I realized shortly after sending out my initial request for scoresheets that I should have waited until the season was underway and people would be more likely to have scoring on the brain (and scoresheets sitting around where they could be easily procured). Oh well.