Running of the Simulators, Part 8 (Recap)

Last in a series (Part 1Part 2Part 3Part 4Part 5Part 6, Part 7)

So what have we learned about the simulations' general accuracy? 

  • In 2010, the Yankees finished one game and the Rays two games shy of the average projection. The Red Sox finished off by six games, but no projection can — and no projection should — project a season like that.
  • In 2009, the Red Sox finished one game better than projected, but the Yankees overperformed by seven games.
  • In 2008, the projections generally punted. The Rays were 12 games better than projected, the Yankees were six games worse, and the Red Sox were three games better. Only the Blue Jays were projected more or less accurately.
  • In 2007, no projection was closer than four games off of the Red Sox' ultimate record, but they pretty much nailed the Yankees'.
  • In 2006, the Sox underperformed by three games, the Blue Jays overperformed by four, and the Yankees were seven games better than the average projection.
  • In 2005, the projections nailed it, placing the Sox and Yankees one game off, one either side, of the clubs' ultimate first-place tie.

Of the 16 AL East teams projected over the course of the last six years, I count the number of successes (two games or fewer) at seven — a 43.8 percent success rate. As for significant misses (four games or more), I also count seven. And twice, the projections were three games off, which is right on the border between being fairly close and pretty obviously not close. 

Then there's the other, arguably more important facet, which is whether they got the standings right, even if one team ended up significantly better or worse than projected. Being seven games off the Yankees' record doesn't matter nearly as much if the Yanks were expected to finish in first anyway, for example.

On that score, the projections got it right in 2005 and 2009. That's it.

They get partial credit for 2006 in correctly projecting the division winner but swapping second and third.

They missed entirely in 2007, 2008 (though correctly projecting the Sox to finish second, I don't count it as a success if you miss on the first- and third-place teams) and 2010, when all three teams finished differently than projected.

So successful, or at least correctly projecting the division winner, three times, missing the boat three times. Taking that with the roughly 50/50 success rate in terms of raw record projection, it seems we can be about 50 percent certain of the projections being correct in any given season.

As Hudson noted on one of the previous threads, it seems like we could randomly pick two numbers between 92 and 98, then assign the higher number to the Yankees and the other to the Red Sox, and we would have just as likely a chance at correctly projecting the AL East as 1,000 simulations of whatever top-flight projection systems are out there today. 

If we did that, we would get it right in 2005 and 2009, get the division winner correct in 2006, and miss it entirely in 2007, 2008 and 2010. Just like the projections.

But that doesn't make them any less fun, which of course is the entire point.

10 comments… add one

  • Three thoughts:
    1. A huge thiank you to you Paul for running these and for summing up so clearly. I’ve often often often wanted some sense of historical track record on projections whenever we are debating them in the pre-season and it’s great to now have this s clearly done here. Really appreciate all the work you did to get here.
    2. I agree very muchwith your conclusion – the Hudson point balanced by “so what, it’s fun anyway”. I think I mentioned this elsewhere but this all reminds me of an book I read in an Econ class in college which is now well-reagrded as a must-read: Malkiel’s “A Random Walk Down Wall Street”, which begins with a monkey (literally) throwing darts at a page of stock-listings, with that method for choosing investment compared to the complicated logarithms and sometimes highly-costly broke-driven systems for advising and selecting options. The monkeys faired pretty comparably. Like Hudson, I’m happy to be a monkey in both that context and this. But as you say, the process is fun.
    3. I am NOT asking that you do this, but I wonder how much mid-season acquisitions could be considered when gauging the accuracy and therefore utility of htese systems. Even more than injuries (which can sometimes be guesstimated a bit by player-ages, injury-track records, etc.) what a team gains (and gives away) as well as what their divisional opponents gain or give away during the season is impossible to predict at all. Were projections looking pretty on-target for instance in some years and then fundamentally thrown off by key moves in one direciton or another? Anyway, like I said, I am not asking that you do any more work here – and I’m too lazy to do it myself, but I found myself wondering this as I read all your great analysis here.

    IronHorse (YF) January 11, 2011, 11:04 am
  • I believe I averaged 1.3 typos per sentence in that last comment. Sorry about that. Squeezing in commentary in what is otherwise a v busy work day!

    IronHorse (YF) January 11, 2011, 11:07 am
  • That thought occurred to me, as well — Victor Matinez had a major effect on the Sox’ 2009 season, for example, while Jason Bay allowed the Sox to continue receiving Manny Ramirez-like production after it became clear Ramirez was unlikely to provide it. Bobby Abreu was a midseason acquisition, wasn’t he?

    Paul SF January 11, 2011, 11:40 am
  • projections of any kind tend to be more like monkeys throwing feces rather than darts, but they can be interesting and provoke good discussion nevertheless…i agree paul did an excellent job re-examining the past few seasons…and you bring up a great point about in-season changes to a team’s lineup whether through injuries, call-ups, or trades, and the extent to which those variables are/can be factored into the pre-season projections, or simply something to be dissected after the fact…while that would clearly require a crystal ball, not attempting to do some of it negatively impacts the credibility of any projection that assumes a team will go through an entire season with it’s opening day roster in tact….probably an impossible task, requiring too much speculation about who/what/when, so it’s too much to ask…

    dc January 11, 2011, 11:45 am
  • Abreu was indeed a mid-season acquisition and he certainly impacted the race. He was superb at the end of that ’06 season (from Aug 1 when he came over from Philly through the end of the season he batted .330 with a .926 OPS). More directly relevant to the projections for the AL East, he was a big part of knocking the Sox out of contention as early as mid-August in the 5-game series in Fenway. In that series he hit .500 with 7 walks in 27 PA. His OBP was .630 and he slugged .700. It was insane. So yes, that is an example of a mid-season acquisition that had substantial impact on the division race and on the accuracy of projections for both the Yanks and Sox.

    IronHorse (YF) January 11, 2011, 3:23 pm
  • Thank you, Paul.

    attackgerbil January 11, 2011, 7:17 pm
  • another pretty good mid-season acquisition the yankees made was david justice in 2000…i can’t prove it since i don’t have paul’s savvy in the research department, but it sure “seems” like he carried the yankees to the finish line that year…

    dc January 12, 2011, 8:00 am
  • Justice is a great example. He provided 3.2 WAR for the Yankees after being traded June 29, 2000, in a year when the Yankees won the division by just 2.5 games (and actually finished with a worse record than the wild card winners).
    Abreu provided 1.6 WAR after being traded to the Yankees on July 30. He and Justice actually provided similar offense, but their defense was obviously not the same. The main difference is that the Yanks won the division in ’06 by 10 games, and while Abreu’s performance was a key factor in the five-game sweep, the Sox had already gone from four games up on July 4 to two games back on Aug. 16, going just 19-19, including 4-7 in their last 11. After losing five straight to the Yankees, they went on to lose a game to the Angels to make it a six-game streak, then won two games, then lost another six straight. I’m pretty confident that Abreu’s presence didn’t mean all that much for the Yankees in terms of that season’s playoff race. The Sox were already faltering by the time that series started. The sweep was just the nail in the coffin.

    Paul SF January 12, 2011, 9:36 am
  • Makes sense Paul – and def the same recollection I had re: Justice dc – backed up by Paul’s numbers on him and the closeness of the race.

    IronHorse (YF) January 12, 2011, 10:32 am
  • The other hugely unpredictable spot that can swing things substantially is the bullpen. Other than the closer in some cases, the set-up and long-relief guys never seem to be the way they are drawn up in the pre-season when projections are being made. Call-ups of young fireballers (Joba) and unexpectedly good or bad performances from guys over the course of the season seem to be commonplace and play a big role in close games. I tend to think as dc does that there are just too many variables generally for projections to be of much use other than the entertainment value, but when you’re looking at the pockets of greatest variability and unpredictability, I’d imagine mid-season call-ups, injuries, and bullpen performance are among the most difficult to capture with any system.

    IronHorse (YF) January 12, 2011, 10:36 am

Leave a Comment