Loaded Dice 2012 Update

(Part of Loaded Dice.)

Introduction

In August 2012 I fetched the BoardGameGeek dataset again, so that I could make comparisons with the 2011 dataset. A summary of my findings:

We'll give a game from 2011 a worse rating now, than we did when it was new. We'll give a 2010 game a worse rating than we did in 2011. This "cult-of-the-new effect" is detectable in games published as far back as 1997.

Despite that, the ratings we give out are inflating over time. Once a game is 15 years old, inflation cancels out the fading of the cult-of-the-new effect, and a game's rating starts going back up.

Despite that, we still rate newer games higher than older games.

All that below, plus lists of games that proved to be underrated or overrated.

Newly added games

There were 7328 games present in the 2013 dataset but not in the 2012 dataset. Only about half of these (3520) were "new" games; that is, games with a publication date of 2011 or later.

893 newly added games had no publication date. The remainder spanned the past thousand years of gaming. The earliest game added to BGG in 2011 was Four Seasons Chess, first known from a book dating from 1283.

Here's a graph of games added to BGG between July 2011 and August 2011, grouped by the year of original publication:

232 games had been removed from the dataset. My spot checks indicated that most of these had been duplicates.

English?

Since so many games added in 2012 were published much earlier, I wanted to see how many of those games were non-English games that were finally being added to the America-centric BoardGameGeek. The BGG API provides information about how much language you need to know to play a game, but it doesn't say what language(s) a game is actually published in. As a very rough approximation, I considered a game to be "English" if it has a name that 1) contains only characters found in ASCII, and 2) doesn't contain any common German words like "Das" or "Eine".

By this criteria 94.8% of the games in the 2011 dataset have an "English" name, compared to only 89.4% of the games added since the 2011 dataset. Non-English games are indeed more prevalent in the new dataset, but the vast majority of games still meet the "English" criteria, and of course there are tons of non-English games, like Qui Croque Quoi?, whose names seen "English" by this rough measure.

Only 224 of the newly added games are both obviously non-English, and published before 2011.

The cult of the new

My main goal in this update was to see how peoples' opinions changed between 2011 and 2012. For a while now, I've predicted that the ratings of newer games go down over time, as the excitement wears off.

In my first attempt to detect this phenomenon, I took 2011's opinion of every year between 1980 to 2012. That is, I took the mean of all the ratings for games published in that year, so long as the game had 6 or more votes in the 2011 dataset. Then I took 2012's opinion of every year between 1980 and 2012, using the same criteria: the game had to have at least 6 votes in the 2012 dataset. (Six votes seemed like a good cutoff point because the median game from 2011 got six new votes in the 2012 dataset.)

People in 2011 had opinions about games that wouldn't be published until 2012, for a number of reasons. They'd been beta testers; they'd played the game at a con; they'd played a print-and-play version; or they hadn't played the game at all, they just had a really good feeling about it. In retrospect, these feelings were way off. The cult-of-the-new effect added three-quarters of a BGG star to the ratings of games that would be released in 2012.

Althouh much smaller, the effect is also present for games published in 2011 (0.3) and 2010 (0.17). It's visible all the way back to 2005 (0.03). The more recently a game was published, the more likely it is that its 2012 rating is worse than its 2011 rating.

As distinctive as that graph is, it's got one problem: the ratings in 2012 incorporate the ratings from 2011. The games are rated lower now than they were, but they'd be lower still if we only considered ratings from the second dataset. To isolate the "cult of the new" effect, we need to graph the average change in rating.

Here's a specific example: the game Say Anything Family Edition. In July 2011, this game had 10 votes and its rating was 7.49. In August 2012 it had 619 votes and its rating was 6.79. The difference between these ratings is due to a factor I'm going to call the cult-of-the-new effect, or CotN effect.

The simplest measurement of the CotN effect is -0.79 BGG stars, the difference between the 2011 rating and the 2012 rating. But that makes the effect look smaller than it is, because the 35 ratings that give the game 6.79 in August 2012 include the ten ratings that gave it 7.49 back in July 2011. I want to factor out those ten ratings so I don't count them twice.

Fortunately we don't need to know how each of those people voted. We can assume that current rating of 7.49 (ratingtotal) comes from ten people (votesa) giving the game a 7.49 (ratinga) back in the first dataset, and 25 people (votestotal-votesa) giving the game... some lower mystery rating, which we'll call ratingb. The equation looks like this:

(votesa * ratinga) + (votesb * ratingb) = (votestotal*ratingtotal)

We know all of these numbers except for ratingb, so we can solve for it:

(10 * 7.49) + (25 * ratingb) = 35*6.79
74.9 + (25 * ratingb) = 237.65
25 * ratingb = 162.75
ratingb = 6.51

So the current rating of 6.79 comes from ten people in the old dataset who gave the game an average of 7.49, and then 25 people in the new dataset who the game an average of 6.51. The difference between these two ratings is the CotN effect on this game between July 2011 and August 2012: -0.98 stars, much larger than -0.70 stars.

Here's a graph of the average CotN effect seen between 07/2011 and 08/2012. I took every game that had at least six ratings in the 2011 dataset, and had acquired at least six more ratings by the 2012 dataset. I calculated the cult-of-the-new effect for each of those games, and grouped the games by the year they were published. This graph shows the mean CotN effect over the period 07/2011-08/2012, for every publication year between 1980 and 2012.

On average, a rating of a 2011 game given today will be 0.66 stars lower than a rating of the same game given in July 2011, due solely to the excitement wearing off. As you go back in time, the 2012 cult-of-the-new effect diminishes. For games published before 1997, the effect becomes erratic and, strangely enough, positive. The graph breaks down completely around 1960, because not many games released before 1960 got six new ratings in 2012.

Rating inflation

Why do older games benefit from a positive CotN effect? My original idea was that good older games get reissued and judged according to modern standards. But all sorts of crappy games benefit from this effect. The CotN effect for Pogs is +1.57 stars, taking its rating from 3.33 to a still-awful 3.80. Monopoly: Pokémon didn't get reissued as far as I know, but it got 19 new ratings that took it from 5.07 to 5.25 (CotN effect: +1.47 stars).

I believe BoardGameGeek as a whole suffers from ratings inflation. In July 2011, the mean game rating on BoardGameGeek was 5.80 (median 6.00, std 1.56). In August 2012, the mean game rating is 5.82 (median 6.00, std 1.58). The mean went up but the median didn't change. Someone who liked a game in 2012 gives it a slightly higher rating than they would have in 2011.

As of last year, the mean of all the individual ratings on BGG was 6.82 (median=6.95, std=0.88). As of this year, the mean rating had gone up to 6.85 (median=6.98. std=0.87). By reusing the ratingb calculation above, I was able to estimate the mean rating handed out in 2012 as 7.00 (median=7.1, std=0.87).

I believe the cult-of-the-new effect is just a countervailing force against a general tendency to rate games more highly over time. 2012's rating of a 2011 game is lower than 2011's rating of the same game, but it's still a lot higher than the site average. A few 8 ratings are swamped by a ton of 7 ratings.

According to this theory, the cult-of-the-new phenomenon should decrease a game's rating, year after year, until it's about fifteen years old. After that, if the game is still being played, ratings inflation will take over and gradually increase its rating.

Formerly Underrated Games

I made a list of the twenty games with the largest positive CotN effect between my datasets. These are mostly newer games that were highly underrated as of July 2011. As more people played them, their ratings went up—instead of down, as you'd expect for a newer game.

As before, I only considered games that had at least six ratings as of July 2011. But this time, I focused on well-known games by looking only at games that got at least 200 new ratings between July 2011 and August 2012.

GameReleased inVotes as of 201107201107 ratingVotes from 201107-201208201208 ratingCotN effect
1989: Dawn of Freedom2012217.502828.441.00
Toc Toc Woodman2008615.943866.550.71
Dungeon Petz2011117.1817167.680.50
Arkham Horror: The Curse of the Dark Pharaoh Expansion (Revised Edition)2011317.732478.030.34
The Struggle for Catan20111126.293596.520.31
Eminent Domain2011856.9722897.270.31
Evolution: The Origin of Species2010626.402496.640.30
Arcana20094796.253486.360.24
Antiquity200410237.863587.920.24
Agricola: The Goodies Expansion20102277.262457.360.21
The Castles of Burgundy20116587.8323377.990.20
Get Bit!20072186.214256.320.16
The Resistance200918297.4433667.540.16
Eclipse2011448.2242488.360.15
Glory to Rome200531147.5120657.560.14
Eat Poop You Cat19905827.542007.570.14
Maori20095556.602606.640.13
Summoner Wars200915567.666467.700.12
Dice Town Extension2011357.462217.570.12
Black Friday20103886.552146.590.12

Note that "underrated" means underrated in 2011 relative to the current rating. "Toc Toc Woodman" had a huge swing, but that just means it went from being considered mediocre to being considered somewhat above average. The swing for "1989" may have to do with component quality: back in 2011, the game was a print-and-play game.

We don't yet know where these ratings will stabilize. Relatively old games like "Antiquity" are still seeing significant year-by-year changes in their ratings. Any one of these games might still be underrated, or might now be overrated.

Formerly overrated games

Similarly, I made a table of the twenty games with the highest negative CotN effects. They were the recipients of hype which has since subsided. Again, I only considered games with more than 200 new ratings between July 2011 and August 2012.

GameReleased inVotes as of 201107201107 ratingVotes from 201107-201208201208 ratingCotN effect
War of the Ring Collector's Edition20106318.862388.41-1.63
JAB: Realtime Boxing2011228.184786.85-1.40
Heroica: Fortaan2011147.462516.21-1.32
The Ares Project2011138.632807.38-1.30
Belfort2011108.8010937.54-1.27
Drum Roll2011158.265507.22-1.07
The Lord of the Rings: The Card Game - The Hunt for Gollum2011358.275097.35-0.99
Strike of the Eagle2011119.082398.18-0.94
Munchkin Zombies2011677.622157.00-0.81
Yomi20119267.905947.59-0.78
Quarriors!2011507.8444297.12-0.74
Dungeon Run2011307.476836.78-0.72
Thunderstone: Dragonspire20116018.219867.77-0.71
Dungeons & Dragons: Conquest of Nerath Board Game20111667.704087.21-0.69
Battleship Galaxies: The Saturn Offensive Game Set20111237.655357.10-0.67
Dungeon Raiders2011347.342416.76-0.66
Thunderstone: Doomgate Legion20105498.233927.95-0.66
Conflict of Heroes: Storms of Steel! Kursk 194320097448.202708.03-0.64
A Few Acres of Snow2011908.3524787.75-0.62
Puzzle Strike20106277.364127.11-0.62

Again, these games were overrated in 2011 relative to their current rating. "War of the Ring Collector's Edition" saw a relatively huge drop in its rating as the novelty wore off, but it's still significantly above average.

Late bloomers

Here are twenty old games (published in 2010 or earlier) that became significantly more popular between my first dataset and my second. I only looked at games that had at least one rating in the 2011 dataset, and at least fifty ratings in the 2012 dataset. These are the breakout hits of late 2011/early 2012.

GameYear published201107 ratings201108 ratingsNew rating percentageAverage rating
My Little Pony Hide & Seek200518598.82%4.87
Big Five201018498.81%6.35
Hanabi2010534598.55%7.48
Cards Against Humanity20093857493.38%7.54
BattleCON: War of Indines20101417191.81%8.19
The Enigma of Leonardo200755991.53%5.15
Timeline: Inventions20105867991.46%6.84
Caveman Curling20101616690.36%6.80
Shoot Out200255190.20%5.43
Sun of York200566190.16%6.79
Rory's Story Cubes: Actions200798989.89%6.92
Die Heisse Schlacht am Kalten Buffet19742218087.78%6.32
Toc Toc Woodman20086144786.35%6.55
Prêt-à-Porter20107447984.55%7.51
Jungle Speed: Rabbids2008106484.38%7.19
Confusion: Espionage and Deception in the Cold War19929657283.22%7.46
Perplexus200195282.69%7.52
Reverse Charades2010115881.03%6.82
Animal Upon Animal: Balancing Bridge20102010380.58%7.13
Evolution: The Origin of Species20106231180.06%6.64


This document (source) is part of Crummy, the webspace of Leonard Richardson (contact information). It was last modified on Tuesday, July 23 2013, 14:05:23 Nowhere Standard Time and last built on Wednesday, October 01 2014, 16:00:05 Nowhere Standard Time.

Crummy is © 1996-2014 Leonard Richardson. Unless otherwise noted, all text licensed under a Creative Commons License.

Document tree:

http://www.crummy.com/
writing/
Loaded Dice/
2012 update.html
Site Search: