< February Film Roundup

Direct Observation of the MST3K-IMDB Effect: The MST3K-IMDB effect, first hypothesized in 2011, is the effect on a movie's IMDB rating caused solely by its having appeared on Mystery Science Theater 3000. A bad movie isn't made any worse by being on MST3K, but MST3K shows movies in the context of a fiction where they're an instrument of torture. You can model an episode of MST3K as a negative advertisement, designed to make the viewer feel negatively about a movie they'd never heard of before. The MST3K-IMDB effect measures the 'success' of the negative ad.

Previously, the MST3K-IMDB effect could only be observed through its gravitational tug on one movie out of a director's oeuvre. But in 2017, Mystery Science Theater: The Return took movies that for years had sat in the dustbin of history and turned them into well-known "MST3K movies". Thanks to my trusty 2015 IMDB data dump, I'm in a position to compare the ratings of all these movies before and after they were on MST3K, providing the first direct images of the MST3K-IMDB effect.

First, here's my control group: a comparison of the 2015 and 2018 ratings for some random movies that weren't on MST3K:TR. As you can see, IMDB ratings for old movies are normally very stable, not changing by more than a couple percent over three years.

Title 2015 rating 2015 votes 2018 rating 2018 votes Percentage change in rating
Night of the Lepus 3.9 2782 4 3484 2%
Embryo 4.9 824 5 1052 2%
The Ninth Configuration 7.3 3486 7.2 4893 -2%
Gorgo 5.5 1846 5.6 2479 1%
O Fantasma (2000) 5.9 1859 5.7 2449 -4%
They’re A Weird Mob 6.6 550 6.6 686 0%

My methodology: the first two movies in that list are the sort of thing you see on MST3K. The Ninth Configuration is a more highbrow obscure film. Gorgo was on MST3K a long time ago, so presumably the MST3K-IMDB effect is already priced in. The last two are movies of MST3K-level obscurity chosen at random. I would love to do this for every single movie in the 2015 data dump and just take an average, but I can't because IMDB doesn't offer full data dumps anymore.

Update: I took another look and this analysis is in fact possible once I write some code to compare the old and new IMDB dump formats. For non-porno feature films, the mean rating change between 2015/01/31 and 2017/12/24 is 0.13 stars. The median is 0.10 stars. Percentage-wise, it's 3% and 1%. So on average, IMDB ratings go very slightly up over time—a phenomenon I also noticed with Board Game Geek ratings.

Now, the moment I've all been waiting for, the experimental group. This table compares the 2015 and 2018 ratings of the movies that appeared in the first season of MST3K:TR. I've also included the rating of the MST3K:TR episode itself, as well as the number of votes IMDB used to calculate each rating.

Here, the percentage change in rating is largely due to the MST3K-IMDB effect. This is the percentage of a movie's previous rating that it lost just by being mocked on MST3K.

Title 2015 rating 2015 votes 2018 rating 2018 votes Percentage change in rating MST3K:TR episode rating MST3K:TR episode votes
Reptilicus 4 1599 3.6 2962 -10% 7.8 498
Cry Wilderness 5.2 37 2 877 -62% 8.3 819
The Time Travelers 6 753 5.1 1682 -15% 7.4 256
Avalanche 4.2 594 3.7 1352 -12% 8 261
The Beast of Hollow Mountain 5 509 4.2 1182 -16% 7.4 230
Starcrash 4 2946 3.9 4699 -3% 7.7 235
The Land Time Forgot 5.7 3085 5.7 4558 0% 7.4 197
The Loves of Hercules 3.2 349 3 787 -7% 7.2 181
Yongary 4.3 592 3.9 1102 -10% 7.8 201
Wizards of the Lost Kingdom 2.6 502 2.6 1037 0% 7.8 187
Wizards of the Lost Kingdom II 1.8 297 1.8 716 0% 7.4 168
Carnival Magic 3.8 120 2.4 463 -37% 7.2 167
The Christmas that Almost Wasn't 5.5 204 3.8 541 -31% 7.1 148
At the Earth's Core 5.2 2276 5.2 3410 0% 7.2 134

The MST3K-IMDB effect is real, but it's not a constant. In the control group we saw movie ratings vary up and down, seemingly at random. With MST3K:TR movies the trend is clearly down. But for five of the fourteen movies we see no effect, or an effect so small that it could have been due to chance.

I wasn't expecting this result! I don't see a pattern in the five movies that lack an MST3K-IMDB effect; they aren't the best-rated movies or the worst, they weren't in better- or worse-rated episodes of MST3K, they aren't from a different time period than the others. Sometimes MST3K just doesn't change peoples' opinion about a movie.

But sometimes it does. Take a look at Cry Wilderness, which lost over half of its IMDB rating due to being on by far the highest-rated episode of MST3K:TR. It went from a regular bad movie to being Manos-level. Part of this is just that its pre-MST3K rating of 5.2 was taken from a very small sample of just 37 votes—the kind of super-obscure film I usually omit in IMDB analyses. But two other movies—Carnival Magic and The Christmas that Almost Wasn't—also have very large rating shifts.

Here's the closest I can come to a unified theory of what happened to those three movies. I took the number of post-2015 votes for each MST3K:TR movie and divided the post-2015 change in IMDB rating by that number of votes. This gives a guess at an incremental per-person MST3K-IMDB effect. So, every time someone saw a movie on MST3K and then rated the movie, we can say its rating went down by an average number of stars.

For most of the movies this number is zero (people gave the movie the same rating as if it hadn't been on MST3K at all) or infinitesimal (a lot of people voted and the IMDB rating went down a little bit), on the order of -1.0*10-3 stars. But three stand out. For Cry Wilderness the per-person MST3K-IMDB effect is -3.8*10-3 stars. For Carnival Magic it's -4.1*10-3. And for The Christmas that Almost Wasn't it's an amazing (relatively speaking) -5.0*10-3 stars.

So in a sense the negative ad campaign against The Christmas that Almost Wasn't was the most successful one here. That was the episode that got people the most riled up against the movie they'd just watched. But that movie's IMDB rating didn't go down as much as Cry Wilderness's, because the Cry Wilderness episode got more people to actually go vote the movie down.

But beyond those three movies, the per-person MST3K-IMDB effect is a lot smaller. What's the difference? Are those three movies unusually MST3K-compatible? Are those episodes meaner? It probably has something to do with the content of the movie, since the two Wizards of the Lost Kingdom movies have identical MST3K-IMDB effects of zero. Beyond that, I don't know.

Obviously there is no real propaganda campaign; no one set out to lower these obscure movies' ratings by making fun of them on Netflix. But I think this is a natural experiment showing what can happen to ratings and metrics even when the stakes are very low and no one has malicious intent.

Filed under:


[Main]

Unless otherwise noted, all content licensed by Leonard Richardson
under a Creative Commons License.