Wondering where the numbers come from -- Rotten Tomatoes

A while back I was taking one of my random walks through Wikipedia and I came across the movie Postal. For some forgotten reason (possibly to see what the critics had to say about Dave Foley, J.K. Simmons or Zack Ward, all interesting actors), I clicked on the link for Rotten Tomatoes.

The movie had a perfect 0% among top critics, but I noticed Peter Hartlaub of the San Francisco Chronicle had a rather kind blurb.

If this movie had been made by an unknown young director, a lot of critics would still be panning the movie for its inconsistencies -- but many others would be praising his courage.

This got me curious so I took a look at the actual review:

Movie review: 'Postal' delivers funny madness
Peter Hartlaub, Chronicle Pop Culture Critic

So what to do with "Postal," which is not only less than horrible, but actually occasionally enjoyable? The much-delayed low-budget movie may be completely beyond the bounds of mainstream taste, but it's also funny, and criticizes our government's hypocrisy and political correctness in a way that's refreshingly pointed. If this movie had been made by an unknown young director, a lot of critics would still be panning the movie for its inconsistencies - but many others would be praising his courage.

...

It's an unfocused movie, with much lower production values than more generously budgeted Boll productions such as "BloodRayne." (For years, Boll movies had been augmented by healthy tax breaks provided by the German government, which no longer offers the perks.) Much of the humor is cliched, and Boll's seething anger at his real-life tormenters often gives his script a bitterness that is more awkward than funny.

But there's still a catharsis that comes from watching the madness unfold onscreen, making the film a potential future double-bill partner with "Team America World Police." Boll's greatest asset is the underrated [Zack] Ward, a longtime character actor who is best known for playing the red-haired bully Scut Farkus in "A Christmas Story," and tends to show up these days in small roles in big projects - including "The Transformers" and "Lost." Clearly grateful to be the leading man (and apparently unfazed by the more unsavory parts of the script), he throws himself into the role. Ward is likable and wry, but still looks right as an action hero when it comes time to start blowing stuff up.

And blow stuff up he does, but this time the low-budget look that Boll embraces seems to be on purpose.

It didn't sound like Hartlaub was going to list Postal in his top ten but overall the review sounded fairly positive. I also noticed this icon at the top of the review.

This figure indicates a "Good" rating. How does that translate to "Rotten"? Apparently it's because the Chronicle scores on a scale of zero to four with two being 'good.' I assume that Rotten Tomatoes is used to dealing with either one to four or one to five scales, both of which would designate two as below average. I don't have an opinion on Postal or on the director Uwe Boll having never seen any of his movies, but this does seem a bit unfair, both to this movie and to others the Chronicle  rated "Good."

What is interesting is the fact that this is pretty clearly a glitch and it's a glitch in the easy part of review aggregation. Rotten Tomatoes also attempts to assign binary ratings to reviews without specified ranking (Pauline Kael called John Huston's The Bible "A sprawling, flawed epic, but with some breathtaking conceptions and moments of beauty." -- is that a thumbs up or a thumbs down?). Metacritic takes things to the next level with a hundred point scale.

This brings up one of my problems with data-driven journalism. Reporters and bloggers are constantly barraging us with graphs and analyses and of course, narratives looking at things like Rotten Tomatoes rankings. All to often, though, their process starts with the data as given. They spend remarkably little time asking where the data came from or whether it's worth bothering with.