Netflix Ratings and The Napoleon Dynamite Problem

Bertoni says it’s partly because of “Napoleon Dynamite,” an indie comedy from 2004 that achieved cult status and went on to become extremely popular on Netflix. It is, Bertoni and others have discovered, maddeningly hard to determine how much people will like it. When Bertoni runs his algorithms on regular hits like “Lethal Weapon” or “Miss Congeniality” and tries to predict how any given Netflix user will rate them, he’s usually within eight-tenths of a star. But with films like “Napoleon Dynamite,” he’s off by an average of 1.2 stars.

The reason, Bertoni says, is that “Napoleon Dynamite” is very weird and very polarizing. It contains a lot of arch, ironic humor, including a famously kooky dance performed by the titular teenage character to help his hapless friend win a student-council election. It’s the type of quirky entertainment that tends to be either loved or despised. The movie has been rated more than two million times in the Netflix database, and the ratings are disproportionately one or five stars.

Worse, close friends who normally share similar film aesthetics often heatedly disagree about whether “Napoleon Dynamite” is a masterpiece or an annoying bit of hipster self-indulgence. When Bertoni saw the movie himself with a group of friends, they argued for hours over it. “Half of them loved it, and half of them hated it,” he told me. “And they couldn’t really say why. It’s just a difficult movie.”

Mathematically speaking, “Napoleon Dynamite” is a very significant problem for the Netflix Prize. Amazingly, Bertoni has deduced that this single movie is causing 15 percent of his remaining error rate; or to put it another way, if Bertoni could anticipate whether you’d like “Napoleon Dynamite” as accurately as he can for other movies, this feat alone would bring him 15 percent of the way to winning the $1 million prize. And while “Napoleon Dynamite” is the worst culprit, it isn’t the only troublemaker. A small subset of other titles have caused almost as much bedevilment among the Netflix Prize competitors. When Bertoni showed me a list of his 25 most-difficult-to-predict movies, I noticed they were all similar in some way to “Napoleon Dynamite” — culturally or politically polarizing and hard to classify, including “I Heart Huckabees,” “Lost in Translation,” “Fahrenheit 9/11,” “The Life Aquatic With Steve Zissou,” “Kill Bill: Volume 1” and “Sideways.”

I wonder if the problem isn't the movies as much as the rating system itself. Instead of a single 1-5 star rating, maybe they should also include a score for the variance or list the probabilities that you will rank the movie from 1-5 stars.

When you looked at Napoleon Dynamite it could say you have a 30% chance of rating it 1 star, 10% chance for 2 star, 20% 3 star, 10% 4 star and 30% 5 star. Whereas X-Men might look like 5% 1 star, 20% 2 star, 50% 3 star, 20% 4 star and 5% 5 star. While on average you are likely to give both a 3 star rating, you could choose whether to take the gamble on a movie that you might love, or stick with seeing on that you are likely to find very average.

More interesting information on the Netflix Prize and how people are trying to solve it in the article.

I am also curious how much the recommendations for any one person differ between the various engines. Does one engine do a significantly better job of recommending movies for some people than the other engines? If that is the case, then Netflix should allow you which engine you want to use, similar to how you could choose which of staff members on the staffs' picks you wanted to go with (but think twice before going with a Gene pick over a Vincent).

via NY Times