Why Netflix Never Used the Algorithm That Won The Netflix \$1 Million Challenge - Saint

Now imagine that, Netflix wanted a new algorithm today and whoever manages to create something game changing can stand a chance to win \$1 million. Exciting isnt it?

So you gather a few of your friends, who knows Netflix inside out, and create something that is bound to change the video streaming industry forever.

By some chance, you and your friends even manage to beat the other competing teams, and end up as winners.

Only to find a few months later, after eagerly waiting for Netflix to implement your algorithm as a feature, that Netflix instead had trashed it.

Thats what exactly happened way back in 2009, with the Netflix \$1 Million Challenge, where the winning team created an algorithm that helped increase the accuracy of Netflixs recommendation algorithm by 10.06%, only to find later that Netflix never implemented that solution itself.

Why Netflix Never Implemented It

Netflix wrote in ablog post discussing their recommendation system, where alongside, you can also find the reasons why they decided to dump the winning algorithm of the Netflix \$1 Million Challenge .

Before you go on to read the post, let me shed some light here.

Netflix is not dumb and its not like they are just offering \$1 million to the winners whilst sadistically laughing and dumping their algorithms into the bin.

As a matter of fact, the Netflix \$1 million Challenge was started in 2007 and each year Netflix chooses a winner, where from there, they decide whether the winning algorithms is right for their product. You can also find in the blog post that, Netflix did use some past algorithms that they thought were useful

Just to note. The first Progress Prize was held in 2007 when the Netflix \$1 Million Challengewas first started.

“A year into the competition, the Korbell team won the first Progress Prize with an 8.43% improvement. They reported more than 2000 hours of work in order to come up with the final combination of 107 algorithms that gave them this prize. And, they gave us the source code. We looked at the two underlying algorithms with the best performance in the ensemble: Matrix Factorization (which the community generally called SVD, Singular Value Decomposition) and Restricted Boltzmann Machines (RBM). SVD by itself provided a 0.8914 RMSE (root mean squared error), while RBM alone provided a competitive but slightly worse 0.8990 RMSE. A linear blend of these two reduced the error to 0.88. To put these algorithms to use, we had to work to overcome some limitations, for instance that they were built to handle 100 million ratings, instead of the more than 5 billion that we have, and that they were not built to adapt as members added more ratings. But once we overcame those challenges, we put the two algorithms into production, where they are still used as part of our recommendation engine.”

What about the 2009 winning algorithm? Nah, just chuck it down the bin.

“We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.”

The Birth of the Video Streaming Service

To be honest you can actually see where Netflix as a business, was heading towards way back then and it had nothing to do with the winning algorithm being crap. The algorithm was effective as it improved the recommendation search by 10.06% but in the end it proved to be useless.

Not only that, the way customers used the service and the transition from renting DVDs to streaming videos changed the way recommendations work.

“One of the reasons our focus in the recommendation algorithms has changed is because Netflix as a whole has changed dramatically in the last few years. Netflix launched an instant streaming service in 2007, one year after the Netflix Prize began. Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs our goal is to help people fill their queue with titles to receive in the mail over the coming days and weeks; selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.”

Whats pretty interesting to find that is that Netflix, with this new recommendation strategy, can know whether you finished that movie you were watching or stopped halfway through it.

This data helps curate the recommendations to cater for each individual and it is something that was very much needed to satisfy the movie needs of one rather than wait for the rental DVDs which take days to arrive.

The Key Takeaway from This Story

Netflix managed to become the forefront pioneer of the video streaming era but it came at a price of a winning million dollar algorithm.

Despite the scrutinizing they might have received way back then, they did what was right for their service, and provided something more valuable.

Maybe if they had implemented that winning million dollar algorithm back then, we wouldnt get to Netflix and Chill today.

Follow me on Snapchat, and I’ll keep you updated with what’s going on every now and then.

What do you think of Netflix’s decision to ditch the winning million dollar algorithm?