Eurovision: Impressive highlights but the full story?

Just like last year, a Scoreboard Journalism team convened to publish a prediction for the 2016 Eurovision Song Contest. This forecast not only selected the correct top-3 nations again but also the bottom-3. The associated blog suggested that Ukraine had a better chance than the odds suggested due to shortcomings in Russia’s song, and it correctly predicted that Poland’s chance was a lot better than its odds. One of the people who helped with this work advised his friends and followers to back Bulgaria to finish in the top-5. Bulgaria was fourth. So, all in all a pretty successful evening, at least it would be if we left our conclusions to cherry picking as many others do but was it the full story?

We wanted to compare our projection with the bookmakers and a variety of other projections. Last year, if you remember, we used predictions from the Guardian’s data team and a couple of specialist Eurovision sites as well as the bookmakers’ odds. However, this year there was no forecast from the Guardian. Instead @BuzzfeedUK produced a similarly data-driven one but I have not been able to source the full prediction as they only published a top-5:

I have asked on Twitter for the full projection so that I could assess it fairly against the rest but have had no response. As the Buzzfeed top-5 was by far the worst of the top-5s I have, I think we can probably safely conclude that the overall projection was not as good as those in my study. I would prefer to be able to analyse the full rank order though to be certain so, if someone has it, please get in touch.

If Buzzfeed’s was the worst, it would be the second year in a row in which a major media organization has produced a data-driven prediction which has been the poorest of the predictions that I found. That is a shame as the data-driven forecasts produced by everyone else have worked much better, suggesting that it is the methodology which is at fault in the data journalism world rather than taking a data driven approach. Colleagues of the people who made Buzzfeed’s data-driven forecast were even tweeting after the contest about how poor it was:

Runners and riders
Like last year, I found the predictions from ESC Chat, a forum dedicated to the Eurovision Song Contest which also asks its users to vote, and ESC Stats, a site whose name is self-explanatory, and uses a data driven approach with five factors equally weighted. Our own prediction this year was developed with the help of @oneminutecoach again but also used @kitlovelace‘s excellent approach to quantifying the songs based on musical trends. I also have a fourth model from @SanderFMC who combined bookmakers’ odds with iTunes data.


Best and worst

Scoreboard Journalism
Best: None spot on this year but Ukraine, Australia, Cyprus, the United Kingdom and the Czech Republic all within one place of where they finished on the night.
Worst: France were predicted in 23rd place but finished 6th.

ESC Stats
Best: One exactly right (2. Australia)
Worst: Three were 13 places out (8. Poland, 9. Lithuania and 11. Netherlands)

ESC Chat
Best: One exactly right (2. Australia)
Worst: Poland were predicted 21st place but finished eighth.

Best: Two exactly right (2. Australia, 23. Croatia)
Worst: Three were eight places out (4. Bulgaria, 9. Lithuania and 24. United Kingdom)

Top-3 and top-10 accuracy
It is probably more important to get the top right than the rest as people bet on this so this area ought to be looked at separately. Our forecast and ESC Chat both selected the correct top three although neither had them in the right order. ESC Stats and Sander had winners Ukraine in fourth position with France and Sweden respectively replacing them. Returning to the Buzzfeed prediction, that had just one of the actual top-3, hot favourites Russia, on the podium.

In terms of top-10 accuracy, Scoreboard Journalism, ESC Chat and ESC Stats did the best here naming seven of the top-10 nations correctly. Sander named six of the eventual top-10 in his projection.

Bottom-3 and bottom-10 accuracy
We didn’t look at this last year but having seen the reaction to naming the UK as the bottom country on Twitter prior to the contest, it seemed to me that this was also worth a look. Scoreboard Journalism was the only prediction to name more than one of the bottom three countries. Our forecast predicted all of the bottom-3, albeit in the wrong order. Everyone else named one correctly.

In terms of countries correctly placed in the bottom-10 however, our prediction only managed to forecast four of them. The best here was Sander who recorded an incredible nine out of 10 correct nations, missing out only on the United Kingdom who he placed 16th, one place above the bottom-10. ESC Chat had five right and ESC Stats four.

Accuracy of full ranking
For full disclosure though, we need to look at the overall rankings. We have done this in two ways, firstly by looking at the standard deviation of the difference between predicted and actual ranks (SD), and secondly by looking at the average difference between predicted and actual ranks (MAE). Both are used in analyses of my football prediction competitions. The latter of these is probably the easiest to understand as it is simply the average number of positions a prediction was wrong by across all 26 ranks. So, for example, Sander’s mean absolute error (as this is what it is called) was 3.31 meaning that he was wrong by just over 3 places on average. This is a very good score. In terms of both the MAE or the standard deviation, a lower score is better.

Sander: SD = 2.41, MAE = 3.31

ESC Chat: SD = 3.51, MAE = 5.46

Scoreboard Journalism: SD = 4.18, MAE = 5.31

ESC Stats: SD = 4.37, MAE = 6.23

Sander the best on the night
So, despite Scoreboard Journalism’s success at the top and bottom, it was Sander’s iTunes/bookmakers combination which proved to be the best due to no bad misses and a very good assessment of the bottom-10 in particular. His forecast was a great deal better than the rest overall as we can see from the metrics we have used above. The standard deviations for ESC Chat, ESC Stats and Scoreboard Journalism were all better than last year’s so there was more accuracy in the forecasts this time around.

But what about the bookies? Aren’t they always the best?
Well, this time the bookmakers selected the correct top-3 but, like those above, in the wrong order as they had Russia as a hot favourite. They only got six of the top-10 this year though after their incredible 10 out of 10 in 2015. This time they placed Bulgaria (4th), Poland (8th), Lithuania (9th) and Belgium (10th) outside the top-10. Lithuania was their worst call of all in 22nd but they also got Australia, Armenia and Croatia spot on. In terms of our metrics above, the standard deviation of their errors was 3.39 and they were an average of 3.85 places out with their predicted ranks.

So, we have a new champion. Sander beat not only the whole field but also the bookmakers themselves. We have invited him to join our little band for next year’s forecast and hope that he accepts. Then again, why does he need us? Oh yeah, we got the correct countries in both the top-3 and the bottom-3. Roll on next year.

This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s