Assessing the Premier League predictions

At the beginning of the Premier League season, and also at the end of the transfer window, we collected a total of 60 predictions for the final rank order of this season’s Premier League. Some were submitted to us with points and others were collected online or from newspapers and therefore did not include points. These were awarded points based on the average for each position from recent seasons. The forecasts came from statistical modellers (including two bookmakers), media and fans. The original post with an overview of these predictions can be found here: 

The Result
So, without further ado, here is the result based on points:

On a points basis, Joe Prince Wright came out best, followed by his colleagues at NBC, Nick Mendola and Jason Le Miere. The opening lines at Sporting Index beat Le Miere but Prince-Wright and Mendola even beat those spreads. Next best was a fan, Liverpool supporter Dick Bustin, who beat the rest of the fans by some distance, and a number of other predictions, al from the media, outperformed the best non-bookmaker statistical model.

Pre-season and post transfer window
As might be expected, some forecasts made after the transfer window had closed (those symbols not filled on the chart) performed better than those made pre-season by the same people. This was the case for Glenn Moore and Steven Maclean as well as John Bassett’s models (just). Soccermetrics however was broadly the same as their pre-season forecast in terms of performance and Steve Lawrence‘s was actually poorer after three games than it had been with no current season information. Very few of the media predictions included points and arguably had an advantage as the points allocated to them were based on averages per position. Thus a fair comparison across all the groups can only really be made on the positions that were forecast.

Forecasting points

Here, the statistical modellers were best with Bloomberg Sports narrowly beating ClubElo to win the competition with Joe Prince-Wright, lead writer and editor of NBC’s Pro Soccer Talk coming in third. The presence of Prince-Wright is particularly interesting as he won the media competition on this basis last year which is early evidence of forecasting skill although he will need to continue this level of performance over many more seasons in order to confirm this to be the case. Prince-Wright’s performance this year was even better than in 2013/2014 as the standard deviation of his prediction errors is somewhat smaller. This could be due to this season’s competition being more predictable though. The BBC’s Phil McNulty also delivered a consistently performance over each of the two seasons.

Which group is best?
Overall, the statistical modellers are better than the media in predicting the finishing positions and the journalists are, in turn, better than the fans. As groups, all beat the naïve base of predicting the clubs to finish with exactly the same points as last season. However, only nine of the 26 statistical models did so along with six of the 18 media forecasts and just two of the nine fan predictions. The pair of fans who achieved this were Liverpool supporter Dick Bustin and Manchester United follower Mark Thompson. They are therefore welcome to come back and represent their clubs next season, as is Crystal Palace fan Edward Porter who was the best of the rest of the fans. I will attempt to find a supporter from each of the other 17 Premier League clubs too. All four predictions, opening and closing lines pre-season, from Pinnacle and Sporting Index beat this naïve base as would be expected from professional bookmakers.

Over both seasons?
Returning to the 16 participants of the competition in both 2013/2014 and a season later confirms Joe Prince-Wright (right hand analysis) as the overall king of the competition and someone who even managed to outperform bookmakers Pinnacle over that period. On the points side it is two models, DecTech and James Fennell, who lead the way. The majority of the 16 outperformed the naïve base prediction of “same as last season” although Soccermetrics, Glenn Moore and Kyle Bonn all failed to in terms of position.

 

And next season?
Next season begins on August 8 and I would like to invite journalists to submit their full predictions to me with points this time around as it makes for a better comparison. I would also like to find one supporter per Premier League club (excluding Crystal Palace, Liverpool and Manchester United who I already have) to submit a pre-season forecast. My aim is to cap the entrants next season at 20 per group and one per person which means that I need a few more statistical model based predictions.

Given that Michael Caley and Stephen McCarthy were both game enough to enter pre-season predictions, I hope that some of the other expected goals modellers will do so as well. In the end, the aim here is to encourage people to produce better forecasts so measuring yourself objectively against all other predictors is a worthwhile exercise for everyone. Thanks to everyone who took part and roll on next season.