New Rankings and Predictions System

I've been working on a new model which I think helps with some of the problems the Poisson model has. It is based on work done at Smart Football Rankings, an effort to develop a rankings and prediction system for college (American) football. I'll call this the averages model.

The idea is this: instead of making assumptions about the distribution of goals, let's just look at how well each team does at scoring and conceding goals compared to their opponents. There are two stages. In the first, I calculate a scoring and defensive factor by taking the difference between a team's goals for (also goals against but I'll just talk about the scoring half for this) in each match and how many goals on average were conceded by the opponent in their matches against other teams. To account for home-ground advantage, I adjust the average up or down for the match based on whether the team is at home or away. The adjustment is simply the difference between the league average goals and the average goals scored by home teams only. For example, Manchester United had given up 1 goal per match before facing Chelsea. Home teams on average score roughly 0.29 more goals per match than away teams. Chelsea scored one goal so their goals-for score from their match with Manchester United is 1 - (1 + 0.29) = -0.29. For Manchester United, Chelsea had conceded 8 goals in 11 matches for an average of 0.727 goals per match. Since United were playing away and failed to score, they would get 0 - (0.727 - 0.29) = 0.437. For the first step, this would be done for every team in each match. A team's scoring factor is then the average of these for each match played.

The second step adds a level. The first step compares your team's scoring to that of the average opponent of your opponent. If the teams you play have played an easy schedule themselves then your extra goals look less impressive. A way to control for this is instead of using average goals against use the scores calculated from step one along with the average goals for and against for the league as a whole. Getting back to the Chelsea - Manchester United example, Manchester United's defensive score from the first step is -0.689. In other words, they've given up about 2/3 of a goal per match less on average compared to what their opponents have scored against other opponents. The league average for the league as a whole is 1.52 goals per match, and home teams 1.80. Chelsea scored 1 goal against United so they get 1 - (-0.689 + 1.80) = -0.111. Chelsea's defensive score from step 1 is -0.83 and away teams average 1.23 goals per match so Manchester United got 0 - (-0.83 + 1.23) = -0.4. Each team's scoring and defensive factor is the average of of these for all matches played.

There are two reasons I like this system better than the Poisson. The first is that it has gotten better results in some testing. I've done something very similar to the PLM where I use ordered logit on the expected goals for each team according to the model and its predictions have been better. If there is interest I'll post more detailed work there but I used two different scoring systems. One just looks at squared difference between predicted probabilities and what actually happened, assigning 1 if the outcome (home win, away win, draw) happened. The other was a betting system where I looked at how much would be made by the PLM using odds given by the estimates of the averages model and the other way around. The averages model outperformed the PLM in both of these tests. Beyond just these things, I think the rankings make more sense when I look at where it rates teams.

The second reason I like the averages model better is that it I think using sums is more accurate than using products. In the Poisson model, home-ground advantage is given by multiplying the expected goals for home team by a number between 1.1 and 1.5 for most competitions. Similarly, the expected goals in a match for a team is their scoring factor times the opponent's defensive factor. A result of that is that high-scoring teams are more sensitive to the quality of their opponents and playing at home than low-scoring teams. This doesn't seem to be reflected in reality. I'll write more on this later, but for better teams playing at home tends to be a bit less important. The averages model assumes it equally important for all teams so that's an improvement. Note that other than including home-ground advantage, a major difference between mine and the methodology used by Smart Football Rankings is that they actually use products instead of sums.

For at least the next few weeks I'll include both Poisson/PLM and the averages model when giving rankings and predictions. Because I believe the averages model, and its logit, to be superior I'm using it as my main model until I work out a better one.

Poisson-Logit Model, a Deeper Look

I've done a bit more work with the Poisson-Logit Model (hereafter PLM), so I thought I would make a post about it that I can reference in the future.

How It Works

As the name implies, it has two stages: the Poisson stage and the logit stage.

The Poisson stage breaks the game down into two parts which are the scoring at either end. Using goals scored and conceded for each team, total home goals to get a home-ground advantage, and the schedule of each team it outputs a scoring and defending factor. When two teams play each other, the average goals a team will score is their scoring factor times the other team's defending factor, times the home-ground advantage multiplier if they are playing at home.

The logit stage uses previous data to get a representation of how often each result (home win, draw, away win) have historically happened when two teams meet with average goals as determined by the Poisson stage. So the inputs are the scoring and conceding factors for each team as well as the home-ground multiplier. The output is a home win probability, an away win probability and the probability of a draw. For occasions where scorelines are of interest I also have run a different version which as the same inputs and gives as an output the probability of all scorelines from 0-0 all the way to 10-9.

How Has It Done?

Past results with it have been quite good. I have run it on 8 seasons of data from several leagues. An important thing is that my suspicions were confirmed and it worked a lot better when I didn't use it for matches where the home team had played fewer than 10 matches. In other words, I waited 10 weeks into the season for there to be some kind of sample size so there wouldn't be crazy data or some of the small sample problems we've seen here (see the Manchester derby predictions).

The first test is how many predicted home wins, draws and away wins the model predicts compared to what actually happened. Here are the results:

Type Pred. Actual
Home 16456 16479
Draw 9565. 9444
Away 8756. 8853

As you can see, it's off by a bit but quite close when you think of them in percentage terms. Using the Chi-Square goodness of fit test, I got a p-value of 0.267. This indicates a good fit.

For the next test, I broke the matches up into groups by how often the model predicted the home team to win (I did the same for draws and away wins and the results were similar). For each group I then compared the predictions to how often the home team actually won to see how accurate the model got the predictions. The results there were quite good. Here's a graph with them. Instead of using bar charts I'm using a line graph as I think it's easier to see where the (slight) differences are:



The groups are every 5%. For example, there were only 26 matches where it was predicted that the home team had less than a 5% chance. These predictions averaged out to 2.96%. The home team actually won 11.5% of the time so you can see that at the bottom of the graph the actual line is a fair bit above the prediction line. It goes back toward the actual line because the model did better from 5% to 10%.

Other than the very bottom group, the predictions of the model were within a couple standard deviations of the predicted percentage, usually a lot less. Running a similar goodness of fit test on this, I got a ridiculous p-value of .695. This model appears to fit the data incredibly well.

Is This Model Perfect?

If it was then quite frankly I would move to Vegas and be rich in a season. Unfortunately for me it's not that simple. I am extremely happy with the model and predictions it makes, but it certainly has limitations. The problem is that it doesn't take into account outside factors, most notably injuries. These things, when you have a huge data set, cancel each other out. Say you have a match where the model predicts the home side will win 40% of the time. That 40% is really an average. Sometimes the home team will have a star player with an injury and really be more like 30% to win, other times they will be healthy and the visiting side will have injury or suspension issues. In these spots they might be 50%. Over the whole set the injuries will even out so the home team will actually win the 40% of the time, but looking at individual matches the predictions can be off even if on aggregate they are perfect or close to it.

Conclusion

I am quite happy with the model. It is just as simple as the Poisson model and makes predictions that are as accurate as possible given that simplicity. I feel quite comfortable using it as the base model so for now I will stick to it. None the less, it should be thought of as a baseline and not the be-all-end-all. Some thought is certainly needed to assess how to adjust for factors outside the model such as injuries, suspensions and even things like tactics if some teams possibly matchup particularly well against a certain style of opponent or something along those lines.

A Look at Predictions

I've been making a lot of predictions using the Poisson model, now I'll look at how well its predictions are.

Determining how good the predictions are is difficult because the predictions are probabilistic (ie 40% chance of home win, 30% of an away win, 30% of draw) but the final result is deterministic (the home team won). I certainly welcome suggestions with other good ideas, here is how I think it best to analyze how good a given model is at making predictions. The first step is to get a large data set and make predictions using only data available at the time. After that, I put the predictions into bins. All matches in which the model predicted a home win less than 5% of the time are put together, all those where the prediction is that a home win will happen between 5 and 10% of the time are together and so on up to those between 95% and 100%. For every match in each of these bins, excel determines the average predicted home-win percentage and how often the home team actually won. For each bin I record how far off the prediction was and the absolute value of these numbers is added. Smaller the better.

If that explanation didn't make sense, I'll show it in graphical form here for the Poisson model. My data set is every season of the Spanish Primera Division from 1999-2000 through last season.I started the predictions early - after the third matchday. Because the predictions don't really work if one team has either not yet scored or conceded a goal, I did have to throw out some data from the early weeks.

The horizontal axis represents the predicted percentage of a home win, while the vertical axis represents the percentage of the time that the home teams for a given prediction probability actually won. For convenience, the red line has slope 1 and represents perfect predictions. Here is the graph for the Poisson model:



This goes in line with what I have written before: the model does a good job with the predictions when the teams are relatively even but overpredicts wins for underdogs and underpredicts them for heavy favorites.

An Improvement: the Poisson-Logit Model

In an effort to improve the predictions, I tried a new technique. The strategy was to take the expected goals for each team that the Poisson model spits out (for the home team their scoring factor times the away team's defending factor times the home-field factor) and then run what is called multinomial logit to get the result probabilities, instead of applying the Poisson model to estimate them.

The results of that were fantastic. Here is a graph of the results of the Poisson-Logit Model (hereafter PLM):



There are a couple nice features in this graph. Firstly, the PLM does not appear to be biased one way or the other. There is a small correlation between the predicted percentage and how far off the predictions are from reality, but that's only from the blip around 0.8 that is almost certainly an anomaly. Looking at the graph, it seems as though the errors are just as a result of noise but the predictions are quite good.

For comparison, I ran a simulation where the predictions were perfect and the sample sizes of each bin were the same as those for the graph above. For example, the data set contains 189 matches where the PLM predicts that the home side will win with probability between 0.7 and 0.75, with an average prediction of 0.726. For the graph below, I used the rand function in excel to simulate 189 matches where the home side has a 72.6% chance of winning.



From this graph you can see the effect of noise with this sample. In the graph above, the predictions were 100% perfect but because of variance and the sample sizes involved the blue line does not just run along the red line.

Comparing this to the graph above, I feel confident in the predictions of the new model. Adding up the squared differences between the predicted and actual percentages in each bin, the PLM actually did slightly better than perfect predictions! That's obviously just due to the sample sizes involved as it's impossible to beat perfect predictions, but it's a strong sign that the PLM is very good at making predictions.

From now until further notice I will use the PLM instead of the Poisson model. I am now working on gathering data for other leagues to look at how good the predictions are early on in the year as well as whether or not it's better to eliminate goals in blowouts.

The Poisson Model - a Deeper Look, Part 1

I've been using the Poisson model to make my World Cup qualification predictions. Let's take a closer look at it. I'm not sure how much a lot of you will be interested in this sort of article, it's a behind-the-scenes kind of thing. I will be on occasion making these kinds of posts for a couple reasons. For one, I think it's necessary to give justification for my predictions. If I were making picks based on my subjective opinion from watching matches I would give some reasoning. The same applies to using a model to make predictions, though in that case it would be justification for the model more than the individual picks. The second reason is that I'll be working to improve the models and writing these posts will help me with that.

Poisson Distribution Overview

The model is based on the Poisson distribution. Roughly speaking, the Poisson distribution describes how many occurrences of something you will have when they are infrequent but have a lot of chances to occur. This seems applicable to football because there are a lot of minutes (or seconds if you like) in a match and the probability of a goal any given minute is low.

The major assumption that the Poisson distribution makes, when applied to football, is that the probability of a goal being scored by a given team is the same every minute. In other words, it doesn't matter what the score is, when the last goal was scored or how much time is left. This assumption greatly simplifies things, but I can't imagine any fan of the game thinking it is correct. When a goal is scored things change for both teams. I suppose it's possible that things perfectly balance to cancel each other out - A attacks more because they are behind but B is more defensive as well so they have the same chance of scoring as when they were tied. That does not seem plausible though. I'll write more in the future on how the scoreline affects each team's chances of scoring. A preliminary look at a single-season of Premier League data suggests that there are significant differences in how frequently goals come for either team with different scorelines.

There are 4 inputs for the Poisson model: goals scored for each team, goals conceded by each team, a list of which teams played at home and away, and how many goals in total have been scored at home. Note that it does not use results from individual matches. Other than a list of fixtures, all inputs for the model are aggregate. This is possible precisely because of the above assumption. If the chance of a goal does not depend on the scoreline then you don't need to worry about how many goals were scored when the team was ahead or behind or how much time each team spent with the score a certain way.

How Well Does It Describe Football Scorelines

If you stopped reading there, you'd probably think that the Poisson model is useless because it is based on flawed assumptions. Before you jump to that conclusion, let's look at how well the model explains the game. In future articles I will discuss how well it predicts the future. That is a lot more difficult, in part because the sample sizes are smaller. Here I am discussing how well the model does with a full season of data. In other words, for every season in my sample I use the end-of-the-season information to run the model and then look at how well the model fits the data.

The data I used was the previous 9 seasons in the Spanish Primera Division. To test how well the model fits, I compared how often they actually happened to the frequency the model assigns to each result (home win, draw, away win) and each number of goals for the home and away team.

Match Outcomes

The first thing I looked at is how often the model indicated the match would result in a home win, draw and away win compared to how often those actually happened. Here is a graph of the predictions. I went with a line graph instead of the more typical bar graph so that you can more easily see how the differences vary by result. The first node is for the home win, the second for a draw and the third the away win. The vertical axis is the percentage of the time they actually occurred or the model claimed they would. The blue line is how often they actually happened, the red line represents the estimates of the model.



As you can see, the model under estimates the percentage of home wins and draws and over estimates the likelihood of an away win. Because the sample size is pretty large, these differences are statistically significant. The p-value for the Chi-squared goodness-of-fit test is less than .02. In other words, looking at the actual results compared to the estimates from the model, we can conclude that the predictions of the Poisson models are off. Despite that, the estimates are pretty good.

Goals

Let's now look at how often the model estimates each number of goals will occur in a match. The first panel is goals for the home team, the second the away team and the last all goals. Again, the blue lines represent the actual frequency and the red lines those predicted by the model.



Looking at the charts, the model overestimates the frequency of 0 goals and underestimates how often teams score 1 goal for both home and away teams. These differences are all statistically significant at the standard 5% level of significance. It also appears that the model underestimates how often the home side scores 2 goals and 3 goals, but more data would be needed to confirm that. Similarly the model may overestimate how often a team scores 4 or more goals for both home and away teams. The chi-square p-value is less than .01 so again we can conclude that the model does not fully fit the data.

You can see why the Poisson model overestimates how often the away team wins. The model is more off for the home team than away when giving the frequency of going goalless. It also seems to underestimate how often the home team scores two goals, which it does not do for the away side. In other words, the model is a bit off whether you look at the home team or the away team, but it is a bit harsher on the home side. I should say that these differences aren't statistically significant, but things appear that way and statistical significance would require an insanely large sample size.

Conclusion

Looking at the figures, the Poisson model does a pretty good job but is not perfect. Based on the difference between home and away goals, it appears that it is off in a systematic way - it underestimates scoring by a larger margin if the expected number of goals is higher. As a result, it will overrate teams that score few goals, concede a lot and/or are playing away and underrate those that score a lot, are stingy defensively and/or are playing at home. This is good and bad news. On one hand the model being wrong in a systematic way means that certain teams will consistently be rated incorrectly and their chances over or underestimated. On the other hand it means that adjustments can likely be made to improve the model.

The next article in this series I will tweak the model to hopefully deal with these issues. I will then compare the mid-season predictions made by the vanilla model and the alternative version(s) to see how much we can improve the Poisson model. I think and hope that improvements can be made.

The Poisson Model

I will frequently use what I call the Poisson model, particularly in the near future. The Poisson model takes as inputs the goals scored and goals conceded for each team, as well as the schedules, and spits out an attacking and defending factor for each team and, in the version I will use, a home-field-advantage factor. I can then plug these back into the Poisson formula for any pair of teams and it gives me the probability of each scoreline if they play against each other in a given location. In other words, it provides a way to translate current and/or past results into how likely each possible outcome is in potential future matches.

The Poisson distribution is used in many fields to describe the frequency of various occurrences. This includes things like how many typos in a page, how many atoms of radium will decay in a given period of time, and the number of accidents at an intersection in a month. In order to give my objective predictions and other analyses, I will assume that the number of goals in a match follows this distribution. I further assume that if A plays B at home then the expected number of goals team A scores against B is A’s attacking factor multiplied by B’s defending factor, times the home-field-advantage factor. B’s expected number of goals is B’s attacking factor multiplied by A’s defending factor, without the home-field-advantage bonus. If you aren’t familiar with probability and statistics, you can think of the expected number of goals as what the average would be if they played each other thousands of times in some parallel universe. Using these assumptions and something called the Maximum Likelihood Estimator, I derive estimates for the attacking, defending and home-field-advantage factors. Once those have been estimated, they can be plugged into the Poisson distribution formula to get the probability of each outcome if any two teams were to play each other. I will use these predictions to not only give previews each week for various leagues, but also to analyze how likely teams are to finish in certain spots in their league, avoid relegation, win the title and so forth.

The Poisson model is not perfect. Without getting too technical, it makes a number of simplifying assumptions that most any football fan would not believe to be true. The main one is that the chance of a goal in any period of time, let’s say a minute, does not depend on the score or how much time is left in the game. In other words, it is assuming that a team down 1-0 is as likely to score as they would be if the score was tied 0-0. Because teams change tactics based on the score, it seems that this may not be the case. Also, in extreme situations such as a team being up 5-0, they likely will call off the dogs and be less likely to score than if they were only up 2-0. As a result of this, the model overestimates the likelihood of certain scorelines and underestimates the chances of others. I will be writing at least one article in the future taking an in-depth look at this.

Despite these problems, the Poisson model does a reasonable job of predicting match outcomes. A huge benefit is that it is simple. People have developed more complicated models that are dynamic. In other words, applying them to football, the probability of a goal would depend on how much time is left in the game, what the score is and perhaps when the last goal was scored and by whom. While these would be more accurate, they are quite cumbersome. Essentially the Poisson model makes assumptions that aren’t quite valid, but drastically simplify things. It is easy enough to use that I can quickly apply it to any league or competition where data is available should a reader want me to analyze some competition I haven’t discussed.

I am working on a better model, which would be more accurate and hopefully not much more complicated, but do not have everything worked out. If and when that happens I’ll start using it, but in the meantime the Poisson model is a solid place to start.