I want to take a deeper look at the GAR and xGAR models from http://www.Evolving-Hockey.com. However, it’s very difficult to evaluate these models at the individual player level. There is no agreed upon way to measure the performance/impact of a player. You can look at points, on-ice goal differential, xG%, the eye test or any number of things. The GAR models try to estimate the impact of a player by factoring in a variety of stats. But we can’t really determine if the model is good or bad, since there is no agreed upon test.
In many ways, player evaluations are just subjective opinions. This is why, my approach is slightly different. I’m not going to discuss whether Valeri Nichushkin is better than Leon Draisaitl. Instead, I will focus on the team level. How well does the GAR and xGAR models describe team performances?
We know how to define a good team. Good teams outscore their opponents and win hockey games. It’s as simple as that. Therefore, I’m going to use goal differential to measure team performance. This gives us a stat that we can compare with team GAR/xGAR.
Just because team GAR correlates well with goal differential at the team level, doesn’t mean it’s necessarily a great descriptive model at the player level. A model can’t be great at the player level and bad at the team level, but it can be great at the team level and bad at the player level.
My first step is to simply look at the correlation between team GAR/xGAR and team goal differential. I’m looking at every season since 2007/2008, which gives us 393 data points (10 seasons with 30 teams and 3 seasons with 31 teams). The data is prorated to 82 games, so that I can use the data from the 2012/2013 and 2019/2020 seasons.
Let’s start by looking at the skaters only. Here’s team G+/- as a function of skater_GAR:
There’s obviously a decent correlation, but not exactly perfect. I have also looked at the xGAR model, and here’s that correlation:
Surprisingly, xGAR describes team results better than GAR does. This is a bit surprising since GAR is supposed to be more of a descriptive model whereas xGAR is supposed to be more predictive in its nature. The difference is not huge, but it’s still interesting.
So, let’s add the goaltender GAR and see what happens:
With this addition the correlation in both models increases, but more in the xGAR model. In fact, the xGAR model already correlates really well with team performance. There are two things worth noting though:
- GK_GAR = GK_xGAR – There’s only one GAR stat for goaltending.
- GAR > xGAR – In general players have a larger GAR than xGAR.
This could indicate that GK_GAR is factored better for the xGAR model than the GAR model.
I also tried using GSAx (based on Evolving-hockey’s xG model) as my goaltender stat. That gave the following result:
Now, we see the GAR model being slightly better than the xGAR model. Part of this is clearly because of factoring. I won’t draw any conclusions yet, but I will address the factoring issue later on. Other than that, I will just leave the graphs as they are now.
Instead, I will try and redefine GAR and xGAR, so that I can compare them directly to goal differential, which is a zero-sum stat. The sum of the goal differentials for all NHL team will always be zero, so therefore I want to reform the GAR and xGAR models into zero-sum models. I’m going to call the models goals above average (GAA) and expected goals above average (xGAA), because that’s what they are. My GAA and xGAA are different than the ones you can find on Evolving-hockey.com though. So, be aware of that.
I have based the zero-sums on year, position and component. This means that the total sum of GAA/xGAA in a season is always zero, the total sum amongst forwards (for instance) is always zero and the total sum of PPO_GAA (for instance) is always zero. The calculations can be found here (Appendix 1).
This has some consequences on the player level, but overall I think the redefined models are better. In general forwards are ranked lower in the redefined models and defenders are conversely ranked higher. PP specialists are also ranked lower whereas PK specialists are ranked higher. I think, the GAR/xGAR models may be biased towards forwards and PP specialists.
With all the calculations out of the way, we can now compare the redefined models with team goal differential. Here’s the skaters only and since the model is now a zero-sum model like goal differential, I don’t have to prorate the data:
We see that Skater_GAA and Skater_xGAA correlates similarly well with goal differential as Skater_GAR_82 and Skater_xGAR_82 did. This isn’t surprising since the changes I’ve made primarily affects forwards vs defenders and PP specialists vs PK specialists, so as long as every team plays 3 forwards and 2 defenders 5v5, you wouldn’t expect a big difference.
Now, we can add the goaltending component – GK_GAA:
Exactly like we saw earlier, the goaltending component fits better with xGAA than with GAA. A big part of this, is due to factoring/weighing of the different components. I will therefore refine the graphs to optimize the correlation. In the process I make sure that 1 GAA/xGAA equals 1 actual goal.
For the GAA model I’ve made the following adjustments:
There’s a relatively big difference between the weight put on EVO and the weight put on EVD. Due to the zero-sum nature of the model, this means that the extreme cases of EVD (either really good or really bad defense) get rated closer to average. This is of course great for someone like Leon Draisaitl, who is elite offensively but really bad defensively. However, the process where we went from GAR to GAA was equally bad for him, since he’s a forward and a PP specialist. Going from GAR to GAA to GAA_adjusted Drasaitl’s league rank went from 23rd to 41st and back to 23rd.
In the end the GAA_adjusted rankings are pretty close to the original GAR rankings.
Anyway, here’s the graph:
When weighing the components accordingly, we end up with a pretty good correlation between GAA_adjusted and the goal differential.
I’ve also refined the xGAA model, and that gave the following adjustments:
Here the differences between the factors are smaller indicating, that the model was already pretty good. Theoretically, you would like the goaltending and penalty components to be factored equally in the two models since they are the same. This is pretty close though.
So, here’s the graph with the adjustments to xGAA:
I would also like to see what happens if we use GSAx instead of GK_GAA. I’ve turned GSAx into a zero-sum model in accordance to the calculations in appendix 1. Here’s the adjusted results, when we use GSAx instead of GK_GAA:
When we go from GK_GAA to GSAx the weight put on the defensive components (EVD and SHD) increases and this is the case in both models. I think this indicates, that GK_GAA isolates goaltending from defense better than GSAx does. However, when we’re using GSAx the models correlate better with goal differential.
Here’s the GAA model with GSAx as the goaltending component:
And here’s the adjustments to xGAA model if we use GSAx:
Which result in the following graph:
Going forward I will be using GSAx as my goaltending component. I don’t necessarily think GSAx is a better stat than GK_GAA, but it fits better with the GAA and xGAA models.
Now, we have two models that can be compared directly, so what happens if we combine them? Here’s the combined model, which I have called sGAA:
I won’t say much about the graph, but just conclude that the combined model correlates better with goal differential than the two models individually.
So, what was the purpose with this exercise? First of all, I wanted to make a descriptive model where you can see the impact of a player directly. If a player has a GAA of +4, he’s simply worth 4 goals more than the average player and he’s worth 4 goals to the goal differential of the team.
Secondly, I want the model to correlate directly to team performance, so that it can be used as the basis in a predictive model, that predicts future team results.
In the beginning of the article, I said that just because a model is good at the team level it’s not necessarily good for making player evaluations. I’ve shown that GAA_adjusted, xGAA_adjusted and sGAA correlates really well with team performance, so internally on a team the models work. If a player is undervalued by one of the models, then other players on the same team must be overvalued.
I don’t trust Evolving-Hockey’s GSAx to be a great goaltender stat. As an example, I think Henrik Lundqvist is overrated in terms of GSAx (especially in the early years), so consequently the skaters on those Rangers teams must be underrated.
The GAR and xGAR models from Evolving-Hockey are primarily used for player evaluations, but I think these redefined GAA models can be great for team evaluations as well. One application could be to make performance charts, where you look at the performance of a particular team over time.
Here’s an example of Anaheim. The x-axis is year (year 8 = 07/08, year 9 = 08/09 and so on) and the y-axis is goal differential. In this performance chart I’ve just compared the actual goal differential to the two models.
And here’s the combined model (sGAA) compared to the actual G+/-
You could also look at the components individually. Here’s an example of Tampa Bay, where I’ve used sGAA components:
The possibilities are endless. You can compare the goaltending on different teams or you can compare EVO vs. EVD over time. The sGAA data can be downloaded here.
Going forward further evaluation of the sGAA model is needed. One way to do so, would be to look at repeatability. How repeatable is each component from year to year? This is a very important knowledge to have, when it comes to evaluating the predictive power of the model.
It’s also worth taking a deeper look at the goaltending stats. Can we improve on the GSAx data?
To end this article, I have added tables of the top 20 skaters in GAR, xGAR and sGAA along with the top 20 goalies in GAR. I will just leave the tables uncommented, but it gives you an opportunity to compare GAR/xGAR to the redefined models.
Top 20 skaters: GAR 2019/2020
|Player||GP||TOI_ALL||GAR||GAA adj||Rank GAR||Rank GAA adj|
Top 20 skaters: xGAR 2019/2020
|Player||GP||TOI_ALL||xGAR||xGAA adj||Rank xGAR||Rank xGAA adj|
Top 20 skaters: sGAA 2019/2020
Top 20 goalies: GAR 2019/2020
|Player||GP||GAR||GAA||GSAx||Rank GAR||Rank GAA||Rank GSAx|
All data from http://www.Evolving-Hockey.com