Interpretation and redefining of Evolving-Hockey’s GAR and xGAR models.

I want to take a deeper look at the GAR and xGAR models from http://www.Evolving-Hockey.com. However, it’s very difficult to evaluate these models at the individual player level. There is no agreed upon way to measure the performance/impact of a player. You can look at points, on-ice goal differential, xG%, the eye test or any number of things. The GAR models try to estimate the impact of a player by factoring in a variety of stats. But we can’t really determine if the model is good or bad, since there is no agreed upon test.

In many ways, player evaluations are just subjective opinions. This is why, my approach is slightly different. I’m not going to discuss whether Valeri Nichushkin is better than Leon Draisaitl. Instead, I will focus on the team level. How well does the GAR and xGAR models describe team performances?

We know how to define a good team. Good teams outscore their opponents and win hockey games. It’s as simple as that. Therefore, I’m going to use goal differential to measure team performance. This gives us a stat that we can compare with team GAR/xGAR.

Just because team GAR correlates well with goal differential at the team level, doesn’t mean it’s necessarily a great descriptive model at the player level. A model can’t be great at the player level and bad at the team level, but it can be great at the team level and bad at the player level.

My first step is to simply look at the correlation between team GAR/xGAR and team goal differential. I’m looking at every season since 2007/2008, which gives us 393 data points (10 seasons with 30 teams and 3 seasons with 31 teams). The data is prorated to 82 games, so that I can use the data from the 2012/2013 and 2019/2020 seasons.

Let’s start by looking at the skaters only. Here’s team G+/- as a function of skater_GAR:

There’s obviously a decent correlation, but not exactly perfect. I have also looked at the xGAR model, and here’s that correlation:

Surprisingly, xGAR describes team results better than GAR does. This is a bit surprising since GAR is supposed to be more of a descriptive model whereas xGAR is supposed to be more predictive in its nature. The difference is not huge, but it’s still interesting.

So, let’s add the goaltender GAR and see what happens:

With this addition the correlation in both models increases, but more in the xGAR model. In fact, the xGAR model already correlates really well with team performance. There are two things worth noting though:

  1. GK_GAR = GK_xGAR – There’s only one GAR stat for goaltending.
  2. GAR > xGAR – In general players have a larger GAR than xGAR.

This could indicate that GK_GAR is factored better for the xGAR model than the GAR model.

I also tried using GSAx (based on Evolving-hockey’s xG model) as my goaltender stat. That gave the following result:

Now, we see the GAR model being slightly better than the xGAR model. Part of this is clearly because of factoring. I won’t draw any conclusions yet, but I will address the factoring issue later on. Other than that, I will just leave the graphs as they are now.

Instead, I will try and redefine GAR and xGAR, so that I can compare them directly to goal differential, which is a zero-sum stat. The sum of the goal differentials for all NHL team will always be zero, so therefore I want to reform the GAR and xGAR models into zero-sum models. I’m going to call the models goals above average (GAA) and expected goals above average (xGAA), because that’s what they are. My GAA and xGAA are different than the ones you can find on Evolving-hockey.com though. So, be aware of that.

I have based the zero-sums on year, position and component. This means that the total sum of GAA/xGAA in a season is always zero, the total sum amongst forwards (for instance) is always zero and the total sum of PPO_GAA (for instance) is always zero. The calculations can be found here (Appendix 1).

This has some consequences on the player level, but overall I think the redefined models are better. In general forwards are ranked lower in the redefined models and defenders are conversely ranked higher. PP specialists are also ranked lower whereas PK specialists are ranked higher. I think, the GAR/xGAR models may be biased towards forwards and PP specialists.

With all the calculations out of the way, we can now compare the redefined models with team goal differential. Here’s the skaters only and since the model is now a zero-sum model like goal differential, I don’t have to prorate the data:

We see that Skater_GAA and Skater_xGAA correlates similarly well with goal differential as Skater_GAR_82 and Skater_xGAR_82 did. This isn’t surprising since the changes I’ve made primarily affects forwards vs defenders and PP specialists vs PK specialists, so as long as every team plays 3 forwards and 2 defenders 5v5, you wouldn’t expect a big difference.

Now, we can add the goaltending component – GK_GAA:

Exactly like we saw earlier, the goaltending component fits better with xGAA than with GAA. A big part of this, is due to factoring/weighing of the different components. I will therefore refine the graphs to optimize the correlation. In the process I make sure that 1 GAA/xGAA equals 1 actual goal.

For the GAA model I’ve made the following adjustments:

There’s a relatively big difference between the weight put on EVO and the weight put on EVD. Due to the zero-sum nature of the model, this means that the extreme cases of EVD (either really good or really bad defense) get rated closer to average. This is of course great for someone like Leon Draisaitl, who is elite offensively but really bad defensively. However, the process where we went from GAR to GAA was equally bad for him, since he’s a forward and a PP specialist. Going from GAR to GAA to GAA_adjusted Drasaitl’s league rank went from 23rd to 41st and back to 23rd.

In the end the GAA_adjusted rankings are pretty close to the original GAR rankings.

Anyway, here’s the graph:

When weighing the components accordingly, we end up with a pretty good correlation between GAA_adjusted and the goal differential.

I’ve also refined the xGAA model, and that gave the following adjustments:

Here the differences between the factors are smaller indicating, that the model was already pretty good. Theoretically, you would like the goaltending and penalty components to be factored equally in the two models since they are the same. This is pretty close though.

So, here’s the graph with the adjustments to xGAA:

I would also like to see what happens if we use GSAx instead of GK_GAA. I’ve turned GSAx into a zero-sum model in accordance to the calculations in appendix 1. Here’s the adjusted results, when we use GSAx instead of GK_GAA:

When we go from GK_GAA to GSAx the weight put on the defensive components (EVD and SHD) increases and this is the case in both models. I think this indicates, that GK_GAA isolates goaltending from defense better than GSAx does. However, when we’re using GSAx the models correlate better with goal differential.

Here’s the GAA model with GSAx as the goaltending component:

And here’s the adjustments to xGAA model if we use GSAx:

Which result in the following graph:

Going forward I will be using GSAx as my goaltending component. I don’t necessarily think GSAx is a better stat than GK_GAA, but it fits better with the GAA and xGAA models.

Now, we have two models that can be compared directly, so what happens if we combine them? Here’s the combined model, which I have called sGAA:

I won’t say much about the graph, but just conclude that the combined model correlates better with goal differential than the two models individually.

Discussion:

So, what was the purpose with this exercise? First of all, I wanted to make a descriptive model where you can see the impact of a player directly. If a player has a GAA of +4, he’s simply worth 4 goals more than the average player and he’s worth 4 goals to the goal differential of the team.

Secondly, I want the model to correlate directly to team performance, so that it can be used as the basis in a predictive model, that predicts future team results.

In the beginning of the article, I said that just because a model is good at the team level it’s not necessarily good for making player evaluations. I’ve shown that GAA_adjusted, xGAA_adjusted and sGAA correlates really well with team performance, so internally on a team the models work. If a player is undervalued by one of the models, then other players on the same team must be overvalued.

I don’t trust Evolving-Hockey’s GSAx to be a great goaltender stat. As an example, I think Henrik Lundqvist is overrated in terms of GSAx (especially in the early years), so consequently the skaters on those Rangers teams must be underrated.

The GAR and xGAR models from Evolving-Hockey are primarily used for player evaluations, but I think these redefined GAA models can be great for team evaluations as well. One application could be to make performance charts, where you look at the performance of a particular team over time.

Here’s an example of Anaheim. The x-axis is year (year 8 = 07/08, year 9 = 08/09 and so on) and the y-axis is goal differential. In this performance chart I’ve just compared the actual goal differential to the two models.

And here’s the combined model (sGAA) compared to the actual G+/-

You could also look at the components individually. Here’s an example of Tampa Bay, where I’ve used sGAA components:

The possibilities are endless. You can compare the goaltending on different teams or you can compare EVO vs. EVD over time. The sGAA data can be downloaded here.

Perspective:

Going forward further evaluation of the sGAA model is needed. One way to do so, would be to look at repeatability. How repeatable is each component from year to year? This is a very important knowledge to have, when it comes to evaluating the predictive power of the model.

It’s also worth taking a deeper look at the goaltending stats. Can we improve on the GSAx data?

To end this article, I have added tables of the top 20 skaters in GAR, xGAR and sGAA along with the top 20 goalies in GAR. I will just leave the tables uncommented, but it gives you an opportunity to compare GAR/xGAR to the redefined models.

Top 20 skaters: GAR 2019/2020

PlayerGPTOI_ALLGARGAA adjRank GARRank GAA adj
Artemi Panarin69142224.817.612
Elias Pettersson68126023.916.923
Ryan Ellis49115123.118.731
Brayden Point66124821.515.244
Brad Marchand70136220.714.555
Nathan MacKinnon69146519.712.669
Ryan Pulock68152419.313.876
Alex Pietrangelo70168918.313.088
Victor Hedman66158718.113.097
David Pastrnak70132717.811.11014
Roman Josi69177817.811.91111
Charlie McAvoy67155216.811.31213
Max Pacioretty71127116.89.91320
Jaccob Slavin68159116.612.11410
Jack Eichel68150216.59.91521
Auston Matthews701468168.61630
Connor McDavid64139915.910.51716
Cale Makar57119815.811.71812
Bryan Rust55108715.811.11915
Mark Stone65126115.89.62022

Top 20 skaters: xGAR 2019/2020

PlayerGPTOI_ALLxGARxGAA adjRank xGARRank xGAA adj
Auston Matthews70146820.918.011
Brad Marchand70136219.917.322
David Pastrnak70132717.415.133
Elias Pettersson68126017.213.745
Jack Eichel68150216.712.459
Nikita Kucherov68128316.613.566
Nathan MacKinnon69146516.111.9716
Artemi Panarin69142215.712.1813
Jared Spurgeon6214001514.594
Mika Zibanejad57123314.712.21010
Travis Konecny66110914.611.91117
Shea Theodore71157914.312.01215
Dominik Kubalik6897814.212.11311
Oliver Bjorkstrand498791412.5148
Alex Pietrangelo70168913.912.11512
Roman Josi69177813.511.31618
Jake Guentzel3980513.412.11714
Leon Draisaitl71160512.97.71846
Brayden Point66124812.79.21928
Anze Kopitar70147312.78.82030

Top 20 skaters: sGAA 2019/2020

PlayerGPTOI_ALLGARxGARsGAARank sGAA
Brad Marchand70136220.719.915.91
Elias Pettersson68126023.917.215.32
Artemi Panarin69142224.815.714.93
Ryan Ellis49115123.19.113.34
Auston Matthews7014681620.913.35
David Pastrnak70132717.817.413.16
Alex Pietrangelo70168918.313.912.67
Nathan MacKinnon69146519.716.112.38
Brayden Point66124821.512.712.29
Jared Spurgeon621400141511.710
Roman Josi69177817.813.511.611
Victor Hedman66158718.111.311.412
Jack Eichel68150216.516.711.113
Charlie McAvoy67155216.811.810.814
Dougie Hamilton47109413.212.710.815
Adam Fox70132314.912.110.616
Oliver Bjorkstrand4987913.91410.617
Nikita Kucherov68128314.116.610.518
Jaccob Slavin68159116.69.810.019
Cale Makar57119815.810.19.920

Top 20 goalies: GAR 2019/2020

PlayerGPGARGAAGSAxRank GARRank GAARank GSAx
Connor Hellebuyck5826.718.025.1111
Tuukka Rask4119.714.511.1223
Semyon Varlamov4516.39.15.93523
Darcy Kuemper2915.611.311.3432
Ben Bishop4415.18.38.6569
Anton Khudobin3014.410.68.36411
Corey Crawford4014.27.510.4794
Antti Raanta3313.48.18.8878
Jonathan Bernier4612.95.44.391726
Jordan Binnington5012.94.77.9102013
Mackenzie Blackwood4712.63.84.5112425
Philipp Grubauer3612.67.22.3121031
Juuse Saros4012.36.05.3131324
Carter Hart4311.95.68.4141610
Elvis Merzlikins3211.86.82.7151130
Jacob Markstrom4311.83.67.4162515
Mikko Koskinen3811.44.69.617215
Pavel Francouz34115.76.3181518
Jake Allen2410.88.09.01987
Robin Lehner3610.64.17.4202214

All data from http://www.Evolving-Hockey.com

4 thoughts on “Interpretation and redefining of Evolving-Hockey’s GAR and xGAR models.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: