Evaluating the projection model

Abstract:

The goal here is to compare my projection model (read more here and here) to other models out there. All of my projections below are made well after the games were played, so the true test of the model will be its performance this upcoming season.

The model is the same for all ten seasons (from 10/11 to 19/20) and it’s based solely on data from the previous 3 seasons. I have refined the model to give the lowest average error and the correlation (R-squared).

I’m using Dom Luszcyszyn’s end of the year reviews to compare my model to its peers.

16/17:

Let’s start off by looking at the 2016/2017 season. Here’s how other predictions went that season. The picture is taken from Dom Luszcyszyn’s prediction review which can be found here.

And here’s how my model would have projected the season:

TeamSeasonpPointsPointsDifference
ANA16-1788.8210516.18
ARI16-1786.977016.97
BOS16-1796.22951.22
BUF16-1779.24781.24
CAR16-1781.50875.50
CBJ16-1791.3110816.69
CGY16-1793.39940.61
CHI16-1793.3310915.67
COL16-1786.414838.41
DAL16-1787.11798.11
DET16-1785.02796.02
EDM16-1789.1910313.81
FLA16-1795.218114.21
L.A16-1793.04867.04
MIN16-1794.3310611.67
MTL16-1792.5810310.42
N.J16-1783.957013.95
NSH16-17103.07949.07
NYI16-1793.97940.03
NYR16-1795.331026.67
OTT16-1789.14988.86
PHI16-1791.30883.30
PIT16-17101.841119.16
S.J16-17104.68995.68
STL16-1795.57993.43
T.B16-1798.78944.78
TOR16-1777.579517.43
VAN16-1782.876913.87
WPG16-1790.19873.19
WSH16-17106.0711811.93
Overall16-179.84

The one team that jumps at you right away is Colorado. They ended up with just 48 points which was historically bad. I don’t think anyone could have foreseen that kind of season. Dom’s model was off by 37 points on Colorado.

Other than Colorado the model did okay, and it would have been the best prediction out there. Obviously, there’s no fame and glory in predicting results 4 years after they happened.

17/18:

The following season would turn out to be the toughest to predict. Here’s how the other models performed, and Dom’s review can be found here, if you have a subscription to The Athletic.

Here’s how my model would have done. It would have been the second-best prediction, but way behind Cosica’s prediction.

TeamSeasonpPointsPointsDifference
ANA17-1890.5910110.41
ARI17-1886.517016.51
BOS17-1895.0511216.95
BUF17-1883.106221.10
CAR17-1896.628313.62
CBJ17-1895.80971.20
CGY17-1892.11848.11
CHI17-1886.927610.92
COL17-1880.309514.70
DAL17-1892.02920.02
DET17-1885.517312.51
EDM17-1892.277814.27
FLA17-1885.979610.03
L.A17-1890.45987.55
MIN17-18100.541010.46
MTL17-1893.467122.46
N.J17-1885.289711.72
NSH17-1899.9211717.08
NYI17-1896.588016.58
NYR17-1894.587717.58
OTT17-1888.606721.60
PHI17-1889.12988.88
PIT17-18100.891000.89
S.J17-18100.261000.26
STL17-1891.50942.50
T.B17-1895.1311317.87
TOR17-1890.9210514.08
VAN17-1878.59735.59
VGK17-1884.1410924.86
WPG17-1896.9011417.10
WSH17-1899.991055.01
Overall17-1811.69

It’s probably fair to say that Vegas surprised everyone. My model would have been pretty high on them, but still way off. It’s also interesting that the model had no clear-cut contenders – MIN, NSH, PIT, S.J and WSH were all projected to get around 100 points.

In the end the model was wrong about most teams, but at least it was less wrong than most other predictions.

18/19:

The picture below here shows the performance of other predictions, and Dom’s review can be found here.

My projection model would have been first by a tiny margin. A lot of the predictions had an error around 8 points.

TeamSeasonpPointsPointsDifference
ANA18-1994.258014.25
ARI18-1987.73861.73
BOS18-1996.4110710.59
BUF18-1982.00766.00
CAR18-1999.90990.90
CBJ18-1999.16981.16
CGY18-1991.2310715.77
CHI18-1980.47843.53
COL18-1988.36901.64
DAL18-1990.09932.91
DET18-1976.20742.20
EDM18-1990.537911.53
FLA18-1990.24864.24
L.A18-1986.707115.70
MIN18-1999.068316.06
MTL18-1986.32969.68
N.J18-1988.447216.44
NSH18-19105.001005.00
NYI18-1986.6410316.36
NYR18-1982.35784.35
OTT18-1978.156414.15
PHI18-1996.938214.93
PIT18-1998.111001.89
S.J18-1998.021012.98
STL18-1994.65994.35
T.B18-19102.5512825.45
TOR18-1997.681002.32
VAN18-1980.63810.37
VGK18-1994.98931.98
WPG18-19100.45991.45
WSH18-1996.411047.59
Overall18-197.66

My model gave Tampa Bay the second highest point projection, but it was still way off. Calgary and NY Islanders were the two positive surprises. Overall the model did pretty good, but you would have liked it to be lower on Anaheim and higher on Boston, since that seemed predictable.

19/20:

Let’s jump to the most current season. The review can be found here, if you have a subscription to the Athletic. All projections are prorated to 82 games.

Again, my model would have been first and by a decent margin. Overall, it was a fairly predictable season and most of the predictions were quite good.

TeamSeasonGPpPointsPointsDifference
ANA19-207186.7077.389.32
ARI19-207083.3386.693.36
BOS19-207097.09117.1420.05
BUF19-206976.5780.814.24
CAR19-2068105.2997.687.62
CBJ19-207096.6594.891.77
CGY19-207092.2492.540.30
CHI19-207088.7284.344.38
COL19-207091.24107.7716.53
DAL19-206993.2997.454.16
DET19-207168.3745.0423.33
EDM19-207192.1395.863.73
FLA19-2069100.4592.707.76
L.A19-207075.5874.970.61
MIN19-206993.0491.511.53
MTL19-207186.0882.004.08
N.J19-206988.7180.817.90
NSH19-2069100.5692.707.86
NYI19-206886.4696.4710.01
NYR19-207082.9092.549.64
OTT19-207173.2771.611.67
PHI19-206993.38105.7712.39
PIT19-206996.28102.205.92
S.J19-207090.0873.8016.28
STL19-2071102.52108.566.04
T.B19-2070111.93107.774.16
TOR19-2070104.4494.899.56
VAN19-206985.6592.707.04
VGK19-207199.6999.320.37
WPG19-207192.1192.390.29
WSH19-2069104.86106.962.10
Overall19-206.90

There were a few surprises though. San Jose being this bad probably came as a shock to most, and Detroit ended up 10 points below replacement level. I don’t think either team was this bad, but sometimes losses lead to more losses. It can be a vicious circle.

The model was too low on Colorado and Boston. Not just compared to the results, but also compared to consensus thinking. Most were bullish on Colorado before the start of the season – My model wasn’t.

Comparison with Dom’s model:

It’s also interesting to compare my projections with those from Dom’s model. The table below shows my projection (pPoints), Dom’s projection (Dom) and the difference between the two from the 2019/2020 season:

TeamPointspPointsDomDifference
FLA92.7100.591.68.9
EDM95.992.183.98.2
WSH107.0104.996.98.0
CBJ94.996.790.66.1
CAR97.7105.3100.15.2
WPG92.492.188.23.9
T.B107.8111.9109.42.5
PHI105.893.491.42.0
OTT71.673.371.51.8
STL108.6102.5100.81.7
CHI84.388.787.11.6
VAN92.785.784.61.1
VGK99.399.799.50.2
MIN91.593.092.90.1
L.A75.075.675.50.1
NSH92.7100.6101.1-0.5
TOR94.9104.4105.6-1.2
DAL97.493.395.6-2.3
NYI96.586.589.1-2.6
NYR92.582.985.6-2.7
ARI86.783.386.2-2.9
DET45.068.471.4-3.0
PIT102.296.399.4-3.1
COL107.891.294.4-3.2
N.J80.888.792.0-3.3
ANA77.486.790.0-3.3
CGY92.592.296.1-3.9
MTL82.086.190.1-4.0
BUF80.876.681.1-4.5
S.J73.890.195.2-5.1
BOS117.197.1104.1-7.0
Average91.591.691.63.4

On average the two models are 3.4 points apart, so there is some difference. Some of that is probably because of goaltending. In my current model I expect goaltending to regress heavily towards average. For the most part it’s a good assumption, but it means that a team like Boston gets undervalued. They consistently get good/great goaltending, but the model expect them to regress every year.

Explaining the differences between the two models would require a very thorough analysis, so for now I will just leave it as it is.

Notes:

The observant reader might have noticed a difference between Dom’s projections in the previous article and this one. That’s because I used his projections from his team previews last time, but those were made well before the season started. The projections in this article are from opening night.

Conclusion:

The projection model seems to predict results quite well, but the true testimony of the model will come next season. It will be interesting to see, how well it predicts future results – both season results and single game results.

The model definitely still needs some work. I would like the goaltender projections to work better, so I could put more weight on them. I would also like to add an age curve to each player, so the age adjustment isn’t done on the team level.

I used articles from www.theathletic.com in this piece.

One thought on “Evaluating the projection model

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: