This Deepest Data Dive You'll Read On NFL Playoff Penalty Stats All But Confirms The Kansas City Chiefs Are Given Preferential Treatment
Perry Knotts. Getty Images.Get some coffee and buckle up. We're about to take the deepest dive into Kansas City Chiefs / NFL referee data available on the internet. Right here in this blog.
There's been a lot of chatter this week debating if the Chiefs are getting preferential treatment from the refs. And for good reason seeing as how terrible some of the calls were in Arrowhead last week. I've never heard Troy Aikman so disgusted. In a season where the Chiefs benefited seemingly from fate with multiple wins off missed field goals or a toe on a line, etc., a target has been put on their back from Chiefs haters. Add the fact this team is vying to become the first three-peat in NFL history. It's too much for Chiefs haters to take having to think the refs are playing into their success.
But that doesn't mean the haters are wrong.
Quick preface. ESPN came in with some recent history stats to set this mid-week wild fire ablaze.
Then Warren Sharp wrote a piece piggy-backing off these stats. I gave that a read, but felt his dichotomizing penalties simply by one team having more or less than another in a single game left out the grainy details. And penalty yards are weird too because I don't really think refs factor that in the split second decision they have to make a call. Unless you think this is just a full-on codified directive from the NFL to have the Chiefs win. For the record Chiefs fans - that's not me. I'm not there.
But I'll tell you this. Shit doesn't look great.
Before I get to my work, let's lay out a few counter points. What are people backing the Chiefs saying?
There was also a great stat Ryen Russillo read on his podcast. In the past two years, Chiefs opponents have been flagged 48 times in the fourth quarter of one-score games. How many flags were thrown on the Chiefs? 47.
Chiefs haters - "THAT'S LESS… RIGGED!!"
Alright, let's dig in. We'll start with Ben Baldwin's question as to why ESPN began the stat at 2021. Any good data scientist knows that when you see a random time range for a given stat, it means whatever the stat maker was hoping to prove wasn't supported either before or after that time range. Oldest trick in the book. A page I turn to often when cutting a date at just the right spot to find funny 'haha' stats. So, let's answer Ben's question. ESPN cut the years off at 2021 for exactly the reason you thought. Because there was nothing weird going on in the first three years of Pat Mahomes' unparalleled successful start to a career.
33 playoff penalties on Kansas City at Arrowhead during Mahomes' first three years as a starter vs 38 for visiting Chiefs' opponents. They didn't play any road playoff games during this span, but did commit 11 penalties for 120 yards in the 2020 Super Bowl vs Tom Brady's Buccaneers. This compared to just four penalties for 39-yards on Tampa Bay (Put a pin in this. We'll be back to it). All in all though - the Chiefs committed 44 penalties during these three years vs their opponents' 42. So all wrapped up nothing looks off from 2018-2020.
As for the regular season. The Chiefs also committed 150 home penalties during that span in the regular season vs 167 from their opponents. It's less, but not glaringly. Much like their penalties on the road are more. Just not glaringly. Certainly within the realm of randomness.
Then 2021-2024 (so far) happened.
Let's expand on that ESPN stat. They isolated the Chiefs, but I want to see that in context. Let's look at every team's total playoff penalties vs opponents in order of lowest penalty percentage during that span. Disclosure, I filtered out any team who's opponents committed fewer than 25 penalties to remove whacky outliers (Ex: Vikings had five penalties vs 15 for their opponents).
There the Chiefs are with the least percentage of penalties (35.29%). But the sample size is still low for the other teams. Because of course it is. No team in the past few years has played as many playoff games as the Chiefs, making it hard to use any of this for comparison. So here's what we're going to do. Let's broaden this same lookup. Instead of using the conveniently selected 2021-2024 period, let's crush every four-year span since the 1970 merger with a minimum of 50 opponent penalties to account for sample size controlling. Perhaps this is all just a product of randomness. If so, certainly some other teams would have randomly benefited as much too, right?
Well, one other team did. John Elway's coming of age Broncos.
So much going on here. The 2021-2024 Chiefs are on the second-best run of any four-year term of all time in terms of penalty luck. Only John Elway's Broncos were luckier. Put a pin in this too.
What else is going on here? Obviously, I highlighted New England and Kansas City. Sorry Pats fans! There you were enjoying this Chiefs slander research and now you've been brought to trial with them. You'll notice since I ran this code on every single four-year span, there are overlapping year ranges. The Patriots have that isolated 2003-2006 span and then from 2013 they have a top-span in four consecutive years that gets INCREASINGLY luckier. This is what we call… a pattern.
Interesting.
Now the Chiefs. Same idea. Look at the spans that start in 2017. 2018. 2019. 2020. All somehow slightly luckier than the prior span as time went on. This is what we call… a trend. A trend of star GOAT conversation quarterbacks being on the right side of luck and having that luck progress over time. Maybe Elway was considered that rising #1 overall pick that hit his prime. I was in diapers so someone else can tell me if that confirms or denies the idea.
Side note - shout out Dallas for winning three Super Bowls during 1992-1995 despite having the third-most penalized span of the entire list. Third to… well.. themselves from back in the late-70s. Which was second to… themselves again. It is a little curious to see "America's team" was treated like a redheaded step-child, but three of the four rows contain overlapping seasons which explains some of it.
Let's get back to Russillo's mentioned stat that sure seems to do a good job rebuking the idea of the Chiefs being favored by the referees. How could any of this be if the Chiefs are being penalized on par with their opponents during one score games in the fourth quarter? There's no way refs can be playing favorites if they're treating both teams equally in the most pivotal parts of the game… Right?
Turns out when you mix playoffs with regular season, things get cloudy. At least for the Chiefs. I ran the numbers for the regular season during the Chiefs 2021-2024 span ESPN highlighted to compare and the Chiefs were penalized pretty much the same as their opponents (388 on Chiefs vs 394 on opponents). And that's what makes the fourth quarter / one-score game stat misleading. Most of that is from the regular season. And there's a statistically significant difference in the ratio of Chiefs penalties vs opponents from the regular season to the post-season with a slight to moderate effect size (d = .29 for the real nerds out there):
Chiefs regular season penalties - 388 vs 394
Chiefs post-season penalties - 36 vs 66
I'll be honest. I'm not exactly sure what to make of this significance testing result. We might just be p-value fishing since we pre-emptively started with one of the luckiest post-season teams and tested this against their regular season from there. And after doing the same thing with Elway's Broncos and the 2016-2019 Patriots, no significance was found with by the t-tests.
Let's unpeel the onion a little more. Yes - there's some onion left. Many penalties are so cut and dry that refs would look really stupid not calling them. False starts. Unabated to the quarterback. You get the gist. But what if we focus on just the major subjective calls? Here are the ones I considered.
Defensive Holding
Defensive Pass Interference
Face Mask
Offensive Holding
Offensive Pass Interference
Roughing the Kicker
Roughing the Passer
Unnecessary Roughness
Unsportsmanlike Conduct
The data doesn't exist to re-run the numbers on just these penalty types going back to the merger, but I can with the 2021-2024 selection. If the Chiefs being that much of a beneficiary on penalties was still attributable to randomness, then it logically follows that there should also be randomness in how many subjective penalties are called on them vs their opponents.
Welp…
Shout out Minnesota for only having two subjective penalties out of ten and still getting nowhere in the post-season. The most Vikings stat you could find. Obviously, the sample sizes here are small for pretty much every team except Kansas City. But only 16 out of 50 subjective penalties is some smoke if I've ever seen it. Less than half of their opponents. So to be on the side of "this is just random", you have to be OK with the fact that the Chiefs just so happen to have had the second-best playoff penalty call luck in the history of the modern game AND when you just look at the subjective calls during that time frame, their opponents get more than twice the amount called on them.
I'm sorry Chiefs fans. You're kind of dead to rights here. The forensic statistics are clear - your team is getting preferential treatment. How many peels of the onion will you need to see this?
Let's pull the receipts on all 50 of these penalties in order of how much time was remaining in the game. Doing so tells a screwy story.
Row two shows the penalty referenced in the counter tweet from the start of this blog. A call from the first minute of overtime on a third down. But that's one of just three total subjective penalties on the Chiefs from the 4th quarter through overtime on this list. That including a roughing the passer on Big Broken Ben when the Chiefs were up 42-14. Even counting that - 3/14 4th quarter subjective penalties were called on Chiefs vs their opponents. And I count nine calls on Chiefs opponents in one-score games.
Little bit of a different story than Russillo's stat now, huh? But it's not just the end of games, it seems the refs also let the Chiefs off to a hot start by only calling 1/7 subjective penalties in 1st quarters on Kansas City. This one I could believe was a product of randomness if not for the substantial circumstantial evidence around it that we've already covered. And I don't buy "the Chiefs are more disciplined" theory in the post-season. Because I could see that angle being thrown out there to explain hot starts and finishing strong. But they're not playing teams like the Panthers at this point. And even when they did, the calls were pretty even anyway as we've learned.
Alright - so what is ACTUALLY going on here?
I think this is some sort of GOAT bias. It's well researched that home team's get better calls on average due to the refs subconsciously not wanting to err at those in the stands around them. That's why home teams typically get a 2.5 to 3 point bump on spreads. But this GOAT bias might be more powerful. Based on all this research I've presented, it seems that only Brady and Mahomes have reached escape velocity in earning it over the course of time, since even Elway wasn't able to keep it up. 8-8 in 1988 probably cut John down a peg along with a 5-11 1990 season.
There's one huge counterpoint to this theory though. Joe Montana. The OG GOAT. I looked into his teams specifically and during his run in the 80s he never had a single four-year span with fewer penalties in the post-season than his opponents. Not one. Maybe some old head out there always thought the league was after Joe. Well guess what. Today's your day.
Even still - that was 40 years ago. Maybe the social media age is to blame for subliminally telling refs everywhere they go that Tom Brady is the GOAT until it was Pat's turn. The increasing nature of both the Brady and Mahomes led teams increase in penalty luck speaks to something going on here. Not a conspiracy. I don't think. Just a bias. But a bad one. One that might even permeate away from penalties a GOAT is directly involved in to anyone else on his team.
All you Chiefs fans really need to here is admit somethings up. That's it. It's not your fault. And despite how horrendous the calls were vs Houston last week in favor of the Chiefs, they were clearly the better team. But honestly… just admit it already. Part of this is probably on Mahomes learning how he can play the refs as demonstrated by finding out where the line is with that LeBron-esque flop on the sideline. But it's still on the league to do something about it.
Anyway - this was a fun one. It makes the Sunday matchup even that much more exciting. I can't wait to watch every snap three inches in from of my TV with eyes looking at everything in full John Nash mode. Not only are we getting the best two quarterbacks of the past several years, but it's layered with this penalty controversy everyone is talking about.
Almost like it was all…
Nope. Not saying it!!!