This is the ultimate one-stat summary for QBs (that we have so far). I haven't seen anything as well thought out as this for NFL stats. Very well done.
Thanks, Matthew. Unfortunately I was pressed for time today and couldn't go through the exact methodology. I'll do that for the post next week, along with adding something for receiver strength.
This is a great article and once again, you amaze me with your analytical skills. I was wondering how to access the EPA metrics that you cited from NFLFastR. There doesn’t appear to be a clear link to the data on their website and I just wanted to be able to see all the EPA stats from there.
Love the work, and your work in general! A few things have me wondering about some of the numbers, however. Let me preface by saying, yes, I'm a Bills fan; I acknowledge there's this weird defensiveness amongst Bills/Chiefs fans re: all things comparing their QBs; last, I believe Mahomes is the best QB in the world. Perhaps that's more so directed towards any replies, but anyways...
Like another reader commented, the "sacks" adjustment - do things like strength of opposing D Lines, quality of OLs (particularly those on the field at a given time when a sack occurs, to adjust for injuries), WR separation/coverage (I know PFF started tracking "perfectly covered plays" this year), etc, get taken into account? Allen's line has been objectively awful compared to, say Mahomes, and he's faced DLs like Tenn., Bal, KC, Mia, NYJ, & NE. Has Allen created his own pressure, of course, as has Mahomes, but surely Allen's escapability and plays made while avoiding pressure surely can't cause him to actually lose EPA from sacks when adjusted.
Secondly, the drops adjustment...Allen has constantly been let down by drops this year, inc dropped TDs, with his WRs leading the league in drops. How is his adjustment a third of Mahomes, and on the lower side as a whole?
Are things like YAC% vs air yards % accounted for? Are all TD passes treated the same? Or is their an adjustment for a 3 yd jet motion pop pass, or PA shuffle to a backside TE worth the same as 10+ or 20+ yard TDs?
Again, I acknowledge this sounds like salty Bills fan, but I'm just curious if you see any of these things as useful in trying to isolate solely the play of the QB, vs scheme/WRs/quality of opponents, etc. Thanks again man! Great stuff as always.
Here's the thing about Mahomes... he's the best QB on Earth but his stats are also inflated by playing for an offensive mastermind and the best TE in the league. He makes a lot of insane plays that no other QB could make but he gets to play on easy mode more often than just about any QB as well.
Here are the yearly leaders in TD passes thrown behind the LOS:
2022: Mahomes - 7
2021: Mahomes - 6
2020: Mahomes - 7
2019: Rodgers - 6
2018: Mahomes - 8
Mahomes has more than double the next QB since he joined the league! Unless you believe he has some special ability to throw a screen pass (I don't), that's a boatload of TD's that are entirely a function of scheme and receivers. This needs to be accounted for when comparing Mahomes to Allen or any other great QB.
For example, this year Mahomes is averaging 6.3 YPA with 7 TD on passes behind the LOS while Allen is only averaging 4.0 with 1 TD on such passes. That's a significant difference that has almost nothing to do with the QB's themselves. If we removed these gimme passes from every QB's ledger, Allen would jump Mahomes in TD passes and probably in MVP odds as well.
As you make adjustments I would expect the adj epa/play variance to fall relative to the raw epa/play variance. Theoretically we're removing noise from the raw? But in this analysis the variance is the same. I'm wondering if the issue could be that some of these adjustments are correlated. If you're treating them independently in that scenario, you could be piling onto qbs, or over-adjusting, by adding correlated adjustments. I'm terrible with statistics principles, so forgive my layman's explanation. Thanks for the content!
I think they're fairly separate, but it's definitely something to look out for. I have a new version publishing today where I have to control for those issues.
This is amazing work! I've been hoping for years to see QB luck quantified so it's awesome to finally see it. I eagerly anticipate the rest of this series.
I'm a bit confused by sack luck... for example when you subtract 11 points from Brady for sack luck, does that mean he held the ball long enough to take 11 EPA worth of extra sacks but his o-line bailed him out?
The one other variable I'd really like to see quantified is weather, or at least dome vs outdoor. It's undoubtedly easier to put up good EPA in a dome, and guys like Cousins and Goff look better than they should due to playing the majority of their games indoors. Conversely, AFCE and AFCN passers are hurt by playing almost all of their games outdoors and often in less than ideal conditions.
Great points. I'm going to look into the Brady thing. It jumped out to me too. Time-to-throw is in the model, but maybe it gets downplayed too much with other factors added in.
Weather is another component I'm going to add with receiver help next week.
However, I still feel a quarterback like Jimmy G is not adjusted downwards enough. I’m only being half facetious to my favorite Jimmy truther.
Quarterbacks who don’t do much, but end up with completions to Kittle like we just saw (that was Purdy, but coulda been Jimmy) might be impossible to quantify. Maybe we just need to ensure we use the eye test a little. Heresy I know. But im starting to believe it’s the only way to complete this seemingly unquantifiable puzzle.
I thought about an adjustment for YAC over expectation, but I think that can limit too much teams that scheme YAC. Maybe YAC after a broken tackle is better. I'll look into that for the update next week.
It’s beyond the scope what you’re trying to do, so apologies for being tangential. But I really am stymied. I don’t think Purdy is particularly good, yet he might end up being a half a deviation worse than Jimmy. Heck, his performance might be barely different with that team. Meanwhile, if you put the last QB taken in the draft behind the LA Chargers or Minnesota lines and running game, I feel the performance will be several deviations below Herbert or Cousins. I have no idea how that can be quantified properly. Or maybe I am, like I mentioned, off on a tangent.
This is the ultimate one-stat summary for QBs (that we have so far). I haven't seen anything as well thought out as this for NFL stats. Very well done.
Thanks, Matthew. Unfortunately I was pressed for time today and couldn't go through the exact methodology. I'll do that for the post next week, along with adding something for receiver strength.
This is a great article and once again, you amaze me with your analytical skills. I was wondering how to access the EPA metrics that you cited from NFLFastR. There doesn’t appear to be a clear link to the data on their website and I just wanted to be able to see all the EPA stats from there.
You can't see the stats on the site. It's a package for a programming language called R. You can find csv files here https://github.com/nflverse/nflverse-data/releases.
Or go to rbsdm.com and you'll find advanced stat box scores, QB EPA, etc
Awesome stuff Kevin. In addition to adding receiver skill, I would love to see adjustments for play callers. Some ideas to test:
Early down pass rate
ADOT
Play action rate
Pre snap motion rate
And for team rushing success:
Rush success rate
Rush EPA
If you adjust for the above and Jimmy G is still top 5, film twitter might finally have to respect him!
I agree with Mr Conner. This absolutely fantastic stuff!
Love the work, and your work in general! A few things have me wondering about some of the numbers, however. Let me preface by saying, yes, I'm a Bills fan; I acknowledge there's this weird defensiveness amongst Bills/Chiefs fans re: all things comparing their QBs; last, I believe Mahomes is the best QB in the world. Perhaps that's more so directed towards any replies, but anyways...
Like another reader commented, the "sacks" adjustment - do things like strength of opposing D Lines, quality of OLs (particularly those on the field at a given time when a sack occurs, to adjust for injuries), WR separation/coverage (I know PFF started tracking "perfectly covered plays" this year), etc, get taken into account? Allen's line has been objectively awful compared to, say Mahomes, and he's faced DLs like Tenn., Bal, KC, Mia, NYJ, & NE. Has Allen created his own pressure, of course, as has Mahomes, but surely Allen's escapability and plays made while avoiding pressure surely can't cause him to actually lose EPA from sacks when adjusted.
Secondly, the drops adjustment...Allen has constantly been let down by drops this year, inc dropped TDs, with his WRs leading the league in drops. How is his adjustment a third of Mahomes, and on the lower side as a whole?
Are things like YAC% vs air yards % accounted for? Are all TD passes treated the same? Or is their an adjustment for a 3 yd jet motion pop pass, or PA shuffle to a backside TE worth the same as 10+ or 20+ yard TDs?
Again, I acknowledge this sounds like salty Bills fan, but I'm just curious if you see any of these things as useful in trying to isolate solely the play of the QB, vs scheme/WRs/quality of opponents, etc. Thanks again man! Great stuff as always.
Here's the thing about Mahomes... he's the best QB on Earth but his stats are also inflated by playing for an offensive mastermind and the best TE in the league. He makes a lot of insane plays that no other QB could make but he gets to play on easy mode more often than just about any QB as well.
Here are the yearly leaders in TD passes thrown behind the LOS:
2022: Mahomes - 7
2021: Mahomes - 6
2020: Mahomes - 7
2019: Rodgers - 6
2018: Mahomes - 8
Mahomes has more than double the next QB since he joined the league! Unless you believe he has some special ability to throw a screen pass (I don't), that's a boatload of TD's that are entirely a function of scheme and receivers. This needs to be accounted for when comparing Mahomes to Allen or any other great QB.
For example, this year Mahomes is averaging 6.3 YPA with 7 TD on passes behind the LOS while Allen is only averaging 4.0 with 1 TD on such passes. That's a significant difference that has almost nothing to do with the QB's themselves. If we removed these gimme passes from every QB's ledger, Allen would jump Mahomes in TD passes and probably in MVP odds as well.
I'm adding a YAC adjustment in the next iteration publishing later today. That hits Mahomes big-time
As you make adjustments I would expect the adj epa/play variance to fall relative to the raw epa/play variance. Theoretically we're removing noise from the raw? But in this analysis the variance is the same. I'm wondering if the issue could be that some of these adjustments are correlated. If you're treating them independently in that scenario, you could be piling onto qbs, or over-adjusting, by adding correlated adjustments. I'm terrible with statistics principles, so forgive my layman's explanation. Thanks for the content!
I think they're fairly separate, but it's definitely something to look out for. I have a new version publishing today where I have to control for those issues.
This is amazing work! I've been hoping for years to see QB luck quantified so it's awesome to finally see it. I eagerly anticipate the rest of this series.
I'm a bit confused by sack luck... for example when you subtract 11 points from Brady for sack luck, does that mean he held the ball long enough to take 11 EPA worth of extra sacks but his o-line bailed him out?
The one other variable I'd really like to see quantified is weather, or at least dome vs outdoor. It's undoubtedly easier to put up good EPA in a dome, and guys like Cousins and Goff look better than they should due to playing the majority of their games indoors. Conversely, AFCE and AFCN passers are hurt by playing almost all of their games outdoors and often in less than ideal conditions.
Great points. I'm going to look into the Brady thing. It jumped out to me too. Time-to-throw is in the model, but maybe it gets downplayed too much with other factors added in.
Weather is another component I'm going to add with receiver help next week.
Thanks!
Great work as usual. Just top notch.
However, I still feel a quarterback like Jimmy G is not adjusted downwards enough. I’m only being half facetious to my favorite Jimmy truther.
Quarterbacks who don’t do much, but end up with completions to Kittle like we just saw (that was Purdy, but coulda been Jimmy) might be impossible to quantify. Maybe we just need to ensure we use the eye test a little. Heresy I know. But im starting to believe it’s the only way to complete this seemingly unquantifiable puzzle.
I thought about an adjustment for YAC over expectation, but I think that can limit too much teams that scheme YAC. Maybe YAC after a broken tackle is better. I'll look into that for the update next week.
That’s a neat idea.
It’s beyond the scope what you’re trying to do, so apologies for being tangential. But I really am stymied. I don’t think Purdy is particularly good, yet he might end up being a half a deviation worse than Jimmy. Heck, his performance might be barely different with that team. Meanwhile, if you put the last QB taken in the draft behind the LA Chargers or Minnesota lines and running game, I feel the performance will be several deviations below Herbert or Cousins. I have no idea how that can be quantified properly. Or maybe I am, like I mentioned, off on a tangent.
Anyhow, enjoying your writing. Great substack.