This Week: Data, Player Analysis, Evaluation, and Scouting
This week's theme is completely different from the usual weekly fare. It's a multifaceted commentary:
- How the public and the league mistrack and misuse data.
- Where the application of data to analyze players can be far more subjective than portrayed.
- The shortcomings of "if it can't be measured, it must not exist" as an analytics-oriented refrain.
- Where and why scouting remains a broken process in the NFL.
Much of what I'm sharing are excerpts of private conversations I've had over the years with a former league employee with a multifaceted background.
A specialist who Jene Bramel once remarked: "He could legitimately call out the 'B.S.' on scouts, analytics professionals, and injury professionals based on the depth and scope of his training, expertise, and experience."
Here we go...
1. Completion Percentage Is An Overwrought Analysis Point
Caleb Williams is QB9 in fantasy entering Monday night. Since Week 8, Williams is QB4 with only Josh Allen, Trevor Lawrence, and Matthew Stafford ahead of him.
Williams has more passing yardage than Allen and Lawrence during this span, and he has the same number of passing touchdowns as Allen. He's second on this shortlist with 258 rushing yards to Allen's 298.
Williams has also thrown half of the interceptions (3) that Allen and Lawrence have thrown, respectively. What has put Allen (9) and Lawrence (5) ahead of everyone else is their combined 14 rushing touchdowns during this span.
Williams has a 57.8 completion rate this season. It's often the first thing Williams' critics raise as a red flag about his ability.
It's also a limited analysis. Completion percentage is more of a team stat than an individual data point, but it's used too often to praise or denigrate an individual player. In this sense, completion percentage serves the same ill-informed purpose as yards per carry for running backs.
Completion percentage doesn't account for throwaways and dropped passes. It doesn't account for pressure or the time a quarterback has to read and manipulate the field.
Completion percentage had a dim view of Josh Allen and Lamar Jackson in college and/or in their early years as professionals. Completion percentage is the favorite tool of your know-it-all uncle who lives in grandma's basement, wears the same clothing he had in 1987, and you can't leave alone with guests for more than five minutes.
Completion percentage is not a one-size-fits-all standard from offense to offense or player to player. Ben Johnson told the media this summer that it would be great if Caleb Williams could deliver a 70 percent completion rate.
This rate was the high-water mark for Jared Goff during his third year in Johnson's offense after several years starting in the NFL. The media made it a point of comparison for a second-year player who endured one of the worst organizational messes that a rookie could endure the year before.
The media made the rate a pass-fail metric for Williams' progress. Williams is in one of the most detailed offenses in the league after surviving 2024's poor offensive line, no veteran QB in the room, no coaches working with him, and a scheme that didn't tie enough drops to routes.
Instead of giving this context credence, the public flocked to gossip columns and engaged in character assassination.
My advice: If you want a good indicator of quarterback accuracy, you'll want analysts who chart the games and at least give you some context that informs you whether the completion percentage is missing important parts of the story.
This was the case with Allen, Jackson, and now, Williams. When you read quarterback evaluations, buyer beware of takes that harp on completion percentage.
2. While We're At It: Yards After Contact Is Overwrought
I've been bringing it up for a few years, but it's still worth mentioning as a reminder for those of you seeking ways to discern worthwhile analysis from projectile data spouting. Yards After Contact seems like a data point tracking power. It sounds good until you dig into how it groups track data.
Yards After Contact can include a cornerback diving for a running back and slapping the runner's thigh pad at the line of scrimmage on the way to a 75-yard gain. It can also include a middle linebacker shooting the A-Gap into the backfield and hitting the RB squarely in the chest after his first step through the exchange with the quarterback, and the RB earns three yards.
The thigh pad slap from a cornerback's hand carries the same weight as a direct hit to the chest from a middle linebacker. It implies that the 75-yard gain is a better display of power than the 3-yard gain.
Data is objective. Yeah, and my dog is a giraffe.
Context matters, or else you will find yourself mistaking a dog for a giraffe.
3. Data: How It Is Tracked Matters
The first two points are examples for this segment. Data can be objective, but a lot of football data isn't as objective as it appears.
Who was the more accurate quarterback in college, Lamar Jackson or Sam Darnold? Completion percentage indicates it's Sam Darnold, but he's your uncle you can't trust around people.
It might well still have been Darnold, but from what I remember from my game charting, what mattered most was whether each quarterback had enough accuracy to project success in this area as a future NFL starter.
They did, and there's a point about "out-evaluating" that I'll get to later. For now, let's stick to why game charting is a better method: You can use buckets for different contexts of accuracy. Here are some examples.
- Pinpoint Accuracy: Targets thrown precisely where the receiver must be to catch the ball with minimum effort.
- Catchable Accuracy: Targets thrown where the receiver must make an effort beyond the demands of navigating the route and position of coverage to catch the ball.
- Throwaways: Targets thrown with the purpose of ending the play.
- Tipped Passes: You could split this into buckets that are forced errors and unforced errors. In other words, tipped passes that the quarterback could have prevented and those that he couldn't.
- Accuracy by Field Range: After all, a QB's accuracy in an offense that's heavily reliant on short-range targets is apples compared to the oranges of a QB's accuracy in an offense heavily reliant on intermediate and deep targets that draw a lot more man coverage.
It's understandable that people want simple answers, but those who embrace context are more likely to succeed because, at least in part, they aren't overreacting to limited information or writing off good solutions in the process.
4. New Measurement Data and 4th Down Zealots
As you see, how we track data is critical to its value. Another great example is fourth-down data and how media analyzes coaching decisions with it.
This segment and most of the others you'll read to the end will include commentary from years of conversations I've had with a source who has significant league experience -- the person I mentioned at the beginning of this feature.
I'm describing this anonymous person in greater detail to underscore why his statements have value, and why I prefer to keep him anonymous.
- He has worked directly with NFL GMs in a variety of roles pertaining to analytics and scouting.
- He has been a scout for multiple teams.
- He worked with multiple teams at the same time, and there are scouts, coaches, and GMs who only know of him in one of the several roles and jobs that he's had.
- He had specific NDA agreements with multiple teams.
As I mentioned before, Jene Bramel and I have both met with this individual in person. I have also used him to vet writers for my site as well as advise others on growing in the analytics and scouting fields.
Here are his thoughts on fourth-down data analysis and the changes that are going to be made public.
"It will be fun to watch the new measurement systems data's impact on the fourth-down analysis in the media space. Now, they'll have a precise measurement of inches versus the old way -- a blurred and useless fourth-and-one. Reality will set in fast that 48 inches to go isn't remotely the same as 12 inches or less.
Every model will eventually be returned and retrained, but smart teams have already made these changes years ago. I know of at least two others in the NFL analytics community who used this 'less than 1 yard to go' bucket in their Win Probability-Decision Models as far back as 10 years ago."
My take: We'll eventually see public analysts walking back some of their previous outrage over past coaching decisions now that their data has caught up with the league.