Dynasty, in Theory: on Model Rot

There's a lot of strong dynasty analysis out there, especially when compared to five or ten years ago. But most of it is so dang practical—Player X is undervalued, Player Y's workload is troubling, the market at this position is irrational, and take this specific action to win your league. Dynasty, in Theory is meant as a corrective, offering insights and takeaways into the strategic and structural nature of the game that might not lead to an immediate benefit but which should help us become better players over time.

The Curious Case of Ladd McConkey

Over a decade ago, I noticed that rookie-year performance in this newfangled efficiency stat called "yards per route run" (or YPRR) seemed to be fairly predictive of career outcomes. I tracked it over time, and my conviction in its utility only grew, so in 2021 I started tinkering with a model to quantify the effect.

This model is now perhaps my most popular feature of the year. After the season ends, I enter each rookie WR's performance, and the model assigns a score relative to every other qualifying rookie since 2006. Historically, these scores have been strongly predictive of total career production. (I'll publish this year's version next week, once we have final statistics for 2025.)

In last year's edition, I noted that several extraordinarily strong rookie performances immediately joined the ranks of the best seasons the model had seen, writing: "2024 places a shocking three players into the 'Superstar' tier. McConkey led the pack based solely on yards, but including touchdowns, edged Thomas ahead. Nabers might grade as the 3rd-best rookie in the class, but that still places him as the 10th-best rookie since 2006."

The sophomore campaigns of those three budding superstars were... underwhelming, to say the least. McConkey finished 27th in fantasy points and 37th in points per game. Thomas was even worse, finishing 44th in points (48th in points per game) and trailing former 6th-round pick Parker Washington and midseason acquisition Jakobi Myers on his own team. (Nabers started much more productively, but was injured and only played four games.)

Someone asked me how unlikely it was for two players rated so highly to both underperform at the same time, and I think it's a good question because it gets to fundamental questions about how models work and even what probability is in the first place.

(The heading above should also refer to Thomas, but I like how the way it's written right now reads like the title to an Agatha Christie novel.)

All Models Underperform

Typically, the starting point for any model is finding a relationship that's so strong it seems unlikely to have arisen by chance alone. Rookies with high YPRR totals had such good careers that it seemed implausible that rookie-year YPRR wasn't telling us something important about them as players.

The problem with this, of course, is that lots of relationships that are unlikely to arise by chance alone will... arise by chance alone. (XKCD famously explains this best.) If you look at 20 variables, on average, one of them will have a correlation strong enough that there was only a 1-in-20 chance it could be spurious

Fortunately, there are ways to guard against this. You could, for instance, build a model from one dataset and then test it on another dataset (which wasn't used in the model's construction). If I observe a relationship in 2015, for example, everything that happens after that point serves to either verify or refute the relationship.

This isn't foolproof, either, though. Harstad's razor states that "any time you see any cool, compelling, interesting, powerful, intriguing, or absurd statistic, you should probably just assume it's selection bias until proven otherwise." Just because it's my razor doesn't mean I'm immune.

Imagine, if you will, a million different alternate realities with a million different versions of me all building a million different models. Many of those potential models will be discarded as I test them and realize they don't perform as well as I'd hoped. Only the highest-tuned models will survive.  But we never see the millions of potential models that never came into being, and so we don't realize that everything we do see is merely the top performers.

This is a problem because, as I've demonstrated before in my column on regression to the mean, top performers tend to be both good and lucky to some degree. In that column, I created a toy model that took players with varying "true" performance levels (plus or minus some random weekly variance), simulated a bunch of seasons, and then looked at the leaders in each year. The leaders tended to come from the very best players in terms of underlying performance level... but they also almost always overperformed their "true mean" too (if they didn't, they were outcompeted by a player with a slightly lower true performance level who did outperform.)

The same is true for all these counterfactual models. The models that come into existence (out of all possible models I could have built) are more likely to have some actual underlying signal... but insofar as there was some luck working in their favor in the original sample, they're also likely to perform worse once they're taken out into the wild. This is true even if they've been rigorously validated ahead of time. It's just the nature of the beast.

All Models Decay