Valuing Draft Picks Based on Career AFL Ratings Output: Addressing LLM Criticisms
/INTRODUCTION
Very much in the spirit of where we’re at as a species right now, I presented that blog post to Claude and asked for its feedback. It was, as is the natural inclination of the modern LLM, quite complimentary in its initial comments, but raised some very pertinent criticisms (which I’m sure a number of you also identified) that we’ll go through in this blog.
CRITICISMS
Claude’s critiques of the original analysis ask two different questions: are we measuring the right kind of value, and are we estimating that value reliably? The first two criticisms challenge the definition of value and the others challenge the data, interpretation or presentation of results.
A change of sample as well as a change of question
Before getting onto the new analyses that address these criticisms it’s important to point out that they are not simply the original analysis with some new calculations. The original approach combined historical data to estimate expected Ratings output over an entire career whereas the main revised analysis instead examines 566 first-entry National Draft selections from 2011 to 2018, including 68 players who never debuted, over their first eight post-draft seasons.
We chose these cohorts because their post-draft careers begin within the period where AFL Player Ratings were available, and all players have a possible eight seasons of follow-up by the end of 2026. This lets us assess a fixed period without projecting the remainder of unfinished careers. The trade-off is that we end up using a narrower sample, whose results may not generalise to every draft era.
Consequently, differences between the original and revised headline curves combine changes in sample, observation period and modelling method, as well as - as we’ll see - changes in the definition of value. In some cases, to distinguish these effects, we do also compare raw and replacement-adjusted outputs within the same sample and retain separate results using the original career model.
1. Value over replacement: how much better is a drafted player than a logical alternative player, not than no player at all?
The original analysis looked at the ratio of career Rating Points generated by players drafted at Pick X versus players drafted as Pick 1, which implicitly assumes that relative value is best assessed by considering only those two players. But, a club without either Pick would still field somebody and so, for a recruitment decision, the relevant benefit comparison might therefore instead be the ratio of each Pick's additional output above that alternative, not the ratio of their total ratings output.
So, for example, a player averaging nine points per game contributes nine points of raw output, but if an available replacement would average seven points, the incremental contribution is only two, not nine. Subtracting the same baseline from all players changes the relative value of players from different Picks substantially.
The new analysis measures these effects in three ways:
It, firstly, estimates a benchmark average Rating from first-entry rookies of 7.56 points per senior appearance, based on 154 players and 2,920 games in their first three post-entry seasons.
Given that anchoring estimate, it then determines the effects of setting fixed replacement levels from seven to nine points per game.
It also tests setting replacement levels equal to 70% and 75% of the fitted Pick 1 player's average rating.
For every selection, the calculation subtracts the assumed replacement output from the player's actual results. Non-debutants contribute zero and negative contributions are retained.
This adjustment makes a very large difference to the Pick versus relative value curve. Within the same eight-season model (more on why this is chosen later) and using 7.56 Rating points per game as the replacement value, Pick 20 falls from an index of 52.7 on raw output to 26.2 above the rookie benchmark, with Pick 1 normalized to 100 in each case. At a replacement level of seven Rating points per game, Pick 20 has an index of 32.2; at eight Rating points per game it has an index of 19.7.
Assessment: the critique is strongly supported. Much of the original disagreement with the AFL DVI depended on counting output that an alternative player might also provide.
One thing that is interesting to note is that there is no discussion in the original DVI document about whether the salaries were measured relative to those of rookies or in absolute terms.
The unresolved question is what number best represents the Ratings output for a genuinely available alternative. It describes the average value for rookies who reached senior football, but not the guaranteed output that can be obtained without recruitment costs or risk.
2. Discounting and control years: when does the club receive the benefit?
The original career total treated a Rating point earned fifteen years after drafting as equivalent to a point earned in a player’s second season. That’s because it was looking to value a player’s entire career. The critique argues that, because the drafting club is less certain about receiving them, distant returns may matter less and should therefore be discounted or ignored.
To address this criticism the new analyses:
count only the first six or eight seasons of Ratings point accumulation after drafting and discount Ratings generated in later seasons at 0% and, say, 8% or 10% annually
count only appearances for the drafting club for up to eight seasons
The six- and eight-season comparisons use the same 566 player selections from 2011-2018, including 68 non-debutants. Every selection has eight possible seasons of follow-up and the clock starts after selection, so a delayed debut consumes part of the window.
Using the rookie replacement benchmark of 7.56 produces results as per the following table:
If we, instead, count only output for the drafting club we get an eight-season estimated index for Pick 20 of 21.3.
Assessment: the window of seasons to consider and the rate discounting of seasons thereafter is important, but its additional effect is modest, especially compared to the replacement adjustment in the first analysis.
There are two important qualifications on this analysis. Six or eight seasons is an arbitrary choice, not based on the average or typical length of contractual control. Also, the drafting-club calculation excludes trade compensation, so it does not measure the club's complete return from a player.
As we might expect the average and typical tenure of a player at his first club after drafting tends to shorten as his Pick number increases.
3. Selection effects: do early picks have longer careers partly because clubs give them more chances?
The original decomposition of career Rating points accumulation showed that career length explained much of the difference between the relative estimated value of Picks. That does not establish why careers differ.
An early pick might play more games because he is better, because the club persists with him longer (perhaps falling victim to the sunk cost fallacy), because his team has fewer competing players, or some combination of these factors. The critique is that "longer career" should not automatically be interpreted as "more underlying talent".
The new analyses to assess components of this criticism were as follows.
Control for club performance and club identity
This addresses a particular alternative explanation: early picks often join weaker clubs (determined using wins, draws and percentage in the year of selection), where there may be fewer established players competing for selection.
The club-performance control asks whether the relationship between pick position and output changes after allowing for how strong the drafting club was. Club fixed effects allow each club its own baseline, accounting for persistent differences between clubs.
Pick 20's eight-season adjusted value moved from 26.2 to 25.8 with the performance control, and to 24.9 with club fixed effects. Those measured club differences therefore explain relatively little of the fitted result.
We should note that this is not a direct test of preferential treatment for early picks as:
The outcome is cumulative replacement-adjusted output, not selection opportunities
Neither control measures the individual player’s underlying ability
Neither control captures competition for a particular position, injuries to teammates or changing selection policies.
Consequently, these results say little about whether a club gives its early picks more patience than its later picks. That could occur within every club and remain entirely consistent with these findings.
Examine early career endings
This establishes whether early picks are much less likely to disappear from senior football quickly.
It found that, among the 80 selections at picks 1 to 10, all players debuted and none made their final observed senior appearance within three post-draft seasons. Among picks 41 to 60, 24 of 147 never debuted and another 27 had their final appearance within three seasons.
However, the same pattern would be expected if early picks are simply better players. It is also expected if clubs give them more chances. The counts cannot distinguish those explanations because they contain no adjustment for ability or performance.
There are two further limitations:
Never debuting and ending a senior career early are different outcomes, although both concern access to senior football.
A final observed senior appearance is not a delisting date. A player can remain listed without playing.
The appropriate interpretation is therefore that early picks have much lower rates of non-debut and early cessation of senior appearances. This identifies a pattern that preferential opportunity could help explain, but does not demonstrate preferential opportunity.
Predict next-season games after accounting for previous games and ratings
This is the most relevant test because it asks whether draft position contains information about subsequent opportunity after controlling for measured previous performance.
Details of the model are shown below.
Conceptually, it compares players with similar previous-season game counts and average ratings, while also accounting for the other included variables. It then asks whether their draft positions predict how many games they will play in the following season.
The fitted model implies that players from Pick 1 play approximately 3.6 more games (with any club) in the next season compared to players from Pick 20, holding those measured characteristics fixed.
That number is determined as described below.
This result is consistent with greater patience for early picks, but we should note that equal measured performance is not equal underlying ability. Consider that:
Clubs observe training, fitness, tactical execution and development that Rating Points do not capture.
A player’s previous ratings may be noisy, especially after few appearances.
Ratings reflect role, playing time and opportunity as well as ability.
Clubs may reasonably expect different future improvement from two players with similar current output.
Draft position itself can contain information about latent talent that remains useful after one season of ratings.
Previous games and ratings also already reflect earlier selection decisions. Controlling for them cannot cleanly separate the effects of ability and opportunity.
The defensible conclusion from the analysis is that early draft position predicts additional subsequent games even after accounting for measured prior games and ratings. This is consistent with preferential opportunity, but could also reflect ability, potential or circumstances not captured by the model.
So, to broadly summarise the three analyses, we investigated three possible explanations for early picks’ longer careers. Club controls showed that controlling for measured differences between drafting clubs makes little difference to the estimated relative value of Picks. Early-career counts confirmed that early picks are less likely to disappear quickly, but did not distinguish talent from patience. A subsequent-games model found that early picks receive more games than their measured prior performance alone would predict. That finding is consistent with additional opportunity, although it does not establish preferential treatment independent of underlying ability.
4. The 2012 window: were older players included only if they survived until ratings became available?
The concern here is straightforward in nature: if we study older careers of players who debuted in a particular season only through players still active in 2012, we miss those who retired earlier. That would artificially increase the estimates of their average career length.
This was not, however, an issue for our original analysis because:
All included careers begin at career game one
There were 683 pre-2012 debutants
294 of them finished playing before 2012 and remain in the dataset
Assessment: the specific survivor-only criticism does not apply to the original career-length estimation.
However, there is a separate ratings problem: older players' ratings are observed only when they played from 2012 onward. Their earlier performance is unavailable in the data.
To assess the impact of that issue, the career models were refitted using modern debutants only. There were two variants:
a Modern debutants: ratings only estimate, for which we use modern debutants to estimate Rating Points per game, but retain the broader 2000–2026 debut cohorts to estimate expected career length.
a Modern debutants: ratings and career length estimate, for which we use only modern debutants for both models.
Both retain the same original estimate of the probability of ever debuting and the same replacement benchmark Ratings per game.
The first analysis checks whether using older players’ partially observed ratings affects the results and the second checks what happens when older careers are removed entirely from the longevity model. The second is not necessarily more reliable becasue many modern debutants are still playing, so estimating their full careers requires more extrapolation.
The results of the analysis for Pick 20 appear in the table below.
The change from an index 32.4 to 40.4 shows that the relative valuation is sensitive to which careers inform the longevity estimate.
We should note that the primary six- and eight-season analysis avoids the problems discussed here by using only fully observed windows beginning after ratings became available.
5. Inter-year Draft comparability: does "Pick 20" represent the same opportunity across Draft years?
Draft lists can include players whom most clubs could not freely recruit, such as matched academy or father-son selections. Consequently, recorded Pick 20 might represent a different level of relative ability across different drafts.
In an attempt to address this a further analysis attempted to construct an open-selection rank from historical signing data. It excluded matched restricted-access selections, rookie promotions and passed selections from that ranking. Explicitly unmatched bids remained open.
This was quite the process, the specific details of which are as follows.
What data were used?
The source was the Draftguru annual draft tables (which were also used in the original analysis). These contain fields including:
Draft year.
Draft type, such as National or Rookie.
Actual selection number—the table’s
#column.Player and selecting club.
A Signing annotation describing particular recruitment arrangements.
The 2018 Draftguru table, for example, has separate columns for selection number, club, signing and player.
The construction code read the tables for 2000 to 2018. An eight-season valuation analysis then used the relevant 2011 to 2018 selections, because those cohorts have complete eight-season follow-up.
The classification was based on recruitment annotations, not subsequent career length, ratings or other measures of career success.
How did the classification work?
The code classified a National Draft selection as restricted when its Signing text contained one of:
Academy
Father-Son
Scholarship
Zone
It made an exception when the annotation explicitly contained Not Matched. Those selections remained in the open category.
A blank annotation was effectively treated as an ordinary open selection.
It’s important to make clear that the code inferred restricted access from the label. It did not independently verify that a bid had been lodged and matched in every case. Indeed, the labels cover different historical mechanisms, some of which were not contemporary bid-matching arrangements. “Selections annotated as restricted-access recruitment” is possibly more accurate than calling every excluded record a “matched bid.”
How was the new rank calculated?
Within each draft year, the code:
Sorted National Draft selections by their recorded pick number
Skipped records classified as restricted according to the rules just described
Numbered the remaining selections consecutively
Assigned those selections ranks 1, 2, 3, and so on.
Excluded players received no open-selection rank.
A concrete example from the cached 2018 draft illustrates the process:
Continuing that count:
Riley Collier-Dawkins at recorded Pick 20 becomes open rank 17
Jez McLennan at recorded Pick 23 becomes open rank 20
The goal is to distinguish between the twentieth recorded selection and the twentieth selection remaining after removing the annotated restricted-access selections.
Why retain explicitly unmatched bids?
The intended distinction was to be between a player recruited through the annotated preferential-access arrangement and one selected without that arrangement being exercised.
For example, the cached annotation for Irving Mosquito in 2018 reads:
Academy (NG)(Hawthorn - Not Matched)
The code therefore retained him. His recorded Pick 38 became open rank 33.
This remains a classification based on the recorded outcome. It does not establish that the player’s availability was equally certain to every club before the draft.
What about rookie promotions and passed selections?
The code included rules to exclude records whose signing annotation contained Rookie, or whose player field was blank or began with pass. It also required a numeric National Draft selection number.
However, none of its retained numeric National Draft records were additionally excluded by the rookie/passed-selection rules. Across the 2000 to 2018 Drafts, the actual classification was:
1,270 records retained as open
102 records excluded through the restricted-access annotation rule
Which players contributed to the rank, and which contributed to the valuation?
The rank was constructed from the available National Draft records before restricting the valuation sample to first-entry players. Thus an experienced player selected through an ordinary open selection could consume an open rank, even though that player was excluded from our first-entry valuation sample.
Within the 566 first-entry selections used for the new analysis, 68 were classified as restricted, leaving 498 for the open-player sensitivity analysis.
The resulting Pick 20 comparisons were:
26.2: all 566 selections, using recorded pick
23.1: the 498 open-classified selections, still using recorded pick
20.6: those same 498 selections but using constructed open rank
The process involved two major steps:
changing the player sample
renumbering that same sample
What does this establish and what does it not?
It shows that the fitted relationship is sensitive to whether we count annotated restricted-access recruits as steps in the selection order but it does not establish an underlying talent ranking or recreate what the draft would have looked like without preferential-access arrangements.
In particular we should be mindful that:
An unannotated selection is assumed open so missing annotations could cause misclassification
Historical access rules differed, and the code did not reconstruct them year by year.
Removing selections does not recreate how clubs would have behaved in a counterfactual draft
Priority selections remain in the count
Differences in draft depth remain
Constructed open rank 20 is not interchangeable with recorded Pick 20 in the official DVI
Priority selections were assessed separately. Excluding them gave a Pick 20 index of 26.7. They were otherwise retained as open choices because priority status changes who holds the selection, rather than necessarily restricting which club could recruit the player.
To summarise, as a sensitivity check, we used the recruitment annotations in Draftguru tables to construct an alternative selection order. We removed selections labelled academy, father–son, scholarship or zone recruitment, except where the annotation explicitly said the arrangement was not matched. We then numbered the remaining National Draft selections consecutively within each year. This provides an approximate ranking of selections excuding those annotated players, but it does not reconstruct every club’s historical rights or the draft that would have occurred without them.
Assessment: selection definition makes a material difference because the player in open rank 20 is a different opportunity to the player from recorded Pick 20.
We should note that reindexing cannot make overall strong and weak draft classes equivalent.
6. Extrapolation: how far beyond the data should the curve be trusted?
A smooth equation produces a number at every Pick number, even where there is little evidence for that particular Pick.
The new charts stop at Pick 70 and do not extend the old approximation to its zero crossing. The fixed-window sample contains 59 selections at picks 61 to 70. Observations beyond Pick 70 still inform fitting, but the extreme tail is not presented as part of the output.
Several curve shapes were also compared as in the original analysis. Their predictive performance was very similar, and their adjusted Pick 20 estimates ranged from 26.2 to 30.8.
Assessment: limiting the displayed range addresses the problem of presenting a potentially misleading curve. Testing alternative shapes exposes some of the model dependence on the choice of functional form.
7. Pick 1 uncertainty: how reliable is the denominator?
Setting Pick 1 to 100 is purely for the purpose of normalisation If the estimated value of Pick 1 changes, every relative value changes too.
In a new analysis 1,000 bootstrap resamples of players for all Pick values were used to assess sampling uncertainty. The curve was refitted for every sample, dividing each Pick's estimate by that sample's own Pick 1 estimate. The rookie benchmark was also resampled.
That analysis produces a 95% interval of 13.9 to 47.9 around Pick 20's central eight-season estimate of 26.2. Pick 1’s absolute excess-output relative to the 7.56 Rating point benchmark estimate is 371 Rating points, with an interval of approximately 217 to 553 Rating points.
Assessment: uncertainty about Pick 1 value is substantial, and the intervals shown here exclude uncertainty about what replacement should mean, the quality of future draft cohorts, and all possible model choices.
8. Position: do similar-looking curves justify one DVI?
The original curves by position shared the same debut-risk and career-exit assumptions. That helped to make their shapes similar.
A proposed new analysis would allow position-specific slopes and curvature. Unfortunately, it has only 18 known-role rucks, and positions are unavailable for all non-debutants and 67 debutants. Also, the Forward and Ruck Pick 1 benchmarks are too uncertain to normalize reliably.
Assessment: there is insufficient data to perform the proposed analyses.
SUMMARY AND CONCLUSION
In summary, the new analysis substantially supports the replacement-level critique, finds only small effects from discounting future Rating point hauls and from applying club controls. It resolves the specific survival-data concern, and raises issues about the positional results that can’t yet be addressed with the available sample.
Given all the new analysis here, I would recommend changing the pick versus relative value curve to be the one produced using the first eight post-draft seasons of output above the rookie replacement benchmark, using recorded pick numbers, including non-debutants, and normalising Pick 1 to 100.
It appears in the plot below along with the curve we had in the original analysis.
It’s also scarily similar to the latest AFL DVI curve, especially up to about Pick 25 after which, presumably, the League’s intention to deliberately undervalue higher order Picks starts to take effect.
The curve is the most defensible empirical output curve.
It measures additional output above an alternative player, rather than against no player at all.
Its 566 selections from 2011–2018 have complete eight-season follow-up, avoiding whole-career projections.
It avoids discounting because choosing a particular discount rate requires an additional assumption, and the tested rates change the results relatively little.
Analyses with club controls and open-selection ranks do not establish a causal “patience correction” or the need for a different replacement value for recorded pick numbers.
The fitted curve is approximately: V(p) = 100 − 29.255 × ln(p) + 1.545 × [ln(p)]²
It should only be used within the displayed Picks 1 to 70, without extrapolating beyond that range. Some selected index estimates are 40.8 at Pick 10 (95% interval 26.2 to 67.1), 26.2 at Pick 20 (13.9 to 47.9), and 13.1 at Pick 40 (5.5 to 26.4).
