Simulation Replicates and Returns to a Perfect Model

The Situation

We’ve built a model designed to estimate the probability of a binary event (say, for example, the probability that the home team wins on the line market in the AFL).

It’s a good model - very good, in fact, because it is perfectly calibrated. In other words, when the true probability of an event is X% it’s average estimate of the probability of that event is X%.

Those probability estimates, however, are the result of running some simulation replicates with a stochastic element, which means that those estimates will diverge from X% to an extent determined by how many replicates we run.

We’re going to further assume that we’ll be using this model to estimate probabilities near 50% so we have the following standard deviation / replicate number trade-off:

• 100 replicates: SD is 5%

• 250 replicates: SD is 3.2%

• 1,000 replicates: SD is 1.6%

• 5,000 replicates: SD is 0.7%

THE DEPLOYMENT

We’re going to take four versions of our model, one based on 100 simulation replicates for each estimate, one based on 250 simulation replicates, one based on 1,000 simulation replicates, and one based on 5,000 simulation replicates and we’re going to pit them against a range of line market setting bookmakers.

Each of those bookmakers sets a line but does so with some level of fallibility.

Our first bookmaker puts out lines where the true home team probability has mean 50% but with a standard deviation of 2%. So, in other words, about two-thirds of his or her lines will produce a true home team probability of between 48% and 52%.

The next bookmaker will have a standard deviation of 2.5%, the next 3%, and so on up to our favourite bookmaker, whose standard deviation is 5%.

Now, bookmakers being bookmakers and needing to eat, they won’t offer us even money odds but will, instead, offer us $1.90, which means that we’ll need to be right just under 53% of the time to make a profit.

Recognising this bookmaker edge, the model variants will only wager if their estimate of the home team’s chances are 45% or lower (in which case they’ll wager on the away team) or 55% or higher (in which case they’ll wager on the home team).

As you work through the numbers to come, bear in mind that we’re assuming each model variant “knows” the true home team probability in the sense that, on average, its estimate would be the true value. They therefore have the extraordinary advantage of knowing when and by how much a bookmaker has mispriced a market.

The final assumption we’ll make is that each season is 200 games long and all bets are level-stakes.

THE PERFORMANCE

For each combination of model and bookmaker variant we’ll consider three metrics:

• In what percentage of games does the model variant wager?

• In what percentage of seasons does the model variant make a profit?

• What is the average return on wagering for the model variant?

How Often Do They Wager?

View fullsize

We find that the model variant based on 5,000 replicates (which will, on average, be a better estimator of the potential value in a wager) bets less often than the other model variants against each of the bookmaker variants.

All models bet least of all against the most precise bookmaker (with the 2% standard deviation) and most of all against the least precise bookmaker (with the 5% standard deviation), as we would hope.

(Note that we can estimate how often a given bookmaker is likely to set a line with an associated probability that’s in our range. For example, the bookmaker with the 2% standard deviation will inadvertently set a line with a 45% or lower probability about 0.6% of the time and will set a line with a 55% or higher probability also about 0.6% of the time. In total then, only just over 1% of his or her markets will represent value. We see that the model variant based on 5,000 replicates does a relatively good job at wagering infrequently whereas the model variant based on only 100 replicates is severely harmed by its relative lack of precision and finds value in one one-third of the markets.)

How Often Are They Profitable?

View fullsize

Again as we’d expect, each given model variant is more often profitable the less precise the bookmaker and, model variants based on more replicates are more often profitable with a given bookmaker variant that model variants based on fewer replicates.

The one exception to this is the model variant based on 5,000 replicates facing the bookmaker with a 2% standard deviation. Here we find that the model bets so rarely that there is a very large variance associated with our estimate of the proportion of seasons in which it is profitable (even though it is based on 50,000 simulated seasons).

It’s interesting to note that the model variants based on 100 or 250 replicates are profitable less than half the time when facing the more precise bookmakers.

How Profitable Are They?

View fullsize

In our final chart we look at the return on wagering and find that, on average, the model variants based on 1,000 or 5,000 replicates are net profitable against every bookmaker.

We also find that all model variants become increasingly profitable as the bookmakers become less precise.

Interestingly, however, we see that the model variants based on 100 or 250 replicates are net loss makers against the most precise of the bookmakers.

FINAL THOUGHTS

To some I’m sure these results are entirely unsurprising but for me it was the first time I’d ever really thought hard about how my choices of replicate numbers might be adversely affecting my models’ performance.

Seeing that variants of a perfectly calibrated model could move from losing an average of 3% per season to returning a profit of over 4% based solely on the number of replicates used to create the probability estimates, was definitely the proverbial eye-opener to me.

As always, I would welcome your feedback and thoughts.