July 2026·25 min read

a data-science essay · rating systems × dinner

ELO for dinner

I made 49 restaurants duke it out 304 times, then ranked them every way possible.

Plate I

0 / 304

press play, or drag the scrubber

Each restaurant goes head to head against another, like a chess match. Press play, scrub around, watch for the red upsets. Everything in this chart is explained in chapter 01.tier-seeded ELO · K = 32

I have been very lucky. Lucky to eat my way across a decent slice of the world, and luckier still in company: family, a partner and friends who are willing to dabble in a tasting menu, and willing to drive an hour in a foreign town for fish and chips on a tip from an Uber driver.

I love food. I love how visceral and evocative it is. A single bite can take you back twenty years (Anton Ego only needed one spoonful of Ratatouille). A slurp of ramen from an unassuming little Kurokawan shop connects you instantly to the elderly owner toiling over the stove. A newly found Putanescca recipe can just make an ordinary after-work Tuesday better. The scale runs all the way from Escoffier’s orderly brigades to the chaos of mum’s kitchen, and I have loved all things at both ends of it.

So, naturally, on this pilgrimage, I kept track of these culinary waypoints. For years the scorekeeping system was, primitively, a single note in my iPhone’s notes app: restaurant names sorted into five tiers, s at the top, f at the bottom, shuffled from memory whenever a meal deserved a move. Forty-eight restaurants lay there, a tangled mess with no methodology.

I kept meaning to share the list. Two things turned that into a project:

sharing it meant restructuring it, and a hand-shuffled tier list just seemed to lack the academic rigor that the pilgrimage deserved.
ELO has been a small mystery to me since I was a kid. Given the toolset I possess now, it seemed an adequate time to finally sate my curiosity.

So I built a tiny app for an audience of one. It shows me two restaurants I have eaten at and asks the only question that matters: which was better and which would I go back to? I answered it 304 times in one sitting, in about 43 minutes.

It looked and felt something like this, here restocked with Australian takeaway royalty so you can play along:

Plate II

which was better? (← / → work too)

top 5 · live elo

A working recreation of the battle app. Pick winners and a live ELO scoreboard sorts the field; after a few battles it starts telling you what it expects. The maths behind it is chapter 01.live ELO · K = 32 · everyone starts at 1000 · nothing is saved

This entire essay then looks at that ‘battle’ data. Every battle is one line in an append-only file:

{
  "n": 1,
  "ts": "2026-07-18T14:52:32Z",
  "left": "salted-egg",
  "right": "crossley-st-catina",
  "winner": "salted-egg",
  "selector": "coverage",
  "ms": 55408
}

The log stores choices rather than scores, so every rating system in this essay is computed after the fact from the same immutable record. I went to town: exploring different models, changing parameters and bootstrapping uncertainty. In this essay, you’ll learn how all of this works, and what I found out.

what else I collected

Battles were the main event, but not the only thing I recorded in that sitting. I also ranked random trios of restaurants from best to worst, and typed a cold 0-100 score for every restaurant on the list. Later chapters use these as checks on what the battles found.

signal	samples	the question it answers
battles	304	which of these two was better?
triplet rankings	51	rank these three, best to worst
absolute scores	49	how good is it, 0 to 100?
Google ratings	48	what does the internet think?

Every ranking system in this essay starts from the same tiny question: two restaurants, which was better?

chapter 01 · the running scoreboard

ELO

If you pressed play up top, here’s the system.

Every restaurant is seeded with a rating based on my prior (tier) system: S tier seeded at 1300, F tier at 900. The reason for this is I simple: I don’t plan on battling out all $1{,}176$ $\binom{49}{2}$ pairs. As such, since we’re using adaptive ‘match-making’, setting out a good set of seeds should help us accelerate the optimisation process.

Then the matches begin, I consider both, select a winnder and each one moves exactly two bars: the winner up, the loser down, by the same amount. Vue de monde fought ten battles, had an Invincible season and finished on top at 1399. Niku Ou lost all seven of its matchups and sank to the bottom. And 68 of the 304 battles were upsets, where the restaurant the scoreboard favoured lost anyway.

The scoreboard behind that animation is ELO, invented by the physicist Arpad Elo to rank chess players (Elo, 1978), and since borrowed by football rankings, online games, and, infamously, by the Zuck in FaceMash. The entire algorithm fits in two lines. Before a battle, convert the rating gap into an expected score of A winning:

E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}

After the battle, nudge both ratings toward what actually happened. With the actual result $S_A$ (1 for a win, 0 for a loss):

R_A \leftarrow R_A + K\,(S_A - E_A)

In words: how surprised were we, and by how much? $E_A$ is the win probability the current ratings imply. If A was already expected to win 90% of the time, a win teaches us almost nothing and moves almost nothing. A shock win moves a lot. $K$ (I used 32) is how hard one result can yank a rating, and the 400 is just a choice of units: a 400-point gap means 10-to-1 odds, and a 200-point gap means about a 76% favourite.

Watch it work on my actual first battle, the one from the prologue. Salted Egg and Crossley St Cantina both sat at 1000, so $E = 0.5$ exactly: a coin flip. Salted Egg won and gained $32 \times (1 - 0.5) = 16$ points; the Cantina lost the same 16. Had they rematched immediately, the scoreboard would already have made Salted Egg a $1/(1+10^{-32/400}) = 54.6\%$ favourite. Small evidence, small update (this is why high ELO players grind so slowly).

Plate III

The probability of winning against the difference in ratings. Left: the rating gap sets A's win probability along a logistic curve. Right: the same fact seen through the noise model, where the shaded area is the win probability. Flip to the normal view to see how little the famous curve choice matters.E = 1/(1+10^(-Δ/400)) · matched σ = 315

That S-shaped logistic / sigmoid curve is actually not arbitrary.

The fundamentals are fairly logical statistically: each restaurant has a true latent strength. When ranked, the performance of that restaurant against another, is its true strength plus some randomness (in this case, the randomness probably primarily driven by my ability to recall the experience).

A wins when its noisy performance beats B’s. Make an assumption about the shape of that randomness (either Normal or Gumbel) and the win probability follows mechanically; the aside below walks the whole derivation. What matters at the table is the shape: near even ratings the curve is steep and every battle is informative; out at the extremes it flattens, and beating a pushover proves nothing.

where the win probability comes from

Give each restaurant a performance on the night: its rating plus noise,

P_A = R_A + \varepsilon_A \qquad P_B = R_B + \varepsilon_B

A wins when $P_A > P_B$ . Rearranging, that is exactly the event

R_A - R_B > \varepsilon_B - \varepsilon_A.

Define $Z = \varepsilon_B - \varepsilon_A$ , the opponent’s luck advantage on the night. Then

\Pr(A \text{ wins}) = \Pr(Z < R_A - R_B) = F_Z(R_A - R_B),

where $F_Z$ is the cumulative distribution function of $Z$ : the function that answers “what fraction of possible luck swings land below this value?”. This is the right-hand panel of Figure 3: the shaded area below $Z = \Delta$ is the win probability, and no further magic is involved.

Everything else is choosing a shape for $Z$ :

Normal noise. If $\varepsilon_A, \varepsilon_B \sim \mathcal{N}(0, \sigma^2)$ independently, their difference is normal with variance $2\sigma^2$ , and

E_A = \Phi\!\left(\frac{R_A - R_B}{\sqrt{2}\,\sigma}\right).

This probit form was Arpad Elo’s original conceptual model.

Logistic noise. Assume instead $Z \sim \mathrm{Logistic}(0, s)$ , whose CDF is the S-curve $F_Z(z) = 1/(1 + e^{-z/s})$ . Then

E_A = \frac{1}{1 + e^{-(R_A - R_B)/s}},

and choosing the scale $s = 400/\ln 10 \approx 173.7$ turns $e$ into $10$ and produces the standard formula. The familiar ELO equation is nothing more than the statement ” $Z$ is logistic with scale $400/\ln 10$ ”.

One subtlety keeps this honest: the difference of two logistic variables is not logistic, so you cannot get the standard formula by assuming logistic noise per restaurant. Two consistent stories exist. Either assume the shape of $Z$ directly and stay silent about the individual noises, or give each restaurant Gumbel-distributed noise, whose difference is exactly logistic (the same trick that powers logit models across economics).

the Gumbel integral, if you want it all

Let $\varepsilon_A, \varepsilon_B$ be iid $\mathrm{Gumbel}(0, s)$ with CDF $G(x) = \exp(-e^{-x/s})$ and density $g(x) = \tfrac{1}{s} e^{-x/s} \exp(-e^{-x/s})$ . Condition on $\varepsilon_A = x$ :

F_Z(z) = \int_{-\infty}^{\infty} G(x + z)\,g(x)\,dx = \int_{-\infty}^{\infty} \exp\!\big({-e^{-(x+z)/s}}\big)\,\tfrac{1}{s} e^{-x/s} \exp\!\big({-e^{-x/s}}\big)\,dx

Substitute $u = e^{-x/s}$ , so $du = -\tfrac{1}{s} e^{-x/s}\,dx$ and the limits flip:

F_Z(z) = \int_0^{\infty} \exp\!\big[-u\,(1 + e^{-z/s})\big]\,du = \frac{1}{1 + e^{-z/s}}

which is the logistic CDF, as promised.

Two sanity checks fall out for free. Symmetric noise means $F_Z(0) = 0.5$ , so equal ratings give a coin flip, and $E_B = 1 - E_A$ , so the probabilities always add to one. And the implied noise is large: the logistic $Z$ with Elo’s scale has standard deviation $\pi s/\sqrt{3} \approx 315$ rating points, which is the model’s way of admitting that single results are noisy and a 200-point favourite still loses one time in four.

the update rule is a gradient step

The $K(S_A - E_A)$ update is not folklore either. Ask: after seeing A win, how should I nudge A’s rating so the result I just watched looks a little more likely? Score the battle by the log of the probability the model assigned it, $\log E_A$ . The logistic curve has the lovely property that its slope is $\sigma' = \sigma(1 - \sigma)$ , so by the chain rule,

\frac{\partial}{\partial R_A} \log E_A = \frac{E_A (1 - E_A)}{E_A} = 1 - E_A.

The push is “actual minus expected” with $S_A = 1$ . If A had lost, the same steps on $\log(1 - E_A)$ give a push of $-E_A$ , which is again $S_A - E_A$ with $S_A = 0$ . Either way, the gradient is $S_A - E_A$ , and stepping along it with a fixed learning rate is literally the ELO update. In modern terms: ELO is online stochastic gradient descent, one pass through the data, one step per battle. Hold that thought for chapter 02.

how K is chosen

$K$ is exactly a learning rate, with a learning rate’s tradeoff: large $K$ tracks changing form quickly but never stops jittering, small $K$ averages over more history but adapts slowly. In practice it gets picked two ways. Chess hand-crafts a schedule: FIDE uses K = 40 for new players, 20 after 30 games, 10 for the elite, which is a decaying learning rate in a trench coat. Serious forecasting systems fit it: FiveThirtyEight tuned its NFL Elo’s K by minimising prediction error on held-out games (FiveThirtyEight, 2015). I did not bother tuning: on my 304 battles, K = 16, 32 and 64 all produce rankings that agree with chapter 02’s benchmark at τ = 0.77, 0.79 and 0.81 respectively, differences well inside the noise band you will meet in chapter 03. Any sane K tells the same story about dinner.

Rock-paper-scissors and why ELO is one-dimensional

ELO assumes restaurants sits on a single axis: if A beats B and B beats C, then A should beat C. My actual battles diverge from this superficial view. Among the 168 trios of restaurants where every pair actually fought, 8 come out as cycles: A beats B beats C beats A, rock-paper-scissors, 4.8% of the time. That is not an error in the data; it is a fact about preferences.

When forced to compare two restaurants directly, preferences are not binary. When presented a churrascaria and tapas - I prefer the Brazilian buffet. When presented the diversity of Australian modern and tapas - maybe tapas wins. Yet, for some reason or other, I’d rather have the mismash of Australian fusion over the buffet.

Every ranking in this essay is therefore a projection: a useful flattening of a higher-dimensional preference. The models cope gracefully (the three members of a cycle just land near each other), but it is worth remembering what got flattened every time you read a rank.

Plate IV

Four of the eight cycles in the data, drawn. Arrows point from the majority winner to the loser: follow any triangle around and you never stop. The other four are reruns with an accomplice swapped, including Cumulus Inc. and FOGO pulling it off twice.majority results over fully-played triads · 8 of 168

Two things hiding in the log

The cycles are not the only quirk buried in 304 clicks. Two more, both about me rather than the restaurants. My left thumb nearly got me: the left-hand card won 166 of the 304 battles, a 54.6% lean, though a fair coin still lands at least that lopsided one time in eight (p = 0.12), so it stays filed under suggestive, not guilty. And my clicking speed tracked how close each matchup was: I dispatched the blowouts fast and agonised over the coin flips (a correlation of −0.27 between decision time and rating gap). You could almost rank the restaurants by my hesitation alone.

There is one more catch, and it is the big one. ELO processes battles in order, and early battles move ratings while the scoreboard still knows nothing. Play the same 304 battles in a different order and you get different final numbers. How different? That question deserves its own chapter.

There is a method that reads the whole season at once before ranking anyone. It thinks ELO has been doing it wrong.

chapter 02 · all at once

Bradley-Terry

Bradley-Terry (Bradley & Terry, 1952) keeps ELO’s model of the world exactly (same curve, same noise); the only thing it changes is how the ratings are formed. Instead of updating one battle at a time, it fits them in batch: give every restaurant a strength $s_i$ , and find the set of strengths that makes all 304 observed results most likely at once:

P(i \text{ beats } j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}} \qquad\qquad \ell(s) = \sum_{(w,\,l)\ \in\ \text{battles}} \log \frac{e^{s_w}}{e^{s_w} + e^{s_l}}

The left equation is effectively ELO’s expected score, in natural units. The right is the score being maximised: the log-probability of every battle at once. If we manage to globally optimise for the likelihood then two things happen:

First, the optimum is exactly the set of ratings where every restaurant’s expected win total equals its actual win total, simultaneously, for all 49 at once; nobody is over- or under-rated relative to their record.
Second, look at that sum: addition does not care about order. Shuffle the battle log any way you like and $\ell$ is the same function with the same maximiser. ELO’s path / order dependence was an artifact of the fitting procedure, never the model.

From the fit, we can observe the strengths. Vue de monde lands at $s = 2.72$ and Chur Fish and Chips at $2.66$ , which results in a win probability of $\sigma(0.06) = 51.4\%$ . The number one restaurant on my list beats number two with a coin flip.

Meanwhile Vue de Monde against Niku Ou ( $s = -5.60$ ) comes out at $99.98\%$ .

Plate V

Bradley-Terry on the left, the two scoreboards from chapter 01 on the right. Aqua lines climb at least five places under the right-hand method, red lines fall at least five: those are the restaurants whose fate depended on how the ratings were fitted, not on how they cooked.bt with alpha = 0.01 · toggle: cold vs tier-seeded elo

How certain are these BT rankings

A core question we haven’t addressed so far: how sure is any of this? The fitted strengths are point estimates. To get confidence intervals, I resampled the 304 battles with replacement (i.e., bootstrapping) a thousand times, refit the whole model each time, and watched where every rank landed.

Vue de Monde’s rank stays in [1, 5] in 95% of resampled seasons: genuinely, robustly excellent (you might be able to tell that I’m a big fan). But Lexy, a late entrant with only 11 battles, gets “rank 7” as its point estimate with a 95% interval of [2, 19]: a lot less confidence.

”Rank 7” sounds precise. “Somewhere between 2nd and 19th” is what the data actually supports.

Plate VI

Every rank with its confidence interval: the 95% bootstrap interval for all 49 restaurants, battle counts on the right. Tight bars are settled; long bars are places the data has not pinned down, and they are almost always the thinly-battled ones.bootstrap x1000 · seed 7 · sorted by point rank

That is the single most important statistical habit in this essay: never print a rank without asking how wide it really is. Figure 6 is that question asked 49 times, and the thinly-battled restaurants wear visibly longer bars.

How did my Notes tier list compare?

Plate VII

The hand-typed prior meets its reckoning. Each column is a tier from the Notes app; each dot is where Bradley-Terry actually put that restaurant. The shaded blocks are where a perfect prior would have landed its members, and the white-ringed escapees left their block entirely.tier list vs bt ranks · biggest escapees labeled

The tier list agrees with Bradley-Terry at τ = 0.71 (τ is defined properly in chapter 03; for now, 1 means identical rankings and 0 means unrelated). But the escapees are the interesting part, because each one is a place where forced choice disagreed with my own ability to rank.

Sushi Kakujuan, a small omakase tucked away on the cobblestone streets of Fukuoka, performs under pressure and rises to the top. Memories of the Chawamushi power it past favourites like San Telmo. In contrast, Attica, a 3-hat, stoops below a Bak Kut Teh restaurant under the harsh light of scrutiny.

Why would that be? It is almost always easier to answer “what do you want for dinner?” when the options are sitting in front of you. Asked in the abstract, I reach for fleeting feelings of that experience; asked to pick between two concrete restaurants, I reach for the memory of actually eating there.

The human machinery for comparison is simply better than the capacity for absolute judgment, and this is a well studied psychological phenomenon. Thurstone built the entire measurement theory behind this essay on paired comparisons because absolute scales kept failing (Thurstone, 1927), and modern work argues that even when we appear to score things in isolation, we are secretly comparing them to whatever came just before (Stewart et al., 2005). The tier list asked me to be a scorer. The battles let me be a chooser, and the chooser turned out to know my palate better.

This generalises well past dinner. Perhaps comparison feels so natural because life itself is largely made of choices: small encounters between competing possibilities, each revealing something about who we are and how we see the world. You do not discover what you love by ranking it from memory. You discover it by living through the matchups, and noticing what, given the chance, you keep choosing.

Bill Watterson · Calvin and Hobbes

It’s a magical world, Hobbes, ol’ buddy… Let’s go exploring!

ELO is one epoch of BT

Chapter 01’s aside showed the ELO update is a gradient step on the log-likelihood of a single battle. Bradley-Terry’s objective is the sum of those per-battle terms, so differentiate $\ell$ with respect to one restaurant’s strength and the per-battle pushes just add up:

\frac{\partial \ell}{\partial s_i} = \underbrace{\textstyle\sum S_i}_{\text{actual wins}} - \underbrace{\textstyle\sum E_i}_{\text{expected wins}}

Setting every one of these to zero is the condition for the maximum, and it reads as the balance property from the main text: expected wins equal actual wins for everyone at once. No battle order appears anywhere in that condition.

The objective is also concave, so the peak is unique (given a connected comparison graph and the small prior). That licences a cheeky description of the two methods: full-batch gradient ascent would reach the peak; ELO is stochastic gradient descent run for exactly one epoch with a constant learning rate and no convergence check, stopping wherever it happens to be standing when the battles run out. Run ELO over the shuffled log again and again with a decaying K and it converges to Bradley-Terry’s answer. The two methods do not disagree about dinner; one of them just stopped training early.

what choix actually runs (a PageRank cousin)

The library I used (choix) does not climb the gradient. It runs a spectral shortcut called iterative Luce spectral ranking (Maystre & Grossglauser, 2015): build a Markov chain over restaurants where each battle “i lost to j” adds a transition from i to j, so losers pass endorsement to whoever beat them, weighted by the current strength estimates. The stationary distribution of that chain, meaning where a random walker following endorsements spends its time, becomes the new strength estimate. Re-weight, recompute, repeat; the fixed point satisfies the same expected-equals-actual condition as the gradient view. If that walker-following-links story sounds familiar, it is because a close cousin of this algorithm ranks the web.

BT just claimed battle order cannot matter. ELO disagrees, measurably.

chapter 03 · order & sample size

Two stress tests for ELO: volume and order

You might recall that chapter 01 posed a fun little hypothetical that decomposes into two questions about ELO:

Hypothesis 1.

Order: The order the battles arrive in changes ELO’s final ranking.

Null. shuffling the battles yields the identical ranking every time.

Method. replay the same 304 battles in 200 random orders.

Prediction. the 200 shuffled rankings scatter instead of coinciding.

Hypothesis 2.

Volume: Fewer than 300 battles would’ve told the same story.

Null. agreement keeps climbing right up to the last battle.

Method. pause the real season every 5 battles, compare to the final ranking.

Prediction. agreement plateaus well before battle 304.

Ground rules

In order to assess both these hypotheses, we need a ‘golden’ ranking to score against: the order-free Bradley-Terry ranking from the last chapter.

The metric used to see how much a list of ranking agrees with another will once again be Kendall’s τ: take every pair of restaurants, ask both rankings which one is better, and score $\tau = (\text{agreements} - \text{disagreements}) / \text{total pairs}$ . Identical rankings score 1, unrelated rankings score about 0, a perfectly reversed ranking scores −1. So τ = 0.8 means the two rankings settle nine out of ten pairwise bets the same way. There are other rulers for comparing rankings, and chapter 05 shops between them, but τ is enough to read everything below.

Results

The order test. Shuffle the same 304 battles into 200 random orders and replay cold ELO from scratch on each one. Only the sequence changes. Scoring every final ranking against the order-free Bradley-Terry answer from chapter 02 gives a spread of answers.

The τ ranges from 0.765 to 0.828 with a mean of 0.795. My actual chronological order scores 0.799, right in the middle of the band. The selector caveat from chapter 01 and the K sweep both live inside this same ±0.012 band.

So H₁ holds: order moves the ranking, though only within a narrow band that no other knob escapes either.

Plate VIII

The two stress tests. Left: order. 200 replays of the same season in shuffled order each end somewhere different (the bars); the gold line is the one order I actually lived, and Bradley-Terry is absent because it gives the identical answer for all 200. Right: sample size. Pausing the real season every five battles and comparing the running ranking to the final answer shows fast progress early, grinding progress late.200 shuffles · cold elo, k = 32 · seed 7

The sample-size test. The right panel replays my real season, pausing every five battles to compare the running ranking against the final answer. Agreement climbs fast and then grinds: τ = 0.41 after 50 battles, 0.61 at 150, 0.80 by battle 304. The second half of my session bought less ranking accuracy than the first quarter did. And the dashed top-10 line never fully settles, ending at 8 matches out of 10, because the podium is where the gaps are coin flips.

H₂ is the messier verdict: broadly supported, with an asterisk. A few dozen battles nailed the broad order, so most of my clicking was indeed redundant, but the very top of the list kept moving until the end and never fully settled. Fewer battles would have told the same story about the middle and bottom, and a fuzzier one about the podium.

ELO has another blind spot. It never says how sure it is.

chapter 04 · other ways to ask

The alternatives

ELO and Bradley-Terry are not the only options. This chapter runs three alternatives over the same data, and the thread connecting them is the same question chapter 02 opened: what does each method know about its own uncertainty?

TrueSkill: a belief, not a number

TrueSkill is Microsoft’s matchmaking system, built for Xbox Live (Herbrich et al., 2007). Its one big idea fixes ELO’s frequentist blindside. Instead of storing a single number for each restaurant, it stores a belief: skill is modelled as a normal distribution, $s \sim \mathcal{N}(\mu, \sigma^2)$ , where $\mu$ is the best guess and $\sigma$ is how unsure the system still is.

Every result then shifts the means toward whatever just happened, and shrinks the sigmas, because each battle teaches the system a little more about both fighters. The shift is sized by surprise, exactly like ELO.

Plate IX

no results yet

The first battle. Both beliefs start where TrueSkill started every restaurant. Declare a few winners: the first result moves a mean by about 4.2 and shrinks both sigmas from 8.33 to 7.20, repeats of the same result move less and less, and an upset after a streak yanks hard. The card on the right shows every quantity the aside derives.mu 25 · sigma 25/3 · beta 25/6 · no draws

Your Halo 3 rank is actually the triangle in the figure. TrueSkill ranks not by $\mu$ but by the conservative estimate $\mu - 2\sigma$ , “we are 97.7% sure the skill is at least this”. After one win, the winner’s conservative score is still below the 25 it started at: one win is evidence you are good, not yet evidence you are confidently good.

Second, that conservatism reshuffles my podium. On the full data Chur Fish and Chips (a delightful little Queenstown chippery) takes TrueSkill’s top spot from Vue de monde, because with a 51/49 coin-flip gap the ranking is decided by uncertainty bookkeeping, not by strength. Meanwhile the sigmas themselves tell their own story: they range from 2.7 for the well-measured restaurants up to 4.26 for Niku Ou, whose seven straight losses left the system sure it is bad but unsure exactly how bad (until it’s ranked against everything else…)

from beliefs to a win probability

Skill and single-night performance are different things. TrueSkill models a performance as skill plus match noise, $p = s + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0, \beta^2)$ : $\sigma$ is uncertainty about skill, $\beta$ is the randomness of one dinner around it. Both beliefs and noise are Gaussian, so A’s performance is $p_A \sim \mathcal{N}(\mu_A, \sigma_A^2 + \beta^2)$ , and the difference $d = p_A - p_B$ is Gaussian too:

d \sim \mathcal{N}(m, c^2), \qquad m = \mu_A - \mu_B, \qquad c^2 = \sigma_A^2 + \sigma_B^2 + 2\beta^2

A wins when $d > 0$ . Standardise and the win probability is

\Pr(A \text{ wins}) = \Phi\!\left(\frac{m}{c}\right) = \Phi(t),

where $t = m/c$ is a signal-to-noise ratio: the expected advantage, measured in units of total match uncertainty. Two matchups can share the same $m$ and disagree wildly on $t$ ; the one with fatter sigmas is simply less predictable.

where beta comes from, and whether it can be estimated

The original deployment fixed the scale by convention: $\mu_0 = 25$ , $\sigma_0 = 25/3$ , $\beta = \sigma_0 / 2 = 25/6$ . Nothing sacred there; it is a calibration, like ELO’s 400. But $\beta$ can also be fit to data: propose a value, let the model predict every held-out match, and keep the $\beta$ that maximises the likelihood of the observed results. That is exactly how TrueSkill 2 was evaluated: not by whether ratings looked plausible, but by how well they predicted matches the model had not seen (Minka et al., 2018). One catch is scale identifiability: multiply every $\mu$ , $\sigma$ and $\beta$ by the same constant and all predictions are unchanged, so you fix the scale first and estimate $\beta$ relative to it.

the update, derived (v shifts, w shrinks)

Observing “A won” is observing $d > 0$ , and conditioning a Gaussian on being positive truncates it. The truncated Gaussian has a known mean and variance, and TrueSkill names the two correction factors:

v(t) = \frac{\phi(t)}{\Phi(t)} \qquad\qquad w(t) = v(t)\,\big(v(t) + t\big)

$v$ measures how much the observed win shifts the expected performance gap (tiny for an expected win, huge for an upset); $w$ measures how much variance the observation removes. The mnemonic is the whole story: v shifts, w shrinks. The updates then follow:

\mu_A' = \mu_A + \frac{\sigma_A^2}{c}\,v(t) \qquad \mu_B' = \mu_B - \frac{\sigma_B^2}{c}\,v(t) \qquad \sigma_i'^2 = \sigma_i^2\left[1 - \frac{\sigma_i^2}{c^2}\,w(t)\right]

Read the mean update as a ratio: player uncertainty over total match uncertainty, times the surprise correction. A restaurant the system barely knows moves a lot; a well-measured one barely moves. That fraction $\sigma^2 / c$ is a per-restaurant, automatically decaying learning rate, which is the principled answer to chapter 01’s “how do you pick K”: FIDE’s hand-carved 40/20/10 schedule is an approximation of exactly this.

the covariance step nobody writes out

Why does the winner’s belief move up at all? Because $d = s_A - s_B + \varepsilon_A - \varepsilon_B$ , and expanding the covariance term by term (skills independent, noise independent of skill) leaves $\operatorname{Cov}(s_A, d) = \sigma_A^2$ and $\operatorname{Cov}(s_B, d) = -\sigma_B^2$ . Worlds where A is better than we thought are exactly the worlds where $d$ runs high, so learning " $d > 0$ " pushes A’s belief up and B’s down, each in proportion to its own variance. The Gaussian conditioning formula $\mathbb{E}[s_A \mid d] = \mu_A + \tfrac{\operatorname{Cov}(s_A, d)}{\operatorname{Var}(d)}(d - m)$ turns that covariance into the update, averaged over the positive $d$ we actually observed.

bootstrap vs sigma: two kinds of uncertainty

Chapter 02’s bootstrap and TrueSkill’s $\sigma$ both produce error bars, from opposite philosophical directions. The bootstrap is frequentist: uncertainty lives in imagined replications (“if I had collected a different-but-similar 304 battles, how much would the answer wobble?”). $\sigma$ is Bayesian: one dataset, no replications, the posterior spread is the uncertainty. The practical differences matter more than the philosophy. $\sigma$ is free and live, updated after every battle, which is why a matchmaking system uses it; the bootstrap is a thousand offline refits. The bootstrap can put an interval on anything you can recompute per resample, including rank itself, which is what a reader actually cares about; $\sigma$ cannot directly, because a rank couples all 49 beliefs and TrueSkill only keeps per-restaurant marginals. And the bootstrap moves all strengths together so it captures “if Vue falls, Chur takes #1”; TrueSkill’s factorized posterior is blind to that structure and slightly overconfident as a result. They agree on the headline, though: less data means wider, in both. When two methods with opposite philosophies point at the same thinly-battled restaurants, believe them.

Triplets: ranking three at a time

A third collection mode: instead of a pair, the app showed me three restaurants and asked me to rank them best to worst. One screen implies three pairwise results, so my 51 triplet screens encode 153 implied comparisons. Plackett-Luce is the model for this (Plackett, 1975)(Luce, 1959): it treats a ranking as repeatedly picking the favourite from whoever is left, reusing Bradley-Terry’s strength-based choice probabilities.

On paper that is more information per screen. In practice it under-delivered: the triplet fit agrees with the full battle ranking at τ = 0.62, versus τ = 0.77 for the same number of implied comparisons drawn from plain battles. Each restaurant appeared in only about three triplets, the trios were kept close in rating, and ranking three things from memory took me roughly five times longer per screen than a single choice. Worth having in the toolkit, and worth referencing later, but not the free lunch the information count promises.

the Plackett-Luce likelihood

For a ranking $a \succ b \succ c$ , peel favourites off the top:

P(a \succ b \succ c) = \frac{e^{s_a}}{e^{s_a} + e^{s_b} + e^{s_c}} \times \frac{e^{s_b}}{e^{s_b} + e^{s_c}}

First the three-way choice of the favourite, then a Bradley-Terry battle among the rest. Chain the same trick for rankings of any length, and note what it buys: Plackett-Luce and TrueSkill can consume ranked lists natively, while ELO would have to smash a triplet into artificial pairs and pick a fictional order to process them in. That asymmetry is exactly why this method earns its place here.

The blunt instrument: just score everything

The simplest alternative of all: I typed a number from 0 to 100 for every restaurant, presented in shuffled order to avoid anchoring. The whole pass took two minutes, which sounds like peak efficiency until you look at what I produced.

Plate X

All 49 scores on one line, stacked where they collide. Half the scale went unused, and the middle collapsed: I handed out '79' six separate times. A scale that keeps saying 79 cannot order the 79s, which is exactly the tie problem the battles never have.0-100 scores, shuffled order · 2 minutes

This is the scorer-versus-chooser problem from chapter 02, now in chart form. Nothing below 10, nothing above 91, and a traffic jam in the high seventies. How this stated ranking squares with what my battles revealed is chapter 05’s opening question.

The internet: everyone else’s ranking

There is one more rating system available for every restaurant on my list: the court of public opinion, Google. Comparing its stars to my battles is the closest thing this project has to a control group. The internet and I agree at τ = 0.18. Nearly orthogonal.

Plate XI

Every restaurant by Google stars (dot size is review count) against my Bradley-Terry rank, split at the two medians into quadrants. The shaded corners are where the internet and I disagree: top-left, places I ranked high that Google shrugs at; bottom-right, crowd-pleasers my battles demoted. The white-ringed dots are the biggest standardised disagreements.google ratings, july 2026 snapshot · z = studentised residual

Three things drive the gap, all visible in the chart. First, compression: 43 of 48 Google ratings live between 4.0 and 4.9, and a scale where nearly everyone scores 4.5 carries almost no ordering information. Second, rating culture: within Australia alone the agreement roughly doubles, and Japanese ratings run systematically lower, so mixing countries adds noise that has nothing to do with food. Third, we want different things. TUDOH, a tiny Hiroshima yakitori bar and my #5, sits at Google 4.0 with 28 reviews. Palermo, a Melbourne steakhouse with 4.9 stars and more than four thousand reviews, landed at #36 on my list. Review count measures fame and consistency at scale. It does not measure my palate, and it is not trying to.

a fragile tau

That 0.18 deserves its own uncertainty bar. Google’s scale is so tie-riddled (dozens of restaurants share the same 4.5) that the answer depends on how ties are handled: tie-aware τ-b on the raw ratings gives 0.18, while breaking ties into a forced ranking gives 0.26.

Five rankings now sit on the table. Time to compare them all.

chapter 05 · every ranking fights every ranking

The showdown

I like to think I am a man of my word: what I say is what I do. My rankings put that to the test. Economists call it stated versus revealed preference. My sliders and tier list are stated (“how good do I say it is?”); my battles are revealed (“what do I actually pick?”). The two agree at τ = 0.75, which is high, but the disagreements are where it gets interesting.

Locavore NXT is the telling case. I scored it 79 and my old tier list had it in S. Under forced choice it landed 10th. The memory of the experience, a clandestine upscale restaurant in Bali you approach through rows of trees, outranks the food when I score in the abstract; head to head against something I would actually eat tonight, it kept losing. And then there is the flagship result of the whole experiment.

Chur Fish and Chips, a paper-wrapped $ takeaway, finished #2 by revealed preference, above half the fine-dining list.

I did not know I believed that until the battles made me say it, 9 times out of 10.

Plate XII

Every ranking against every other, darker is closer. The battle-fed methods form a bright block; Google sits alone in its pale row. Flip to ρ and every number inflates: same data, a more generous ruler.tie-aware τ-b · spearman ρ · full pairwise matrix

Line every ranking up against every other and the surprise is how much they agree. Every method built from my own judgment clusters tightly; only Google, alone in its pale row, stands apart. Different maths, mostly the same dinner.

So what was it all for?

the takeaway

What I actually learned

Forty-three minutes of clicking, five rating systems, and one long essay later, here is what survives.

Ranks without uncertainty are lies. Lexy is “rank 7”, and rank 7 means somewhere between 2nd and 19th. Print the interval. If a list will not tell you how wide its ranks are, assume wide.
Forced choice beats scoring. The slider pass took two minutes and mostly photocopied my tier list, pileup and all. The battles took forty-three and surfaced a $ fish-and-chips shop at #2, a thing I believed and had never once said. Ask people to choose, not to rate.
Check transitivity before promising a total order. Nearly five percent of my fully-fought trios are rock-paper-scissors cycles. A ranking is a projection of something messier, and that is fine, as long as you say so.

One more thing worth knowing: none of this machinery is retro. Lichess runs Glicko-2, ELO’s descendant with an uncertainty term. Dota 2 switched its matchmaking to a Glicko variant in 2023. League of Legends runs a hidden matchmaking rating descended from Elo underneath its public ranks. Halo still runs TrueSkill’s successor, tuned by predicting held-out matches (Minka et al., 2018). And Call of Duty’s skill-based matchmaking is so enticing that when Activision quietly weakened it for half its players, the players it protected got worse retention and came back complaining. Every lobby you have ever been matched into was a $\Phi(t)$ with a marketing team.

As for the restaurants: the season you replayed at the top of this essay is frozen now, a little world where Vue de monde goes 10 and 0 forever. But I keep eating, and the living ladder, with every restaurant on a map you can fly around, lives at /restaurants and will keep evolving.

Anthony Bourdain

If I’m an advocate for anything, it’s to move. As far as you can, as much as you can. Across the ocean, or simply across the river. Open your mind, get up off the couch, move.

coda · the sources

References

Rating systems & rank comparison

the paired-comparison lineage, chess Elo through Xbox matchmaking, plus the rulers used to compare the rankings they produce

Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029
Elo, A. E. (1978). The rating of chessplayers, past and present. Arco Publishing.
Herbrich, R., Minka, T., & Graepel, T. (2007). TrueSkill: A Bayesian skill rating system. Advances in Neural Information Processing Systems, 19, 569–576. https://papers.neurips.cc/paper/3079-trueskilltm-a-bayesian-skill-rating-system.pdf
Luce, R. D. (1959). Individual choice behavior: A theoretical analysis. Wiley.
Maystre, L., & Grossglauser, M. (2015). Fast and accurate inference of Plackett–Luce models. Advances in Neural Information Processing Systems, 28. https://papers.nips.cc/paper/5681-fast-and-accurate-inference-of-plackett-luce-models
Minka, T., Cleven, R., & Zaykov, Y. (2018). TrueSkill 2: An improved Bayesian skill rating system. Microsoft Research Technical Report, MSR-TR-2018-8. https://www.microsoft.com/en-us/research/wp-content/uploads/2018/03/trueskill2.pdf
Plackett, R. L. (1975). The analysis of permutations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 24(2), 193–202. https://doi.org/10.2307/2346567
Silver, N., & Fischer-Baum, R. (2015). How we calculate NFL Elo ratings. FiveThirtyEight. https://fivethirtyeight.com/features/introducing-nfl-elo-ratings/

The psychology of choosing

why forced choice beats scoring: a century of evidence that humans compare well and rate poorly

Stewart, N., Brown, G. D. A., & Chater, N. (2005). Absolute identification by relative judgment. Psychological Review, 112(4), 881–911. https://doi.org/10.1037/0033-295X.112.4.881
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. https://doi.org/10.1037/h0070288