Sorting with LLMs: the vocabulary, the failure modes, and the number that measures each.

Ranking ordinal

An order and nothing more: A above B above C. It tells you nothing about gaps. Most LLM-sorting tools stop here — and most decisions cannot, because “fund the top 5” needs to know whether item 6 was a coin-flip away from item 5.

Scaling cardinal · interval or ratio

Every item placed on one shared scale, so the gaps are numbers. An interval scale (Elo, Bradley–Terry, TrueSkill) makes differences meaningful but not ratios: Arena scores are interval, you cannot say “A is twice as good.” A ratio scale supports exactly that claim — and is only obtainable when the elicitation itself carries magnitude. A ranking is a scaling with the spacing deleted.

Top-k a boundary, not an order

“The best 10 of 150.” Cheaper than a full order if the method knows to spend its comparisons at the boundary — and intrinsically less stable than people assume: the flagship pairwise sort reproduces its own top-10-of-150 across seeds at only 0.3–0.7 at the default budget, and no other method beat that. measured E14. Any top-k claim without a stability number is unmeasured.

Reading magnitude ± σ

What a good instrument returns per item: a position on the shared log-ratio scale and a posterior standard deviation. Two items whose gap is inside their joint 2σ are not ordered by the data; they are ordered by presentation. The error bar is the honest answer to duplicates, near-ties, and “is #6 really below #5?” — questions no bare rank can answer.

Does a one-dimensional scale even exist for your attribute?

Before sorting by “prose beauty” or “employee performance”, it is fair to ask whether a single shared scale is there to be found, or whether quality is plural and any order is an artefact. Measured: six attributes × four judges (gpt-5.6-terra, gemini-3.1-pro-preview, claude-haiku-4.5, qwen3.7-max), 202 calls per judge-pool, $19.18 in total.

Scale existence is the norm when items span real quality variation: the textbook taste attribute agrees across four judges at .897 and the flagship contested one at .984 — the battery's highest. What contested attributes do instead is depress a judge's agreement with itself on closely matched, criteria-trading items. The one truth column that splits is code correctness, and it splits by capability (terra .843, gemini .916 vs haiku .434, qwen .277): the judge, not the attribute, was the limit. One caveat on the qwen column: its provider caps top logprobs at 5 and needed reasoning disabled (OPENROUTER_DISABLE_REASONING=1). measured scale-existence-battery-2026-08-31.

Pointwise “rate this item 0–100”

One item per call, a score back. Cheapest per item, O(n) calls, and the answer everyone tries first. The scores are not comparable across calls — no shared anchor — and they cluster: one judge compressed sixteen rivers into three distinct values, sorting a close-packed pool worse than chance (truth-ρ −0.19). breaks tie blocks, anchor drift, no error bars. measured E12. As a top-k screen it is disqualified: tie blocks silently dropped up to 70% of the reference top-10 at the slice cut (E14).

Listwise “sort this list”

The whole list in one prompt, an order back. One call, so it feels free. It is the k = n special case of setwise ordering minus every safety net: position bias (the answer depends on the order you pasted), context limits, silently dropped or hallucinated items, and one malformed reply loses everything. Hard-capped around 26 items by the lettered-slot alphabet. It is fine exactly when the gauge would have been clean — and you have no gauge. measured E12.

Pairwise ordinal “which has more X, A or B?”

Two items, a direction back. The gold standard for reliability — a model reads two things side by side far better than it rates one in isolation — and the tempting way to spend O(n²). It throws away how much. It also inherits a position bias: the second-presented item wins more; in lettered slots the last slot was ranked last at 2.2× its fair share, and the first two slots were under-picked for first. The fix is to ask both orders and measure the flip rate, not to assume it away. On small judges (under ~30B parameters) this is the instrument that works: seven of nine small models could not read a magnitude ladder but agreed with the reference order at +0.40–0.80 (pair-level Spearman) on direction. measured judge bakeoff.

Pairwise ratio “how many times more X does A have than B?”

Two items, a ratio on a fixed ladder back (1.0 … 26×). Each answer is a noisy measurement of s(A) − s(B) in log space, so answers compose along paths: sparse comparisons cover the list and over-determine a global fit with free consistency checks. This is the instrument behind llmsort. It needs a solver and a judge that can actually read the ladder — the bet pays on some (model, attribute) pairs and loses on others, and the tool measures which. Judges also compress: true log-ratios come back at about ⅔ of their real magnitude (truth slopes 0.55–0.68) wherever rank is good (E2), so the scale is faithful in order and shrunk in spread. On the countries pool with known truth the pairwise sort tracks reality at rank correlation 0.87–0.97 (the setwise ring 0.86–0.91); on the close-packed rivers pool it fell to 0.31/0.10 while setwise held 0.55–0.71 (E12) — the pool decides as much as the method.

Setwise k items in lettered slots, one full order back

Six to eight items per call, lowered to k(k−1)/2 ordinal observations. At k = 6–8 it matches the same model's pairwise test–retest band at about ⅓ the cost, across four models and two corpus families, on items from 400 to 8,000 characters. A 24-item sort costs $0.0054 live. The load-bearing addition is its own warning light: re-present each subset shuffled and count direction flips. measured E6, 38 live cells. Best–worst scaling (pick the best and the worst of k) was refuted as built: the worst-pick is weak (ρ median 0.25).

Logprob evidence the model's whole answer distribution in one call

When the answer is a single token (a letter on the ladder, or A/B/=), the provider's top-k logprobs are the judge's entire probability mass over answers — a full distribution for the price of one point answer. Measured effect on a single-token rail: ±0.020 vs ±0.464 statistical error at identical cost, a 23× reduction (E9). The same trick on multi-token answers made things worse in 10 of 12 cells: sequence probability measures fluency, not conviction (E7). Not every provider exposes logprobs, some expose ones that disagree with their own sampling, and reasoning models usually refuse them.

Position bias / slot bias

The item presented second wins more than it should, and lettered slots carry their own bias — the last slot was ranked last at 2.2× its fair share in one measured judge, with the first two slots under-picked for first; +0.98 nats (natural-log ratio units) of pure slot preference in another. Some judges are “slot-locked”: 92% “A ahead” regardless of content. gauge ask both orders; report order agreement (null: 50%) and the order residual in nats. Cancel by design (counterbalance, rotate, Latin square), never by regression after the fact.

Tie blocks (rating collapse)

Pointwise scores pile onto a few values: sixteen rivers → three distinct values. Everything inside a block is unordered, and a top-k cut through a block drops arbitrary members. gauge count distinct values; compare truth-ρ on a pool with known ratios.

Order sensitivity (listwise and setwise)

Re-present the same subset shuffled; the returned order changes. gauge direction-flip rate across re-presentations. One-sided screen from 38 live cells: flip < 0.20 ⇒ agreement with the pairwise reference ρ ≥ 0.64 (median 0.79), and every bad cell was flagged.

Intransitivity and cyclic residual (curl)

A ≻ B ≻ C ≻ A. With ratios: ln r_ab + ln r_bc + ln r_ca should be ≈ 0 around every triangle; the part that isn't is the cyclic (Hodge curl) fraction. On anchor pools curl told a truth-ρ 0.965 judge from a 0.350 judge (0.039 vs 0.129) when the composite band could not. gauge curl fraction ≤ 0.10, and stochastic-transitivity violations deeper than 2 SE over triads with repeat draws.

Reference-frame ownership (the pivot halo)

Ask for k − 1 ratios against one reference item and the reference owns the answer: 795 of 870 ratios came back below the pivot (mean −0.95 nats), and rotating the pivot flipped 80–93% of implied pair directions. Cheap observations are worthless if the frame decides them. measured E1.

Magnitude compression

Judges return log-ratios at about ⅔ of their true magnitude (truth slopes 0.55–0.68 on anchor pools) wherever rank is good. The order is right; the spread is timid. Readings are comparable to each other, not to physical units, unless calibrated on anchors. It is not universal: gemma-4-31b on country populations, read through the logprob rail, returned slope 0.999 against truth — no compression at all — so calibration is per (judge, instrument, pool). measured E2, perturbation spectrum.

Dropped, duplicated, and hallucinated items

Listwise answers omit items, repeat them, or invent them. One malformed reply silently eats a whole sort. gauge parse coverage as a denominator on every run (e.g. 768/768 calls parsed); refuse loudly rather than print a partial order.

Top-k instability

Subjective top-k is intrinsically unstable for every method: the flagship pairwise sort reproduces its own top-10-of-150 at 0.3–0.7 across seeds. The honest deliverable is the top-k with that number. measured E14.

Degenerate pools

Real lists carry near-duplicates and boilerplate. Identical texts land a median ~13 ranks apart, direction coin-flipping across seeds — inside their error bars (twin gaps exceeded joint 2σ in only 0–4 of 45 pairs per cell). The gauge certifies pool-level order, never item-level distinctions. measured E15.

Scale in n: per-item budget does not transfer

The recipe that works at n = 24 degrades at n = 150 with the same per-item budget (agreement 0.81–0.91 → 0.55–0.71) because the comparison graph's diameter grows. The fix is structural — a second overlapping round — not more budget. measured E13.

Reasoning burn and refusals

Hybrid reasoners silently spend the answer budget as thought and return nothing parseable; some judges refuse 55–80% of calls under a strict answer format. Disabling reasoning effort on one judge improved per-call noise (σε 0.260 → 0.181 nats) and cut cost 13%. gauge refusals and failures as denominators, per model.

Framing sensitivity, ordered (the perturbation spectrum)

How much does the answer move when you change something that should not matter? Measured on one judge (gemma-4-31b, local, logprob rail), 12 countries by population, all 66 pairs × both orientations × 8 nonce suffixes, 6 whitespace jitters and 6 paraphrases — 2,640 of 2,640 calls parsed. The scatter of the answer grows with how deep the perturbation reaches:

Two consequences. The judge's stated answer distribution is an honest forecast of its own framing sensitivity: per pair, the stated sd predicts the paraphrase scatter with correlation 0.865 and slope 0.956 (n = 132) — confidence you can plan with. And on this factual pool the judgement is right, not merely stable: sign accuracy 1.000 over 66 pairs spanning a 2.2% gap to ~280×, rank correlation 0.972, slope 0.999 against true log-ratios. Position is the largest perturbation by a factor of four over paraphrase, which is why counterbalancing is the one design choice that is never optional. The subjective-pool arm of this run is pending. measured perturbation-spectrum-2026-09-06.

The anchored ring

Tile the shuffled pool with overlapping windows so consecutive groups share two anchors and the last wraps; the comparison graph is connected in one round. Structure substitutes for a second round at n = 24 (E11); at n = 150 a second round is what buys the density the per-item budget cannot (E13).

Comparison sorts and KwikSort

Merge sort or quicksort with the model as comparator costs O(n log n) calls and trusts every answer; one wrong comparison propagates. KwikSort (randomised quicksort read as rank aggregation) is the honest version of this family — it keeps a constant-factor guarantee even when comparisons are intransitive and it parallelises, which is why Sato's survey recommends it as a starting point. What no comparison sort returns is an uncertainty: it does not know which of its answers were the noisy ones. Fusing the same comparisons as evidence, instead of executing them as control flow, is what turns the schedule into a measurement.

Active selection

Spend the next comparison where the posterior is thinnest — highest effective resistance on the comparison graph, near the top-k boundary. The planner's efficiency claim was benchmarked against uniform random selection, failed, was fixed with anchor-diverse exploration, and is pinned two-sided in the test suite with history. “Must beat random” is a measurement, not an assumption.

The funnel

“Best 10 of 150”: setwise screen the whole pool, then a certified pairwise refine of the top-30. Brackets the reproducibility ceiling in every cell at 0.3–0.6× the cost of pairwise alone (E14). Shipped as llmsort sort --setwise --top-k K.

Evaluate your own judge, in three commands

1. One pair, the full symmetry group. llmsort judge @a.md @b.md --by "X" --orbit asks the same pair through all eight sign-and-order presentations and reports the invariant component beside every named bias (order, polarity, wording) as a share of the energy; --spin (6 calls) reports whether the belief survives an asker who leans. If the invariant part is not the largest coefficient, this judge is not reading X on these items.

2. The list, with probes. llmsort sort list.txt --by "X" --two-sided --also-by "X, reworded" --trace t.jsonl. Read, in order: order flips a/b (null 50%; the healthy-sort demo flipped 11 of 51); probe [opposite] … consistency and probe [paraphrase] … — ≥ 0.70 consistent, 0.30–0.70 shaky, below 0.30 incoherent for this judge; the error budget line, which splits statistical error from order asymmetry from cyclic energy so you know which the next dollar buys down; and consistency, the expected agreement of an independent rerun (calibrated: 66% predicted vs 74% measured over 11 reruns).

3. Before any budget is spent at scale, the pre-flight. Same pair, both presentation orders: fraction of pairs whose direction agrees. Chance is 50%; on the production ledger (1.16M comparisons, 25 models) a 12B judge on the ratio instrument anti-correlated with itself across orders (19% same direction, order correlation −0.54); its 31B sibling went from 55% and +0.29 on the ratio instrument to 93.5% and +0.905 on the ordinal one. The rule that came out of it: no (instrument, judge) cell consumes budget until same-direction-across-orders clears about 0.7. ledger-audit-2026-09-08.