llmsort and the other LLM sorts: what each yields, what each cannot tell you.

llm-sort llm install llm-sort · Python · Apache-2.0

A plugin for Simon Willison's llm CLI with an interface like GNU sort: lines in, lines out, ordered by relevance to a query. The model is asked, per pair, “Which line is more relevant to the query? Please answer with ‘Line A’ or ‘Line B’,” and Python's sorted uses that as the comparator; a sliding-window mode exists too. Ranking techniques cite the PRP paper.

Where it is the better choice: a quick relevance sort of a short file from the shell, in the tool you already use. What it cannot tell you: how sure it is, whether the answer depended on which line was A, how much better line 1 is than line 2, or what it cost. A comparison sort also assumes transitivity: one wrong answer propagates through every subsequent split. The field guide has the measured sizes of those effects; they are not small.

Sorting with TrueSkill blog post · Python trueskill

The closest cousin. Batches of items are given to the model to order; each batch is treated as a multi-player game and TrueSkill updates a Gaussian skill (μ, σ) per item; the final rank uses μ − 3σ. The author notes that 1–2 matches per item can suffice and that high-σ items should be scheduled next. This is the right instinct — batches, a probabilistic aggregate, uncertainty, adaptive scheduling — and llmsort's setwise path is the same shape, measured.

The differences: TrueSkill is an interval scale from wins (it cannot say “twice as good”); position bias inside the batch is not addressed, and it is the largest single error source we measured in lettered slots (the last slot was ranked last at 2.2× its fair share); there is no gauge for whether the batch orders were stable under re-presentation, which is the one number that predicts agreement with a careful sort (flip < 0.20 ⇒ ρ ≥ 0.64 over 38 cells); and cost is not tracked. If you like this approach, llmsort sort --setwise is it with the warning light attached.

raink and SiftRank Go · OpenAI or Ollama

Bishop Fox's ranker, built for triaging security findings: batches of ten items, ranked by the model, shuffled and repeated (ten passes by default), positions averaged, then the top recursively refined. The successor paper, SiftRank, adds convergence detection and reports 2,197 items in 99 seconds for $0.82. It is fast, practical, and the strongest engineering rival on this page — and it is the same shape as llmsort's setwise path.

The differences: averaging positions over shuffled passes treats disagreement between passes as noise to wash out; llmsort treats it as the measurement (the flip rate is the gauge that predicts whether the order can be trusted at all, and σ per item says which ranks are real). raink reports no uncertainty and no cost; it has two providers (OpenAI, or a local Ollama model); and position bias is handled by shuffling rather than measured. If your list is a thousand findings and you want a plausible order in two minutes, raink is a good tool. If the decision downstream needs to know how far to trust rank 12 over rank 13, it cannot tell you.

llm-sorter pip install llm-sorter · Python · OpenRouter

Merge sort with the model as comparator, against any OpenRouter model. The README is admirably honest: it names non-transitivity (“A > B, B > C, C > A cycles”), length, style and positional biases, and run-to-run nondeterminism — and the cost: ~664 comparisons for 100 items. Naming the problems is where it stops; nothing measures or cancels them, and one wrong comparison propagates through the merge. The same 100 items on llmsort's default budget are 400 comparisons with both orders counterbalanced inside that budget, or ~34 setwise calls (two rounds of 17 windows).

Sato, “Sorting with LLMs” technical report · February 2026

The best theory explainer in the space, and the page that ranks first for “sorting with LLMs.” It lays out the pointwise / pairwise / listwise / setwise taxonomy, the cost model, sliding windows, tournaments, top-k insertion, and it makes the recommendation most people should hear first: “start with either the sliding window method or quicksort (KwikSort)” — KwikSort because it keeps a guarantee even when comparisons are intransitive, and parallelises. The worked example (295 headlines, 2,324 comparisons, $0.85 on GPT-5-mini) is exactly the kind of denominator this site insists on.

What it does not cover is the instrumentation half: uncertainty per item, choosing the next comparison by where the posterior is thin, per-model reliability, and the gauges that say whether the judge read the criterion at all. Those are the subject of the field guide; read the two together.

LOTUS sem_topk and the database line pandas accessor · VLDB 2025

The academic answer to “LLM ORDER BY”: df.sem_topk(instruction, K, method=quick|heap) embeds pairwise prompts inside quicksort or heapsort and returns the top K, with a cheap-model-then-big-model cascade to control cost. The follow-on papers are the clearest published statements of the budget-versus-accuracy knob — Access Paths for Efficient Ordering treats the choice of sorting algorithm as query optimisation, and Are Optimal Algorithms Still Optimal? re-costs bubble, quick and heap by inference count under batching and caching (classical optimality inverts). None of them return an uncertainty per item, and the comparisons are executed as control flow rather than fused as evidence, so a wrong answer is not a measurement, it is a branch taken. If your items already live in a DataFrame and you want the top K of a semantic predicate, LOTUS is the natural fit.

ProposalArena and pairwiseLLM the closest cousins

Two small projects sit in exactly llmsort's niche. ProposalArena runs pairwise LLM judgements over research proposals and fits Bradley–Terry; pairwiseLLM (R) is the only other tool found that chooses the next pair adaptively by information. Both get the aggregation right and stop at an interval scale from wins. pairwiseLLM also checks forward/reverse consistency and estimates cost before a run; ProposalArena does neither, but it validated against human reviewer rankings (arXiv 2512.10895). Neither reports a per-item σ beside a stability gauge, and neither asks for magnitude. They are the right instincts with part of the instrumentation left off.

llm-gpt-sort, LLMSort, and “just ask it to sort” proof-of-concept

Send the whole collection, ask for it sorted. llm-gpt-sort documents that it stops working on bigger numbers and longer words. Johnston's LLMSort measured the failure carefully on alphabetical sorting with gpt-4o: 50% accuracy around 36 words, 0% at 500, at $0.027 per sort — and drew the right conclusion. We measured the same thing on semantic criteria (E12): single-call listwise is fine exactly when a gauge would have said so, one malformed reply loses everything, and the lettered alphabet caps it near 26 items. It is the k = n special case of setwise minus every safety net.

rank_llm, llm-rankers, avnlp/prp — the IR toolkits Python

Serious toolkits — for a different problem. These rerank retrieved passages for a query (MS MARCO, TREC, BEIR): sliding windows of twenty, or pairwise heapsort and setwise picks, evaluated by NDCG; llm-rankers is the reference implementation of the Setwise paper, and avnlp/prp is a careful PRP with both presentation orders and ties on conflict. They are the strongest published tools for that task, and llmsort does not compete with them there. They do not aim at “sort my grant proposals by expected impact”: no criterion-as-attribute over arbitrary items, no per-item uncertainty, no cost accounting, no order-sensitivity gauge. If your list is passages and your criterion is a query, use rank_llm or llm-rankers.

Elo and Bradley–Terry over an LLM judge the Arena pattern

Collect pairwise wins, fit a logistic strength. This is the aggregation most home-grown pipelines land on, and it is sound as far as it goes: an interval scale with confidence intervals if you bootstrap them. What it discards is magnitude — a 1.1× and a 20× preference are both “a win” — and what it does not do is choose the next pair, cancel position bias by design, or tell you whether the judge's answers were consistent enough to fit at all. llmsort's solver accepts direction-only evidence too (that is what setwise lowers to), so the Bradley–Terry case is contained in it; the ratio question is what it adds on top when the judge can answer it.

llmsort.com unrelated

An email-sorting product using natural-language rules. Not a list-sorting tool; listed here only because the name collides.