How Typical works, what we measured, and the control that says whether a model read the question at all.
Software that routes a ticket or rates an alert wants one thing from a model: a probability for each option it already knows about, and a way to say that none of them fit. What it usually gets is a sentence, which it then has to parse.
Typical reads the probability out of a pretrained causal language model. The state is encoded once. Each question is a short suffix carrying its own candidates. A head partway up the stack scores the end of that suffix against each candidate. No token is sampled, and the answer is one of the options you passed or an explicit abstention, by construction.
This page is the paper with the proofs left in and the notation taken out. Every number on it is transcribed from that paper; the paper says where each one was measured.
The state goes in once as an ordinary causal prefix and we keep its key-value cache. Every question after that is a short suffix: the question, the candidates rendered as lettered lines, and a terminal decision token. The suffix attends to the cached state; suffixes batched against one state do not attend to each other.
A decision is one forward pass over the state, the question and the candidates together. The expensive half comes from cache.
A decision is a tuple: a state S, a question Q, a candidate set C supplied at request time, and a type. The model returns a probability for each candidate and one more for abstaining, and they sum to one.
Every candidate you hand in gets a probability, and so does refusing to answer. They add to exactly one, so the model always accounts for all of its confidence — none of it goes missing, and abstaining competes with the answers rather than sitting outside them.
No token is sampled. C is text, supplied per call, so the model has no output index for any label and has to read the candidates to score them.
At the tap layer the pass gives a terminal decision state hD and, for each candidate, a contextual state hj: the mean hidden state over the tokens where that option was rendered. Both are projected, and the score is a bilinear form with a small learned correction on the elementwise product.
The decision and each candidate are projected into the same small space, then compared three ways: how far they point in the same direction, a learned weighting of where they agree dimension by dimension, and a small network reading that same agreement. Adding the three gives one score for that candidate.
g is a two-layer MLP. The option letters stay in the rendered text and are never read out.
Abstention is a gate outside the candidate softmax, not a line in the list. It reads permutation-invariant statistics over the scores — maximum, top-two margin, mean, log-mean-exp — plus the projected decision state, and returns a single r.
Softmax turns raw scores into probabilities, and T sets how sharp: low T commits to a winner, high T spreads the mass. Then r is held back as the chance of not answering at all, and every candidate is scaled by whatever is left over.
Two properties fall straight out of that shape, and both are exact rather than measured.
Pairwise odds do not depend on abstention
Take any two candidates. The factor (1−r) multiplies every non-null candidate, so it cancels:
Put any two candidates over one another and the (1−r) sits on the top and the bottom, so it cancels. However likely the model is to abstain, it can never reorder the candidates — which is why you can drop the abstention and reuse the rest unchanged.
So how likely the model is to abstain never reorders the candidates, and a caller who has already decided not to abstain can use the renormalised distribution unchanged. The scores themselves stay set-dependent; that is the part the gate does not touch.
The gate is temperature-invariant
T enters only the candidate softmax, and r reads untempered statistics. Recalibrating temperature therefore never moves p∅ — you can tune the sharpness of the answer without disturbing the decision to answer at all.
The types are contracts on that distribution. Choice is the K-way readout. Score reuses it over ordered levels with a smoothed target, t̃j ∝ exp(−|j−y|/τ) at τ = 0.7, which adds no parameters and makes a near miss cost less than a distant one. Noul takes a dedicated Bernoulli head, P(yes) = σ(wn⊤hD), on a suffix that renders no candidate text at all — which is why label order cannot enter its computation.
M decisions about one state cost O(|S|) once, plus M·O(|Q| + |C|) for the suffixes, through only the kept 71% of layers. That is the whole reason the second question is cheap.
Once the pass is done there is more than one place to look. N1 reads the next-token logits of the option letters. That is the standard multiple-choice interface, never sampled. N2 scores the decision state against a frozen embedding of each label's text. N3, the one Typical ships, scores it against each candidate's own contextual state: the hidden state over the tokens where that option was rendered, inside the same pass.
All three arms train on the same data for the same steps with the same option shuffling. They differ in what the decision state is scored against.
N2 is the informative failure. Give the model the label's meaning but not the slot it was rendered in and among-K accuracy drops 10 to 12 points, half the question-conditioned signal goes, and on Banking77 with 77 options it collapses to .012 against N3's .497. What the decision state has learned is partly which rendered slot holds the answer, and an independently embedded label cannot carry that.
N3 keeps N1's question-dependence (.106 against .109 across seeds) and is far steadier when the option set changes. Add three unrelated options and N1's pairwise log-odds move by .44 to .46; N3's move by .09 to .13, a quarter as much.
A decision model can score well above chance on a benchmark without reading the question at all, by learning which option strings tend to be correct together. Accuracy will not tell you. Calibration will not tell you either: one of the variants below has an expected calibration error of .04.
So we measure Δq: accuracy with the real question, minus accuracy with the question replaced by an unrelated one. A model that scores the same either way has not used the question. Below is every variant we tested, ranked by accuracy. That is the view a leaderboard gives you.
12 variants · MMLU-Pro, n = 1,200
Six variants computed the decision state before seeing the candidates. All six sit at zero. Every readout that processes the candidates inside the forward pass carries .09 to .12, and so does the listwise teacher they were distilled from.
We tried to move it and could not. Cross-attention over every candidate token: −.003. Supervising the change in the teacher's log-odds across seven candidate sets: exactly zero. Reading all 28 layers instead of a mid-depth tap: −.008. Their accuracy of .12 to .17 sits well above the .10 chance rate and comes entirely from priors over which labels go together.
One alternative we cannot rule out is optimisation: the zero-initialised set-conditioning attention stayed inert through 4,000 steps of direct supervision. This may be hard to train rather than impossible to represent.
We truncate the backbone at about 71% of its depth and never compute the layers above. That was not an efficiency decision.
Our first full-scale model read the final layer and scored 58.6 on SNLI. That is the hypothesis-only baseline: the number you get by ignoring the premise and guessing from the sentence you are asked about. The last layer had specialised into predicting the next token and had stopped carrying the evidence.
Mid-depth carries both the evidence and the question-conditioned knowledge in one model. Every released and ladder checkpoint taps there, and LoRA at rank 16 adapts the top eight kept layers while everything below stays frozen.
A decision model that memorises “this kind of ticket gets that label” is worthless the moment the policy changes. So most of the workflow training is counterfactual: the same state and the same option list appear under different rubrics with different correct answers, and only the rubric decides.
Counterfactual training is the largest training-distribution effect we measured, larger than anything we changed in the architecture. Held-out families, grammars and rubric styles go from .481 / .507 / .491 to .827 / .881 / .888. Held-out rubric-flip accuracy goes from .477 to .710.
The control is the part worth reading. Accuracy under a shuffled rubric falls from .446 to .405. The model got worse at answering when we handed it the wrong rule, which is what depending on the rule looks like.
It works inside the grammar we generate and stops at its edge. Level-7 composition, which mixes temporal, unit and expected-value reasoning and which our generator never produces, moves from .477 to .498. The curriculum that moved every in-distribution family by thirty-odd points moved that one by two.
Training truncates long states from the right. We rendered the supporting facts at the end of every row of our long-policy corpus, so most of those rows lost their evidence before the model ever saw it. We were training the model to answer long policies from states that no longer contained the answer. It learned to expect the case block at the end, and learned not to look anywhere else.
Two arms with byte-identical flags, same 23,318 rows, rendered facts-first or facts-last, scored on 605 held-out long states with nothing truncated at evaluation time. Facts-first adds 11.2 points, p = 5×10−10, and the gap replicates at a second seed.
That is a rendering choice, not a modelling one. It is the kind of thing that hides inside a data pipeline and reads as a capability limit for months.
Every trained checkpoint against its own frozen control: the same backbone, no training, letter logits, three examples in the prompt.
Training is worth 15 to 20 points of standard-tier accuracy at every size we ran, and the 14B reaches .931. The shipped 4B reaches .861 against its frozen control's .764.
Read the intervals before reading the ranking. The standard tier is 72 items drawn from 36 states written two ways; adjacent rows on this chart are not separable, and we do not treat them as if they were.
And on the hard tier the ordering inverts. The frozen 14B scores .559 where our trained 14B scores .450. Training helps on the tier we built data for and hurts on the tier we did not. That is the clearest open problem in the project.
In the research harness a single decision at two options takes 45 ms at 1.7B and 60 ms at 14B. On the optimised serving path a warm decision is 15.5 to 17 ms at 1.7B and 19 to 21 ms for the Qwen3-4B model. The Qwen3.5-4B model measures 34 to 46 ms, on reference DeltaNet kernels that nobody has tuned yet.
For a loop, what matters is the next decision. The state is already cached, so each further question about it is a short suffix and costs a few milliseconds.
The paper has the derivations, the full tables and the appendices this page skips. The weights are on Hugging Face and the inference package needs torch, transformers, safetensors, huggingface_hub and numpy.
pip install typical-aifrom typical_ai import Typical
m = Typical.from_pretrained("OzLabs/typical-medium", device="auto")
m.choice(state, "Which team should own this ticket?", ["billing", "shipping", "support"])If you find a decision it gets confidently wrong, that is the interesting case and we would like to see it.