What AI accuracy hides

We use language models to read scientific papers and label what is in them, so that the absence of a label can be treated as evidence of a gap in what has been studied. That only works if the labelling is both accurate and complete. A label the model invents is a false finding. A label it misses is a gap that isn’t there.

Two numbers govern that. Precision is: of the labels the model applied, how many were correct. Don’t say things that aren’t true. Recall is: of the labels it should have applied, how many it actually caught. Don’t miss things that are true.

Almost every quality number you see reported is precision. It is the cheap one: take what the system produced and check it. Recall is expensive, because measuring it means knowing what the right answer was, which means building an answer key before you can score anything at all.

So we measured precision, it looked good, and we shipped. Four findings later, here is what that cost us.

One. A 97% score concealing a third of the work

Our pipeline runs two models. A tagger proposes labels and must quote the sentence it relied on. A judge, which is a different model, tries to knock each one down. We tested three versions of the judge’s instructions on the same 40 papers, so the rows compare directly.

Accuracy audited blind by a third model, shown only the abstract and the label’s definition, never the tagger’s quote.

Read the first two rows and it is the familiar trade-off: relax the strictness, get more labels, lose accuracy. 97% down to 83%.

The third row breaks it. Against that second version, handing the judge the same written rulebook the tagger had raised accuracy 83% → 91% and halved the wrongly-killed labels 24% → 10%, at the same richness. Better on both counts at once.

The diagnosis was mundane once someone looked: the judge had never been shown the rulebook. It was rejecting labels by comparing them against its own guess at what each one meant. The strictest version threw away nearly a third of perfectly good labels, and its 97% score concealed that completely.

A reviewer without the standard isn’t strict. It’s arbitrary, and it will score beautifully while being so.

Two. The ceiling was a token limit

Precision held in the mid-eighties across three separate corpora. Recall, once we finally built an answer key and measured it, was 53%. The pipeline was finding about half the clearly-supported labels.

We expected a model-capability problem. It was three pieces of plumbing.

The second row stings. Raising the judge’s output limit, one number with no prompt change and no model change, bought twelve points of recall. The judge’s verdicts were truncating mid-JSON, and any label whose verdict got cut was silently dropped as unconfirmed. Confirmed labels were being thrown away by a character budget.

The third row is the fix that mattered most, and it is embarrassing in hindsight: the tagger’s instructions were pure restraint. Paragraphs on what not to over-claim, and not one word telling it to be thorough. Adding “assign every value the text supports; omitting a grounded label is as much an error as inventing an ungrounded one”, plus room to write the output, moved recall from 53 to 75 while precision held.

Three. The bigger model was worse

That last row deserves its own heading. Holding everything else constant, we swapped the small, cheap tagger for a larger one from the same family. That is the obvious move, since more capability should mean it spots more.

Recall went down: 75 to 67. Cost doubled.

The larger model systematically under-asserted the plainest classification questions (how long the study ran, whether it was basic or applied, what spatial scale), recovering 18%, 27% and 34% of them respectively. It appears to have treated them as too obvious to be worth stating. The cheaper model, told explicitly to be complete and given room to answer, beat it on every axis we measured.

Recall was capped by what the generator was told to do and how much room it had to say it, not by how much model we bought.

We are not claiming small models are generally sufficient; this is one ablation, one task, one corpus. The narrower claim is the useful one: we would have bought the bigger model on intuition and been worse off, and only the measurement told us. The upgrade was available, obvious, and wrong.

Four. Nothing was checking the answer key

The three findings above are all about the tagger. Then we looked at the thing doing the marking.

Every stage of our pipeline has a second model checking the first. The tagger has the adversarial judge from finding one. The step deciding whether a paper is really about a given species has one. The rulebook itself now gets reviewed by a separate model before use.

The answer key had nothing. It is produced by one model reading an abstract and listing what it believes is true, and whatever comes back becomes correct by definition. That is what makes it an answer key. There is no stage after it to disagree.

So we pointed our own adversarial judge at it: same model, same prompt, same per-label rules it uses to vet the tagger every day. We asked it about the labels the answer key said we had missed, the ones being counted against us.

Control: the same judge, shown labels both the answer key and the pipeline agreed on, with the supporting quote stripped so it worked from the same information, refused 7%. So the 41–50% is not the judge being harsh about missing quotes.

The control arm is what makes this readable. Answer-key labels carry no supporting quote, so a judge might reject them for that alone. Strip the quote from labels everyone already agreed on and it rejects 7%. The remaining gap is the answer key’s own content failing the same standard we hold our output to.

We had been holding the answer key to a lower standard than the work it was grading.

We want to be careful about the conclusion. It is tempting to read this as “our numbers were better all along,” and that is not what we think it shows. It shows we do not know what the real number is. We know the one we had was measuring against something unvetted, in five subject areas, for months, and that the reason nobody caught it is that an answer key is the one artefact you are not supposed to question.

The check is also deliberately one-directional, which is worth saying plainly rather than burying: it examines only the labels the pipeline failed to produce, so it can only ever move the score upward. Our reasoning is that a label both the pipeline and the answer key produced has two independent sources behind it, while one the answer key produced alone has one, and a single-sourced claim needing verification is the same rule we already apply to the tagger. But a correction that can only flatter you deserves suspicion, so it stays opt-in, it prints the before and after side by side every time, and a label the judge declines to rule on is left standing.

The common thread

Four findings, all pointing the same way. A judge that scored 97% while discarding a third of the good work. A quality ceiling that turned out to be an output limit. A model upgrade that made things worse at twice the price. And an answer key, in five separate subject areas, that half the time could not defend its own corrections.

None were subtle once found. All survived months of looking like success, because the number we were watching, accuracy on what we kept, is structurally incapable of seeing them. Everything these failures cost had already been deleted by the time the score was computed.

The operating rule we took from it is simple to state and awkward to follow: audit the set you discarded, not just the set you kept. It costs more, because it means building an answer key before you have anything to score against. It is also the only way the first three became visible.

The fourth adds an uncomfortable corollary. Ground truth is a claim, not a property. Ours was assembled the same way everything else in the system is assembled, by a model reading a document and producing an answer, and we exempted it from scrutiny purely because of the job we had given it. Whatever you measure against is also a measurement, and it deserves the same adversarial treatment as the thing it is grading.

Every number here came from our own logs. Samples are deterministic, so before-and-after comparisons are like-for-like, and three independent models are used across the tagger, the judge and the blind auditor. The measurements are specific to this pipeline and this corpus; what generalises is not the figures but the method that produced them.