Insights  /  Research

Did the action work?

A language model can tell you, almost perfectly, that a study tested a conservation or farming action. Ask it whether the action helped or hurt, from the abstract alone, and you are asking a different and harder question. We set a bar for that second question before we measured, and against two independent human-coded evidence bases, our reader did not clear it.

Much of what COMPASS ID does is read a scientific paper and record what is in it: the species, the place, the method, the kind of result. One label we wanted to add is more consequential than the rest. When a study tests a deliberate action, a protected area, a restoration method, a payment, a fertiliser, a planting regime, we wanted to record not only that it tested one, but which way the result went, and whether that was good or bad for the action’s own goal. A map of what has been tried, with the verdicts attached, would be worth a great deal.

So we built a reader for it and, before measuring, we wrote down what it would have to achieve to ship. Two separate questions, two separate bars.

The first question is does this study evaluate an action at all. The second is which way did the action’s effect go, up or down, and good or bad for the goal. We call the first one screening and the second one direction.

The easy half: finding the studies that test an action

Screening is the part a model does well. We checked it against two collections of studies that human teams had already read and classified, one in conservation and one in agriculture, and asked only whether our reader agreed that an action was being evaluated.

Does the study evaluate an action? · agreement with the human-coded set
Evidence baseStudiesReader agreesBar
Conservation actions2160.93n/a
Agriculture (resilient-farming trials)1710.990.90

Unit: studies. The agriculture figure is 169 of 171, measured on a fresh sample committed before the reader saw it. The conservation figure is on the set the reader was developed against, so it is a development number, not a clean test.

This is genuinely useful on its own. It means that across a corpus, we can separate the studies that put an action to the test from the ones that describe, model or survey without testing anything, and we can do it reliably enough to map. That alone answers a real question: where has an action actually been trialled, and where has it only been discussed.

The hard half: which way did it go

Direction is where it came apart. Here the measure that matters is not raw agreement but Krippendorff’s alpha, which is agreement after you subtract how often two coders would have landed on the same label by chance. That correction matters most when the categories are lopsided, and in this kind of evidence they are: most tested actions are reported as having helped, so two coders who both guessed “it helped” every time would agree most of the time while knowing nothing. Raw agreement rewards that. Alpha does not. We set the bar at an alpha of 0.667, a common floor for calling a coded variable usable.

Which way did the action go? · reader against the human-coded direction
Evidence base and readerStudiesRaw agreementAlphaBar
Conservation, first reader (summing each outcome sentence)1160.660.350.667
Conservation, redesigned reader (development)1390.820.560.667
Agriculture, redesigned reader (fresh test)1070.750.460.667

Unit: studies, counting only those where both the reader and the human coders recorded a direction. Raw agreement reads respectably in every row. Alpha clears the bar in none.

Look at the gap between the two number columns. On the agriculture test, the reader and the human coders landed on the same direction three times in four, raw agreement 0.75 on 107 studies. That sounds like a working classifier. The alpha for the same rows is 0.46, below the bar, because almost all of that agreement is the easy agreement that the categories being lopsided hands you for free.

Raw agreement of three-in-four looked like a working classifier. Corrected for the lopsided categories, it was not one.

Where the disagreement actually was

The interesting part is not that the reader scored below a line. It is why. We expected it to be misreading sentences. It was not.

The agriculture set makes it clearest. On studies the human team had coded as the action raising yield, the reader agreed 60 times out of 64. On studies coded as the action lowering yield, it agreed 20 times out of 43, and read 19 of the rest as a rise. So the disagreement is almost entirely on the studies that went the unfavourable way.

We read a sample of those by hand. In each, the reader’s reading is faithful to the abstract: earthworm inoculation raised rice grain yield by 45%, rhizobia inoculation improved soybean yield over the control, and the reader recorded a rise. The human dataset’s “down” code for the same paper comes from the full text, where its team compared a different pair of treatments, a cropping system against another, deficit irrigation against full, and took the median across all of the pairs they extracted. The abstract headlines one contrast; the coders summarised a different one.

The reader was not misreading the sentence. It was reading a different comparison than the one the human coders scored, because the abstract and the full text do not headline the same one.

This is not a bug we can prompt our way out of, because both readings are correct answers to different questions. “What does the abstract say the action did” and “what did the study’s own authors conclude across every comparison they ran” are not the same measurement, and an abstract often cannot supply the second. The conservation set failed for the same reason: its human codes are the strategy against no strategy on one chosen variable, while the abstract reports every contrast it has room for.

What we therefore ship, and what we do not

Because we had committed to the bar in advance, the decision was already made when the number came in. We do not publish a paper-level verdict on whether an action worked. There is no “this helped” or “this hurt” field, in any atlas, derived this way.

What we do publish is the part that passed and the part that is verifiable sentence by sentence. We mark which studies evaluate an action. And for each outcome a study reports, we show the single sentence the reading rests on, quoted from the abstract, so a reader sees the claim and its source together and can judge the comparison for themselves. A quoted sentence that says a yield rose is a true statement about the abstract. A site-wide tally of whether actions “work” would not have been a true statement about anything, and we are not going to print one.

The method, which is the point

The figures here are specific to two evidence bases and one reader. What we would keep from them is the procedure, which is the same one we apply everywhere.

Write the bar down before you measure, so a disappointing result cannot quietly become the new definition of success. Check against evidence that someone else classified, not against your own model’s opinion of its own work. Report the chance-corrected number, not the flattering one, when the categories are lopsided. And when the task does not pass, say which half did, ship that half, and name the half you are holding back and why.

Finding the studies that test an action is a map we can build and stand behind. The verdict on whether each action worked is a harder reading than an abstract can carry, and we would rather tell you that than hand you a column of confident labels that agree with the record only as often as guessing would.

Evidence bases. Conservation: the Conservation Effectiveness evidence tables (Burivalova and colleagues, with Mongabay; conservationeffectiveness.org), in which people coded each finding as a positive, negligible or negative effect of the strategy across nine strategies. Agriculture: Evidence for Resilient Agriculture v1.0.1 (Rosenstock et al., Scientific Data, 2024; CC BY 4.0), crop-yield observations its team extracted from full texts. We report only aggregate agreement with those codings and reproduce none of the underlying records. The reader is a single language model reading title and abstract; the human codings are the independent check.

COMPASS ID makes research discoverable by what it covers, and by the gaps it has left dark. Browse the evidence atlases →