Research journal · Decisions API · Development study
From one screening decision to an ensemble with 99.3% recall.
How we taught a language model to exclude more cautiously: check sources first, make criteria explicit, test exclusions against evidence and combine different assessment roles.
554 of 558 retained
observed recall
still count as misses
across the original population
These are offline development results on twelve previously studied reviews. They offer no guarantee for new reviews and do not measure a deployed product feature.
01 / The research question
Missing a relevant article costs more than reading an extra abstract.
Systematic review screening has asymmetric costs. A wrongly retained article takes extra review time. A wrongly excluded article can disappear from the evidence base. We therefore started with a concrete goal: retain at least 95% of relevant articles, then assess how much screening work we can save.
The first question was simple: could we use the OpenAI Decisions API to predict a category for each title and abstract? We requested defined choices rather than free text: include and exclude for the overall assessment, and also unknown for criterion questions. That last category has an explicit meaning in the processing logic.
The API also returns choice probabilities and a confidence value. We validate that output, but do not treat it here as a calibrated probability of relevance. The comparison uses categorical choices and predefined logic, without tuning a favorable score threshold to the test results afterward.
Include here means: retain for further assessment. It does not mean an article definitively meets every requirement. Final inclusion may require full text, additional information and human judgment. This distinction shaped almost every major design decision.
We also explored embeddings. Our earlier analyses initially used the review title as the search question, without the full selection criteria. We then added criteria and context, with an API preflight check. That is a different experiment: embeddings help determine reading order. The new Decisions test measures a fixed retain-or-exclude decision. The 99.3% in this article comes from that decision test; this configuration has no embedding veto.
A metric that fits the goal. Recall is the proportion of reference positives retained: TP / (TP + FN). Accuracy alone can look overly favorable in a collection with many irrelevant articles. We therefore report recall, misses and remaining workload together.
02 / The first correction
Better predictions first required better evaluation conditions.
An early result of roughly 65% average recall was insufficient. Investigating the errors showed that predictions were not the only problem. Some benchmarks had issues with registration, criteria or the relationship between labels and the original review. Evaluating a model against the wrong task gives a misleading score, however well the API works.
We established a persistent dataset policy. Chou, Jeyaraman and Moran are excluded from new tests; Muthu is quarantined until its source and screening-stage scope are clarified. Shared benchmark code enforces this policy so later analyses cannot silently reuse unsuitable datasets.
Twelve eligible reviews remained, containing 36,243 records and 558 positive benchmark labels. On this cohort, the stored early binary predictions achieved 75.33% macro recall and 68.28% micro recall, missing 177 positive records. This recalculated existing responses on a different cohort. The increase from roughly 65% to 75% therefore cannot be attributed solely to a better prompt.
We distinguished three kinds of problems. A demonstrably incorrect registration requires source correction or dataset exclusion. An unresolved mismatch between criteria and labels requires investigation of the publication, version and selection stage. A complex but legitimate real-world case requires better screening logic. Removing a difficult review because the model performs poorly would optimize the wrong thing.
Eligible does not mean certified error-free. Individual questions about source and label scope remain within the twelve retained reviews. A final-inclusion label is not automatically equivalent to a human decision to advance from title/abstract screening to full-text review.
03 / An initial ensemble
More votes helped, but shared errors remained.
The literature suggested combining assessments. Sanghera et al. (2025) studied parallel ensembles where one inclusion vote is enough to retain an article. This fits our error costs. However, their results with different models do not establish that the same approach with one base model will perform equally well.
Our first pilot used three roles within the same model: an overall assessment, grouped criterion questions and a cautiously worded inclusion check. We retained an article if any role chose include. This OR rule achieved 87.9% macro recall and 88.4% micro recall. Of 558 positive records, 65 were still missed. Only four of twelve reviews reached 95%.
Majority voting was less suitable: two incorrect exclusions can overrule one correct inclusion. But the OR rule also fails when every role follows the same mistaken interpretation. We therefore examined all 65 shared misses rather than focusing only on improving the average.
| Primary observable issue | Count |
|---|---|
| Exclusion insufficiently supported by supplied evidence | 25 |
| Essential abstract information missing | 12 |
| Real-world complexity or interpretation boundary | 14 |
| Reference label conflicts with criteria; source cause unresolved | 12 |
| Possible publication or metadata mismatch | 2 |
One agent performed this assessment with knowledge of the outcomes. It is useful development analysis, not independent expert adjudication. The categories describe observable problems; they do not establish what the model internally reasoned.
The recurring patterns were concrete. Unreported details were treated as demonstrably absent. An ineligible additional treatment arm could overshadow a potentially eligible comparison. Compound criterion questions obscured which part was supposedly violated. The next version needed to address these problems in the general method.
04 / The decision logic
Unreported, explicitly violated and permitted are three different situations.
An abstract is a summary. It need not report every detail required by a review protocol. Criterion assessment must therefore distinguish three states: satisfied when supported by the text, violated when the text demonstrates an applicable violation, and unknown when information is insufficient.
Consider a generic requirement: at least twelve weeks of follow-up. An abstract that omits follow-up duration yields unknown and is provisionally retained. An abstract explicitly reporting only four weeks may violate the criterion. Missing information is not an independent reason for exclusion. Conversely, uncertainty about another criterion does not undo a clear, applicable violation.
Logical structure matters too. Some requirements must hold together; others form alternative inclusion routes. In multi-arm studies, requirements must be assessed within the same eligible arm or comparison. An intervention from one arm cannot be combined with a comparator from an unrelated arm. An ineligible additional arm also cannot invalidate every other route.
route = condition₁ AND condition₂ AND …
Provisionally retain while at least one route
is not blocked by a demonstrated applicable violation.
This is conservative logic for title/abstract screening. A route need not yet be fully proven eligible to justify further assessment. The final full-text decision remains a different task.
We encoded unknown in the processing logic as well as the instructions, making the rule inspectable. This does not solve everything: the model can still incorrectly call unknown violated. We therefore added an exclusion-evidence check.
Example: an abstract without follow-up duration stays in the review
with your review criteria.
Digital medication reminders for adults with hypertension: a randomized trial
Abstract
Objective. To assess digital reminders for medication adherence in adults with hypertension.
Methods. A randomized trial compared smartphone reminders with usual care.
Outcome. Medication adherence was assessed during follow-up.
Read the abstract → decide → screen the next article.
Your assessments inform the next ranking.
Fictional example, not a benchmark record. The abstract omits follow-up duration: unknown keeps the article available for human assessment and full-text checking. The illustration shows the reviewer's next action, not an executed model decision.
05 / Flexibility without benchmark-specific customization
Source criteria determine the questions; benchmark names do not.
The first pilot used manually written criterion groups for each review. That helps diagnose errors, but a general tool continually receives new research questions. Our next step was a source-grounded compiler: preparation that converts original criteria and additional context into a structured review plan.
This preparation uses the Responses API with structured output. The plan contains individual requirements, source passages, applicability scope and relationships between requirements and alternative routes. The compiler sees no articles, positive labels or earlier model choices. Later assessments retain access to the original criteria, so the derived plan is not the review's only representation.
A minimal illustrative plan shows the structure. A source passage must occur verbatim in the original criteria; every route refers to existing criterion IDs. Actual plans contain all requirements and their relationships, including exceptions.
{
"criteria": [{
"id": "follow_up",
"requirement": "Follow-up of at least twelve weeks",
"role": "follow_up",
"unit_scope": "The same eligible comparison",
"evaluation_scope": "review_eligibility",
"source_quotes": [{
"field": "criteria",
"quote": "Follow-up of at least twelve weeks"
}]
}],
"eligibility_routes": [{
"id": "route_1", "criterion_ids": ["follow_up"]
}],
"unresolved_issues": [],
"interpretation_notes": []
}
We checked preparation before permitting article calls. In an early version, several plans were incomplete and one preparation hit the output limit. A more compact representation and source-grounded diagnosis resolved this. We then found a subtler issue: explicit permission in one requirement could be overridden by another, overly broad exclusion requirement.
The correction was general: exceptions must propagate to every applicable requirement. We added no rule containing a benchmark name or an answer for a known miss. Changes were based on source criteria before the new article assessments. All twelve final plans passed model checking and a separate source comparison by an agent.
A small technical test suite with nine synthetic medical, software and environmental cases then checked that the general processing worked beyond one domain. All nine passed the final smoke version without fallback. This demonstrated a functioning technical chain, not new benchmark recall or general reliability.
A shared compiler remains a shared risk. If a plan mistranslates an exception, several later questions may inherit the same mistake. We therefore check the plan against the source and retain assessments that see only the original input.
06 / The new design
A general vote, checked criteria and a blind inclusion check.
The new primary variant combines three assessment paths: A OR BC OR D. All use the same base model, gpt-6-luna. We are studying role and instruction differences, rather than diversity between base models.
Digital medication reminders for adults with hypertension: a randomized trial
Complete screening decision using the original criteria. Sees no derived plan or other responses.
B assesses requirements. A proposed exclusion triggers quotations and the informed check C.
A fresh complete assessment focusing on permitted routes and missing information. Sees no earlier votes.
A: an independent general assessment
A receives the original review information and article. It makes a complete include/exclude choice with conservative instructions for missing information. This path prevents every decision from depending entirely on the source compiler.
B and C: an exclusion must survive more than one categorical choice
B answers the automatically derived criterion questions. Code combines the answers according to the defined routes. Only a proposed exclusion triggers a request for verbatim evidence through Responses. Local processing checks whether those passages actually occur in the supplied source fields.
C receives the proposed violations, quotations and original texts. It assesses whether a quotation supports the applicable violation, accounting for scope and exceptions. An unproven or uncertain violation does not block a route. C can reverse an exclusion; it cannot change an article retained by B into an exclusion. This focuses the additional step on recovering wrongly excluded articles.
This approach was inspired by the criterion questions and critical checking in Hu et al. (2025). Our chain and evaluation are our own design choices. That publication also uses routed human assessment; its reported recall does not directly measure an autonomous model.
D: the same sources, a different task, no earlier responses
D looks for a valid provisional inclusion route and checks whether missing information is being misused as exclusion evidence. It makes a complete screening decision. A favorable assessment of the population alone would not be a valid inclusion vote.
D sees only original criteria, context and article information. C sees the earlier assessment. This distinction matters: a critic can anchor on a mistaken initial explanation. Blindness does not guarantee independent errors; the same model and sources may still produce the same miss.
# Illustrative aggregation, not an API request.
votes = [general_A, checked_BC, blind_D]
if any(v == "include" for v in votes):
outcome = "include"
elif all(v == "exclude" for v in votes):
outcome = "exclude"
else:
outcome = "fallback" # further human assessment
Technical failure is a separate state. It must not count as a successful model inclusion. All components were valid in the final test, so fallback did not increase measured recall.
07 / Evaluation design
The same articles, fixed variants and a repeated-call control.
We tested all 558 original positive benchmark records plus 200 randomly selected negative records per review: 2,958 articles in total. This matches the earlier ensemble pilot's selection. Within the new test, all variants receive exactly the same source information. Reference labels and evaluation metadata remain local and are not sent as model input.
The primary variant A OR BC OR D, other comparisons, prompts, selection and aggregation rules were fixed before article calls. Every role and control ran on every article. We can therefore compare paired article outcomes and identify which misses were actually recovered.
We added an important control: four separately executed general A requests with identical API bodies. Three form an ensemble of ordinary repetitions. One cached response does not stand in for several votes. Each repetition slot has its own stored request and response; resumption reuses only the response from that same slot.
The ensemble of three ordinary A assessments uses the same number of Decisions calls as A OR B OR D. A second repetition control matches A OR BC's article-request count. The latter does not exactly match costs or tokens, because BC also uses Responses and review preparation. This avoids presenting more calls as a better ensemble without further analysis.
A development test, not an unseen final test. These twelve reviews had already informed source investigation, error analysis and design choices. A frozen new run prevents changes during the comparison, but does not make the cohort retrospectively independent.
08 / Results
Complementary assessments recover articles that repetition misses.
The prespecified primary variant retained 554 of 558 positive records: 99.28% micro recall and 99.61% macro recall. All twelve reviews reached at least 95% in this run. The lowest per-review recall was 98.08%. This is a clear improvement within this development material.
The new variants, on the same 2,958 articles
| Variant | Macro recall | Micro recall | Missed | Reviews ≥95% | Work remaining¹ |
|---|---|---|---|---|---|
| A · General assessment | 88.9% | 87.6% | 69 | 4/12 | 21.4% |
| B · Criterion questions | 79.6% | 79.2% | 116 | 3/12 | 16.8% |
| BC · Criteria + evidence check | 94.7% | 94.3% | 32 | 6/12 | 43.1% |
| D · Blind inclusion check | 97.6% | 96.1% | 22 | 10/12 | 66.9% |
| A OR BC | 96.9% | 97.0% | 17 | 9/12 | 45.6% |
| BC OR D | 99.5% | 99.1% | 5 | 12/12 | 73.6% |
| A OR B OR D | 98.8% | 98.0% | 11 | 11/12 | 67.6% |
| A OR BC OR D · Primary | 99.6% | 99.3% | 4 | 12/12 | 73.8% |
| 3× A · Repeated assessment | 88.9% | 87.6% | 69 | 4/12 | 21.4% |
¹ Estimated retained share of the full original population; negative records weighted for sampling. Macro recall gives every review equal weight. Micro recall pools the 558 positive records.
Criterion questions alone are insufficient
B achieved 79.6% macro recall, below the new general assessment A's 88.9%. Decomposition produces visible, auditable answers, but can also create more opportunities to incorrectly label something violated. A longer prompt is not inherently an improvement.
Evidence checking made a large difference: moving from B to BC recovered 84 positive records without losing any previously retained positives. Macro recall rose to 94.7%. BC improved substantially, but still did not meet the target in every review.
A blind vote complements the informed check
D alone achieved 97.6% macro recall, but remained below 95% in two reviews. Adding D to BC recovered 27 positives. Adding D to A OR BC recovered another 13, leaving four shared misses.
The simpler BC OR D combination retained 553 of 558 positives, compared with 554 for the primary variant. Adding A therefore recovers one extra record at the cost of 2,958 additional general article calls. This makes BC OR D a useful candidate for further validation. We do not retrospectively replace the prespecified primary result with the most favorable tradeoff.
Ordinary repetition changed no categorical choices
Repeated A choices were identical for all 2,958 articles. Three ordinary votes therefore achieved the same recall as one A vote: 87.6% micro recall. At the same Decisions-call count, A OR B OR D recovered 58 positives compared with the repetition control. In this run, different tasks produced the improvement; repeating the same task added no coverage.
This is an empirical finding for this execution. It does not establish that repetition never helps with other models or conditions, nor that the roles' errors are statistically independent.
What workload does higher recall require?
The primary variant leaves an estimated 73.8% of the original population for assessment: a 26.2% estimated reduction. Giving each review equal weight leaves an average of 83.3%, or a 16.7% average reduction per review. These are different weightings of the same experiment.
Because all positives were deliberately included in the sample, the raw retained fraction of the 2,958 test articles is not a sound population estimate. We weight each review's negative sample back to its original negative-record count. These are counts of records requiring assessment, not measured working hours saved. Recall intervals also do not describe uncertainty in this workload estimate.
View the primary outcome per review and its 95% Wilson interval
| Review | Retained / positive | Missed | Recall | 95% interval |
|---|---|---|---|---|
| Appenzeller-Herzog 2019 | 26 / 26 | 0 | 100.0% | 87.1–100.0% |
| Bos 2018 | 10 / 10 | 0 | 100.0% | 72.2–100.0% |
| Donners 2021 | 15 / 15 | 0 | 100.0% | 79.6–100.0% |
| Hall 2012 | 102 / 104 | 2 | 98.1% | 93.3–99.5% |
| Meijboom 2021 | 37 / 37 | 0 | 100.0% | 90.6–100.0% |
| Menon 2022 | 73 / 74 | 1 | 98.6% | 92.7–99.8% |
| Oud 2018 | 20 / 20 | 0 | 100.0% | 83.9–100.0% |
| Sep 2021 | 40 / 40 | 0 | 100.0% | 91.2–100.0% |
| van Dis 2020 | 71 / 72 | 1 | 98.6% | 92.5–99.8% |
| van de Schoot 2018 | 38 / 38 | 0 | 100.0% | 90.8–100.0% |
| van der Valk 2021 | 89 / 89 | 0 | 100.0% | 95.9–100.0% |
| van der Waal 2022 | 33 / 33 | 0 | 100.0% | 89.6–100.0% |
09 / Remaining errors
Four misses remain four misses, even when their source questions are complex.
The final shared exclusions came from Hall, Menon and Van Dis. We reexamined them without changing labels or retrospectively removing those reviews.
- Hall: predicting fault counts. An article describes reliability growth modelling, while the supplied criteria exclude reliability modelling. The positive label may reflect another selection stage or interpretation. This source question remains unresolved.
- Hall: defects after inspection. The model confirms an interpretation about document type and predicted defects. The word document alone does not prove the study falls outside software code. A genuine quotation can therefore coexist with an overconfident exclusion.
- Menon: recreational contact with surface water. The article discusses illness caused by pathogens, but a passage about the disease mechanism does not establish that every permitted exposure route falls outside the review. Scope and the label relationship need further checking.
- Van Dis: pharmacotherapy for social phobia. The described comparison includes medication and placebo, with no reported CBT arm. This raises a publication, version or selection-stage question. Identity and full text must be checked before calling it a registration error.
The lesson extends beyond these four articles. Checking a verbatim quotation protects against fabricated evidence passages. It does not automatically prevent an incorrect inference from a real passage. Source identity, applicability scope and interpretation remain separate checks.
We therefore do not claim the model actually made no mistakes. The four records remain in the denominator and count as missed. This analysis shows where additional human source checking is valuable and where semantic evidence checking needs further validation.
10 / Engineering and reproducibility
An interrupted API run must still be a fair experiment.
The initial bulk execution with 24 workers encountered HTTP 429 responses and a two-million-token-per-minute limit. This was a transport problem, not a substantive model outcome. We preserved failed requests and dependent cases, and implemented resumption separately from the frozen screening logic.
The recovery wrapper used 16 workers and a shared 1.4-million-token-per-minute budget, waiting according to rate-limit information and applying bounded transient retries. Prompts, model input, article selection and aggregation did not change. Across 351 recovered cases, 1,877 already successful component responses were preserved exactly. Successful votes were not requested again to seek more favorable results.
All 2,958 article comparisons were ultimately complete, with no component fallbacks. The full execution, including controls and preparation, contained 20,994 stored request events: 19,016 through Decisions and 1,978 through Responses. The recovery run produced no new 429 responses. The original 358 exhausted transport requests remain visible in the audit archive.
The primary variant required 12,083 article requests plus 37 review-preparation requests. The standard-rate estimate was approximately $3.58 for article calls and $0.03 for preparation. The complete experiment with all controls totaled around $4.55 based on stored successful usage. These are not invoice amounts: unknown usage from failed requests, retries, cache writes and possible surcharges is not fully included.
The sum of individual request durations is also not a parallel run's elapsed time. Some durations include budget waits and retries. We use these records for auditable execution accounting, without inferring unsupported product latency.
An offline replay reconstructed every choice and ensemble outcome from the stored original API responses. It compared request bodies, source texts, labels, quotations, repetition slots and aggregation. All 20,994 events and the full paired comparison passed the audit. In addition, 50 targeted tests passed for screening logic, source preparation and transport handling.
What we retain. The local research archive contains inputs, original populations of eligible reviews, source plans, exact executed code, raw responses, paired outcomes, failed requests and recovery. SHA-256 hashes protect every archive file. Downloads on this page contain only aggregates and methodology.
This archive makes the local evidence auditable. It does not freeze a future API model: model aliases and provider implementations can change. Repeating the same request later differs from reproducing the stored execution offline.
11 / From development to validation
A strong development score is the start of the next test.
Observed recall now exceeds our target. Yet all twelve reviews reaching 95% is not a proven lower bound for future use. Hall, for example, achieved 102 of 104: 98.08% recall, with a 95% Wilson interval of approximately 93.26–99.47%. Bos achieved 10 of 10, or 100% observed recall, but a lower bound of roughly 72.2%. Small positive cohorts limit certainty.
The 99.18–100% macro bootstrap interval describes the average across these studied reviews. It does not remove selection effects or the development history. A high cohort average also does not guarantee sufficient recall for every new review.
The publications emphasize this distinction. Akinseloyin, Jiang and Palade (2026) investigate criterion questions and score combinations for prioritization. Their retrospective WSS@95 results do not establish a safe fixed exclusion threshold. The OLIVER preprint by Godhwani and Benrimoh also shows that an actor–critic configuration can vary substantially between reviews. Calibration, ranking quality and inclusion recall require separate assessment.
The next phase therefore requires unseen reviews with source criteria audited in advance and explicitly recorded label scope. Splitting articles from the same previously studied reviews is insufficient to test generalization to new protocols. Model roles and aggregation rules will be frozen again before that evaluation.
We also want human checks of exclusions and critical interpretations, a larger negative sample for better workload estimates, and a comparison between the primary configuration and simpler BC OR D variant. Any combination with embeddings will be studied as a separate prioritization design. Low semantic similarity must not silently become an exclusion reason.
The practical lessons are concrete: check the source before trusting the score; leave missing information unresolved; preserve alternatives and exceptions; require testable evidence for exclusion; and measure whether another assessment role actually recovers different misses. This produces a system that is easier to investigate and improve than one ever-longer prompt.
Research notes
Publications and result files
These publications informed the design. They are not independent validations of our implementation; differences in models, selection, screening stage and human involvement prevent directly ranking their reported percentages.
- Sanghera et al. (2025), High-performance automated abstract screening with large language model ensembles. JAMIA. Parallel ensembles and inclusion-focused prompting; motivation to test retention with one inclusion vote.
- Hu et al. (2025), Enhancing Evidence Synthesis Efficiency. Cochrane Evidence Synthesis and Methods. Criterion questions, critical agents and human routing; inspiration for an auditable assessment chain.
- Akinseloyin, Jiang & Palade (2026), Large language model-based multiagent collaboration for abstract screening toward automated systematic reviews. Biology Methods and Protocols. Model diversity and score combination for ranking.
- Godhwani & Benrimoh (2025), LLM-Assisted Abstract Screening with OLIVER. Preprint. Separate selection stages and varying actor–critic performance across reviews.
- Parmar et al. (2026), Collaborative large language models are all you need for screening in systematic reviews. Preprint. Retention under disagreement fits the high cost of misses. Unresolved differences in positive counts between tables limit direct numerical comparison.
The downloadable summary gives the protocol hash and SHA-256 of the complete result file, identifying the frozen execution behind this page. The full local audit archive also contains the data needed to reconstruct the original calculations.
Version: 13 October 2026 · Protocol generic-ensemble-v3 · Source preparation source-preparation-v5 · All original reference labels retained · No new model calls for this article.