Offline benchmark · October 9, 2026
Less screening. Measured against known results.
Our tested embedding classifier found 95% of the relevant studies after reviewing an average of 36.2% of the records across 19 systematic reviews.
What “64% fewer abstracts to screen” means
On average, 63.8% of the records remained unreviewed when the simulation reached 95% recall. We round that to 64%. The comparison is with reviewing every record, at a target of finding 95% of the known relevant studies.
This is a benchmark average, not a guarantee for a new review or a measured saving in working hours. Finding 95% does not mean finding every relevant study.
Compared with our existing classifier
| Configuration | Records reviewed |
|---|---|
| Existing TF-IDF / Naive Bayes classifier | 50.1% |
| Tested embedding / logistic regression classifier | 36.2% |
That is 13.9 percentage points less screening, or a 27.7% relative reduction in the average screening fraction. The candidate improved on 16 of 19 reviews. The paired 95% bootstrap interval for the improvement is 6.8 to 21.9 percentage points.
How we tested
- SYNERGY 1.0: 19 review datasets, 55,185 records and 1,253 relevant studies according to the benchmark labels.
- 760 simulations: two embedding providers, four classifier and score-combination configurations, and five random startseeds. Each review counts equally; the simulations reuse the same collection.
- The featured result uses google/embeddinggemma-2 at 768 dimensions, balanced logistic regression and the existing dynamic score mix. Both training and ranking use only decisions already observed.
- Queries are the original review titles. Ten long abstracts were explicitly shortened identically for both providers. Structured research questions and criteria were not evaluated in this test.
- Recall was measured retrospectively using known relevance labels. These results do not validate an automatic stopping rule or the SAFE procedure.
This is an exploratory benchmark on a previously examined collection, not independent external validation. Features, classifier and class weighting changed together, so the improvement cannot be attributed to embeddings alone.
A tested candidate configuration
At benchmark completion, the candidate classifier had not been implemented or deployed in the application. This page reports its offline results, rather than measured performance of the current production workflow.