Screening methodology

When to Stop Screening in a Systematic Review

Stopping rules can identify diminishing yield in an AI-prioritized queue. They cannot prove that every relevant record has been found, so the rule, validation, and acceptable risk belong in the protocol.

By Lumina Editorial Team Published Updated Methodology guide
Systematic review screening queue showing progress and human inclusion and exclusion controls
A stopping decision should be evaluated against the documented screening state and remain under reviewer control.

Start with the review objective

Prioritization changes the order in which records are seen. Early stopping changes which records are assessed at all. That second decision needs stronger justification because a relevant record may still appear late in the queue.

Screening every record remains the clearest option when the protocol requires exhaustive assessment, when missing a study could materially change a conclusion, or when the ranking has not been validated on comparable data.

Three signals Lumina can monitor

Observed streak

Consecutive irrelevant records

A long sequence of exclusions may indicate declining yield in a ranked queue.

Limit: the result depends on ranking quality and the chosen streak length. A streak does not estimate how many relevant records remain unseen.

Curve shape

Knee detection

The method looks for a bend in the cumulative discovery curve where new inclusions become less frequent.

Limit: curves can be noisy, especially when relevant records are rare. The detected knee is a mathematical feature, not a recall estimate.

Recent yield

Slope or yield threshold

This signal monitors the inclusion rate over a recent window and alerts when observed yield drops.

Limit: window size and threshold can materially change the alert. A low recent yield can still be followed by a late relevant record.

A defensible decision process

  1. 01

    Pre-specify the rule

    Name the algorithm, parameters, minimum training data, and action the team will take when the signal appears.

  2. 02

    Validate on relevant evidence

    Use retrospective data, a known benchmark, or a protocol-defined validation sample that resembles the current review.

  3. 03

    Check ranking behavior

    Do not stop if inclusions remain frequent, ranking quality appears unstable, or reviewer criteria changed during screening.

  4. 04

    Report the full decision

    Report screened and unscreened counts, the stopping rule, parameters, validation, deviations, and any sensitivity analysis.

What to write in the methods section

Records were prioritized using [tool and model]. The review protocol specified [stopping rule and parameters]. After the stopping signal was reached, the team performed [validation procedure]. Screening ended after [screened count] of [total count] records; [unscreened count] records remained unassessed. Final eligibility decisions were made by human reviewers.

Adapt this text to what your team actually did. Do not report an estimated recall unless you calculated it with a defensible reference standard.

Sources and further reading

Callaghan & Müller-Hansen Statistical stopping criteria for automated screening. van de Schoot et al. Active-learning evaluation with ASReview. PRISMA 2020 Reporting guidance for systematic reviews.

See prioritized screening before configuring a rule

Explore how records are ranked and how human decisions remain visible in the workflow.

Try the demo Read about active learning