What We Evaluated

This prototype evaluated whether AI-assisted abstract screening can support a living systematic review of youth homelessness interventions. The goal was not to replace human reviewers. Instead, the goal was to test whether AI can help identify potentially relevant studies, reduce missed studies, and make evidence updates more repeatable.

The evaluation had two parts:

Phase 1 used the baseline review data from 2008–2017 to develop and compare screening prompts.

Phase 2 applied the best-performing prompt logic to newer studies from 2018–2025 to test whether the workflow could support an updated living review.

Across both phases, AI screening decisions were compared with human screening or validation decisions. Performance was assessed using recall, specificity, accuracy, precision, F1 score, running time, and cost.

How to Read the Metrics

Because this website is designed for researchers, policy analysts, students, and community partners, the key metrics are explained in plain language.

Recall asks: Of all studies human reviewers considered relevant, how many did AI successfully catch?
For abstract screening, recall is especially important because missing a relevant study too early can remove it from the review before full-text screening.

Specificity asks: Of all studies human reviewers considered not relevant, how many did AI correctly exclude?
Higher specificity means the AI is better at filtering out irrelevant studies.

Accuracy asks: Overall, how often did AI agree with the human screening decision?

Precision asks: Of all studies AI flagged as relevant, how many were actually relevant based on human review?

F1 score combines recall and precision into one summary metric.

For this prototype, high recall was prioritized because the purpose of AI at the abstract screening stage is to support triage, not to make final inclusion decisions.

Phase 1: Prompt Development Results

In Phase 1, we tested different ways of asking AI (GPT-4o) to screen study abstracts.

The first strategy asked AI to act like a strict reviewer. This approach had high precision but low recall. In plain language, when the AI said a study should be included, it was often right. However, it only caught about 43% of the studies that human reviewers considered relevant. That means it missed more than half of the studies humans would have kept for further review. For abstract screening, this is a problem because relevant studies that are missed early may never move forward to full-text review.

The second strategy reframed AI as a screening assistant rather than a final decision-maker. Instead of asking AI to include only studies that clearly met every criterion, this prompt asked AI to flag studies that seemed potentially relevant enough for human review.

This improved recall to about 74%, meaning the AI caught nearly three out of every four studies that human reviewers considered relevant.

The third strategy expanded the inclusion logic even further. It helped the model recognize studies that might not use obvious youth homelessness language but were still relevant, such as mixed-age samples, housing-related interventions, systems-level services, broader public health interventions, and homelessness-adjacent contexts. This prompt achieved the highest recall, about 89%, meaning the AI caught almost nine out of every ten studies that human reviewers considered relevant.

Overall, Phase 1 showed that the way we frame the prompt matters. When AI was asked to make strict final inclusion decisions, it missed too many relevant studies. When AI was asked to support human-guided triage, it became much better at catching studies that should move forward for human review.

Phase 2: Validation on Newer Evidence

In Phase 2, we tested whether the best Phase 1 prompt could also work on a newer set of studies from 2018–2025. This was important because the goal of a living systematic review is not only to perform well on older data, but also to help update the evidence base as new studies are published.

The Phase 2 results showed that the prompt maintained very high recall. The AI caught about 91% of the studies that human reviewers considered relevant. In plain language, this means that for every 100 relevant studies identified by human reviewers, the AI successfully flagged about 91 of them for further review.

This is a strong result for early abstract screening because the main concern at this stage is avoiding missed studies. If a relevant study is excluded too early, it may never reach full-text review.

A recall of about 91% suggests that the AI-assisted workflow was useful for helping the review team identify most of the potentially relevant evidence in the newer 2018–2025 search.

At the same time, the Phase 2 prompt had lower precision (20%). The AI flagged many studies for review that human reviewers did not ultimately consider relevant. In practical terms, this means the AI was being intentionally broad. It reduced the risk of missing relevant studies, but it also created more studies for humans to check.

For a living review workflow, this tradeoff can be acceptable. The AI is not making the final decision. Instead, it helps screen broadly so human reviewers can focus their attention on studies that may deserve closer review. The results below are from GPT-4o.

Model and Prompt-Type Comparison

After identifying the best overall screening logic, the workflow also tested different combinations of models and prompt formats. This matters because AI performance does not depend only on which model is used. It also depends on how the task is presented to the model.

For this project, the documented implementation compared GPT 5, GPT 5.2, GPT 5 mini, and GPT 5 nano across several prompt formats. These tests helped answer practical questions such as:

Which model catches the most relevant studies?
Which prompt format reduces missed studies?
Which option is faster or less expensive to run?
Which setup gives the best balance between recall, specificity, accuracy, time, and cost?

Why Model and Prompt Format Matter

What the Prompt Formats Mean

These prompt formats matter because they change how cautious or inclusive the AI becomes.

A stricter setup may exclude more irrelevant studies, which can reduce human workload. But it may also miss relevant studies.

A broader setup may catch more relevant studies, which is useful for early abstract screening. But it may also send more irrelevant studies to human reviewers.

For a living systematic review, this tradeoff is important. The best setup is not always the one with the highest accuracy. The best setup depends on the goal of the review stage.

Zero-shot prompting

A zero-shot prompt means the AI is given instructions but no examples.

In plain language, the prompt says:

“Here are the screening rules. Now screen this abstract.”

This matters because zero-shot prompts are simpler, shorter, and easier to reuse. They may also be faster and cheaper because the prompt contains less text. However, without examples, the model may interpret the screening task differently than the research team intended.

Few-shot prompting

A few-shot prompt gives the AI a few examples before asking it to screen new abstracts.

In plain language, the prompt says:

“Here are the screening rules. Here are a few examples of studies that should be included or excluded. Now screen this abstract.”

This matters because examples can teach the model what the review team means by “relevant.” Few-shot prompting can be useful when the topic is complex or when relevance is not obvious from the abstract. However, it can also make prompts longer, slower, and more expensive to run.

Chain-of-Thought prompting

A chain-of-thought prompt asks the AI to reason through the decision before giving an answer.

In plain language, the prompt asks:

“Think through the screening criteria, then decide whether this abstract should be included or excluded.”

This matters because reasoning prompts may help the model consider the study more carefully. They can sometimes improve specificity or make decisions more transparent. However, they can also take longer, cost more, and do not always improve performance.

For this website, you can describe this as reasoning-based prompting if “chain-of-thought” feels too technical.

A decision format is the way the AI is asked to give its screening answer.

In abstract screening, the AI reads a study title and abstract, then must return a judgment about whether the study may be relevant for the review. But that judgment can be requested in different ways.

For example, the AI can be asked to give a simple yes/no decision, such as Include or Exclude. Or it can be asked to rate how relevant the study seems on a scale, such as clearly not relevant to clearly relevant.

This matters because the format of the answer can change how the AI behaves. A simple yes/no format may make the AI more decisive, but it can also be too rigid when abstracts are unclear. A rating-scale format gives the AI more room to express uncertainty, which may help catch more potentially relevant studies.

In this project, we compared two decision formats: binary and Likert-style.

  1. Binary Decision Format

    A binary format asks the AI to choose between two options:

    Include or Exclude

    This matters because binary decisions are easy to analyze and compare with human screening decisions. However, they can be too rigid when abstracts are ambiguous. Some studies may be “possibly relevant,” but the model still has to force them into yes or no.

  2. Likert-Style Decision Format

    A Likert-style format asks the AI to rate relevance on a scale instead of making only a yes/no decision.

    For example:

    1 = clearly not relevant
    2 = probably not relevant
    3 = uncertain
    4 = probably relevant
    5 = clearly relevant

    This matters because abstract screening often involves uncertainty. A Likert-style format lets the AI express degrees of relevance. This may help catch more potentially relevant studies, which can improve recall. However, it may also flag more studies for human review, lowering specificity or accuracy.

What Is a Decision Format?

The sections above explain the building blocks of the comparison: different models, different prompting approaches, and different decision formats. The next step is to look at how these choices performed when applied to the screening task.

In other words, the tables below ask a practical question:

When we combine a specific model with a specific prompt format, which setup works best for AI-assisted abstract screening?

The Phase 1 comparison uses the baseline review data to examine how each model-and-prompt combination performed in terms of recall, specificity, accuracy, running time, and cost.

Moving from Concepts to Model Comparison

Phase 1 Model Comparison

In Phase 1, the highest recall came from GPT 5 mini with zero-shot chain-of-thought Likert and GPT 5 nano with zero-shot chain-of-thought Likert, both reaching 94.6% recall. However, these high-recall configurations had low specificity, meaning they also allowed many irrelevant studies through.

The best balance between recall, specificity, accuracy, time, and cost appears to depend on the review goal:

  • If the priority is maximum recall, Likert-style prompts performed well.

  • If the priority is balanced screening, binary prompts often preserved better specificity and accuracy.

  • If the priority is cost control, GPT 5 nano was cheapest, but often had lower specificity.

  • If the priority is speed and high recall, GPT 5.2 zero-shot direct decision Likert was especially promising in Phase 1 because it achieved 92.5% recall in 12.20 minutes at relatively low cost.

In Phase 2, the highest recall was achieved by GPT 5.2 zero-shot direct decision Likert, with 96.6% recall. This means it was the strongest configuration for catching relevant studies in the updated 2018–2025 evidence set.

However, it also had lower specificity and accuracy. This means it was more inclusive and flagged many studies for review. For early abstract screening, this may still be useful if the goal is to avoid missing relevant studies.

The strongest balanced configuration in Phase 2 was GPT 5.2 zero-shot chain-of-thought binary, which reached 82.8% recall, 81.1% specificity, and 81.3% accuracy. This configuration may be more useful when a review team wants both sensitivity and better exclusion of irrelevant studies.

The cheapest configurations came from GPT 5 nano, but the highest-recall nano configurations had much lower specificity. This suggests that lower-cost models may be useful for exploratory screening, but should be carefully validated before use in a formal review workflow.

Phase 2 Model Comparison

  • The results from this project are specific to the youth homelessness intervention review, the abstracts included in this evidence base, the eligibility criteria, and the prompts tested.

    If another review team applies this workflow to a different topic — for example, child welfare, mental health, education, housing policy, public health, or workforce interventions — the results may not be the same. Different evidence areas use different language, study designs, populations, and intervention terms. These differences can change how well a model identifies relevant studies.

    The main lesson is not that one model or one prompt format will always perform best. The main lesson is that AI-assisted screening should be tested and validated within each review context before being used in practice.

  • The same model can behave differently depending on how the task is framed. A prompt that asks the model to make a strict include/exclude decision may produce different results than a prompt that asks the model to rate relevance, reason through criteria, or identify studies that may be relevant enough for human review.

    For policy and evidence-review teams, this means prompt writing is not just a technical step. It is part of the review method. The prompt shapes how broad or narrow the AI screening process becomes.

  • In this project, high recall was especially important because the goal was to reduce the risk of missing studies at the abstract screening stage. In plain language, recall tells us how many of the human-relevant studies the AI successfully caught.

    For early screening, a high-recall approach can be useful because human reviewers can later remove irrelevant studies. However, if AI misses a relevant study too early, that study may never move forward to full-text review.

    This does not mean every review should always maximize recall at any cost. It means review teams should decide what matters most at each stage of the review.

  • Some prompt formats, such as Likert-style relevance ratings, produced very high recall in this project. This suggests that allowing the model to express degrees of relevance may help it catch more potentially relevant studies.

    However, this often came with lower specificity. In plain language, the AI also let more irrelevant studies through. That creates more work for human reviewers.

    For policy teams, this is a practical tradeoff:

    A broader AI screen may reduce missed evidence, but it may not reduce workload as much.
    A stricter AI screen may reduce workload, but it may increase the risk of missing relevant evidence.

  • Reasoning-based prompts, such as chain-of-thought prompts, did not always produce better results. In some cases, they improved specificity or balance. In other cases, they increased running time without improving recall.

    This matters for applied review teams because a more complex AI setup is not automatically more useful. A simpler prompt may be easier to run, easier to explain, cheaper, and just as effective depending on the review goal.

  • A living review must be repeatable. If the workflow takes too long or costs too much, it may be difficult to rerun when new studies are published.

    For policy organizations, the best AI setup is not only the one with the highest performance score. It also needs to be practical, affordable, transparent, and sustainable.

    This is why model comparison should include not only recall, specificity, and accuracy, but also running time and cost.

  • There is no single best model or prompt format for every systematic review.

    A review team may choose different configurations depending on the purpose:

    • To avoid missing relevant studies: prioritize recall.

    • To reduce human screening burden: prioritize specificity or accuracy.

    • To support frequent updates: prioritize cost and running time.

    • To support transparency: choose a prompt format that reviewers can understand and audit.

    The best configuration should be selected based on the review question, the evidence base, the risk of missing studies, available staff time, and how often the review needs to be updated.

Overall Findings

The results suggest that AI-assisted screening can support a living systematic review when it is used as a human-in-the-loop triage process.

For this youth homelessness intervention review, high-recall configurations were especially useful because they reduced the risk of missing relevant studies at the abstract stage. However, these findings should not be treated as universal rules. If another team applies this workflow to a different systematic review, they should expect to test and validate the model and prompt choices using their own data.

The most important contribution of this prototype is therefore not one specific model result. It is a practical framework for review teams:

Define the review goal.
Test different prompt formats.
Compare AI decisions with human decisions.
Evaluate recall, specificity, accuracy, cost, and time.
Choose the configuration that fits the review purpose.
Keep humans responsible for validation and final decisions.

In all cases, AI should support evidence screening, not replace human judgment. Human reviewers remain essential for interpreting ambiguous studies, validating AI outputs, and making final inclusion decisions.

Bottom Line