How the Prototype Works

This prototype uses artificial intelligence to support one early stage of a living systematic review: title and abstract screening. In a systematic review, this is the step where reviewers decide whether each study appears relevant enough to move forward for full-text review.

The goal of this workflow is not to let AI make final inclusion decisions. Instead, AI is used as a screening assistant. Human screening decisions remain the comparison point for evaluating whether the AI is helping identify potentially relevant studies.

The prototype was developed in two phases. Phase 1 used previously human-screened studies from the 2008–2017 review period to develop and test prompts. Phase 2 applied the best-performing prompt from Phase 1 to newer studies from 2018–2025 to examine whether the workflow could support an updated, more living review process.

Phase 1: Developing the Workflow with Baseline Data

Phase 1 used the prior human-led systematic review as the baseline dataset. This review searched for studies published from 2008 to 2017 and focused on interventions for youth experiencing homelessness or housing instability. The human-screened data from this review provided a comparison point for testing whether AI could approximate human title and abstract screening decisions.

The Phase 1 search included major electronic databases, web-based publication sources, and professional outreach. Electronic databases included EBSCO-hosted databases, CINAHL, Cochrane Library, ERIC, and PubMed/MEDLINE, along with additional searches in Google and Google Scholar. Youth- and family-service websites were also searched, including sources such as the California Evidence-Based Clearinghouse, CrimeSolutions.gov, Blueprints for Healthy Youth Development, the OJJDP Model Programs Guide, and Urban Institute publications. Professional outreach was also used to identify studies or databases that might meet the review criteria.

In this phase, the workflow followed six main steps:

  1. Abstract data from the baseline review were prepared for AI screening.

  2. Prompts were developed to guide the language model’s screening decisions.

  3. The AI generated screening decisions for each abstract.

  4. Human screening data were imported into Python.

  5. AI and human screening decisions were merged into one dataset.

  6. Performance was evaluated using metrics such as recall, precision, AUC, and F1.

Phase 1 was used to compare different prompt strategies and select the prompt that best supported the goals of abstract screening.

Phase 2: Applying the Best Prompt to Newer Evidence

Phase 2 tested whether the best-performing prompt from Phase 1 could support screening for a newer evidence update. This phase focused on studies from 2018 to 2025, extending the review beyond the original published search window.

The Phase 2 abstract data were gathered from systematic review search sources commonly used for health, social science, education, and youth-services evidence. These included sources such as EBSCO-hosted databases, PubMed, Scopus, CINAHL, Social Services Abstracts, Cochrane Library, and related academic databases.

The Phase 2 workflow followed a more streamlined process:

  1. Newer abstracts from 2018–2025 were prepared for AI screening.

  2. The best-performing prompt from Phase 1 was applied.

  3. AI screening decisions were generated.

  4. AI decisions were compared with human validation data.

  5. Performance metrics were calculated.

  6. Results were used to assess whether the workflow could support an updated living review.

This phase matters because it moves the project beyond prompt development. It tests whether an AI-assisted screening workflow can help extend an existing systematic review as new studies become available.

Why Human-in-the-Loop Screening Matters

This prototype uses a human-in-the-loop design rather than relying on AI decisions alone. Human reviewers provide the validation standard against which AI-assisted screening is evaluated.

For Phase 2, abstracts were screened by two independent human review teams. Team 1 included two reviewers who screened the abstracts and made inclusion/exclusion decisions. Team 2 included another two reviewers who independently validated the screening decisions. Agreement between reviewers was then assessed to calculate human reliability and to establish a stronger benchmark for evaluating AI performance.

This design matters because systematic review screening often involves ambiguity. Abstracts may not clearly report participant age, homelessness status, intervention type, study design, or implementation context. In youth homelessness research, relevant studies may also involve mixed-age samples, housing-related services, broader public health interventions, school-based programs, or systems-level supports that intersect with homelessness even when the abstract does not use exact homelessness language.

AI was therefore evaluated as a screening triage tool, not as a final decision-maker. The AI-generated decisions were compared against human screening and validation data to assess how well the model identified potentially relevant studies. Human reviewers remained responsible for validation, interpretation, and final inclusion decisions.

The purpose of this human-in-the-loop approach is to ask a practical question: Can AI help reduce screening burden while still preserving human oversight, reliability, and methodological rigor?

How This Supports a Living Review

A traditional systematic review often becomes a snapshot of evidence from a specific search period. Once the review is completed, new studies continue to appear, but updating the review can require repeating many labor-intensive steps.

This prototype supports a more living model by making abstract screening easier to repeat. When new studies are collected, the workflow can be reused: abstracts can be screened with AI assistance, compared with human validation, evaluated, and refined. In this sense, the “living” part is not only the website. It is also the workflow behind the website — a repeatable process for helping evidence stay current over time.

The next sections show how this workflow was built and evaluated. Prompt Development explains how the screening prompts were designed and refined. Results summarizes how the AI performed across Phase 1 and Phase 2. Code & Resources provides access to the GitHub workflow, reproducibility materials, and technical documentation.