Code & Resources

This page provides code, prompt templates, and documentation for the AI-assisted abstract screening workflow used in this living evidence prototype.

The workflow was developed to support a living systematic review of youth homelessness intervention evidence. It uses large language models to help screen titles and abstracts, compare AI screening decisions with human reviewer decisions, and evaluate whether AI-assisted screening can support future review updates.

The purpose of sharing these materials is not to suggest that AI can replace human reviewers. Instead, the code and resources show how review teams can test AI-assisted screening in a transparent, human-in-the-loop way.

View GitHub Repository

The GitHub repository includes:

  • Python scripts for running AI-assisted abstract screening;

  • prompt templates for different screening strategies;

  • data templates showing how input files should be structured;

  • evaluation code for comparing AI and human screening decisions;

  • documentation for adapting the workflow to other evidence reviews.

The repository is designed to help researchers, policy analysts, evidence synthesis teams, students, and community partners understand how the workflow was implemented and how it could be adapted for other topics.

What Is Included

1. Prepare the abstract file

The review team begins with a spreadsheet or CSV file containing study records. Each row represents one study and includes information such as the title, abstract, publication year, journal name, authors, and keywords.

2. Apply a screening prompt

Python sends each study record to an AI model using a structured screening prompt. The prompt tells the model how to interpret relevance for the review.

3. Save AI screening outputs

The model returns either an include/exclude decision or a relevance score, depending on the prompt format. These outputs are saved in a new file.

4. Compare AI decisions with human decisions

The AI screening file is merged with human screening decisions using a shared study ID. This allows the review team to compare where AI and human reviewers agreed or disagreed.

5. Calculate performance metrics

The workflow calculates recall, precision, specificity, accuracy, and F1 score. These metrics help assess how closely AI-assisted screening aligned with human screening decisions.

What the Code Does

The GitHub repository includes example data templates so other teams can see how their files should be structured.

The abstract input file includes columns such as:

  • study ID;

  • title;

  • abstract;

  • publication year;

  • journal name;

  • publisher;

  • author;

  • author affiliation;

  • keywords.

The human screening file includes:

  • study ID;

  • human include/exclude decision.

The repository does not include the full original abstract dataset because database records and abstracts may be subject to licensing, copyright, or project-specific restrictions.

Data Templates

How Other Review Teams Can Use These Materials

Other review teams can adapt this workflow by:

  1. preparing their own abstract file;

  2. preparing their own human screening file;

  3. editing the prompt files to match their review question;

  4. selecting a model and prompt type;

  5. running AI-assisted screening;

  6. comparing AI decisions with human decisions;

  7. selecting the prompt and model setup that best fits their review goal.

The results from one topic area may not transfer directly to another. A prompt that works well for youth homelessness intervention evidence may not perform the same way for child welfare, education, behavioral health, housing policy, or another evidence area.

Each review team should test and validate the workflow using its own data, eligibility criteria, and human screening decisions.

AI screening should be treated as screening support, not final judgment.

Different prompt designs may produce different results. More inclusive prompts may catch more potentially relevant studies but may also send more irrelevant studies to human reviewers. Stricter prompts may reduce human screening burden but increase the risk of missing relevant evidence.

Model names, availability, cost, and behavior may change over time. Review teams should document the model, prompt, date, settings, and evaluation results used in their own workflow.

Human validation remains necessary.

Important Notes