Jan 19, 2026
Methodological Approach for Screening and Evaluation of Literature using the Rayyan Tool
- Saylen S Koscelny1,
- Eric S McLamore1,
- Diana Vanegas1
- 1University of Arkansas, Department of Biological and Agricultural Engineering
- Diana Vanegas: * corresponding author email: [email protected]
Protocol Citation: Saylen S Koscelny, Eric S McLamore, Diana Vanegas 2026. Methodological Approach for Screening and Evaluation of Literature using the Rayyan Tool. protocols.io https://dx.doi.org/10.17504/protocols.io.dm6gp1325gzp/v1
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: January 17, 2026
Last Modified: January 19, 2026
Protocol Integer ID: 238813
Keywords: open source rayyan tool, efficient systematic literature review, evaluation of literature, rayyan tool, systematic review, preferred reporting items for systematic review, rayyan, published literature, methodological approach for screening, rayyan tool this protocol, literature, screening, method for gamification, gamification, analysis, tablet, evaluation, process for use, preferred reporting item, methodological approach
Funders Acknowledgements:
PIRE project
Grant ID: 2230696
ESM
Grant ID: 2023-38821-39796
Abstract
This protocol describes a process for use of Rayyan to screen published literature. The technique is embedded into a framework based on PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses). The open source Rayyan tool may be used on a mobile device, tablet or as a web application to screen literature based on title and abstract. In this protocol, we describe a method for gamification of Rayyan screening for rapid and efficient systematic literature reviews. The complete process as described in Koscelny et al, (2026) it is shown in Figure 1, with the key steps utilizing the Rayyan Tool illuminated in red.
Figure 1. Process flow for systematic reviews using PRISMA approach. The steps for use of the Rayyan tool are highlighted in red.
Guidelines
Data management plan
The data management plan herein includes: i) types of data, ii) standards for metadata, iii) archiving metadata, iv) sharing policies, and v) policies and provisions for derivatives.
i) Types of data
Data include bibliographic information from systematic literature review using Web of Science (WoS).
ii) Standards for metadata
Bibliographic data were exported directly from Web of Science (WoS) in .txt file format. No additional metadata standardization was performed. The exported file contains structured bibliographic information, including authors, title, abstract, keywords, publication year, journal name, and citations. This file format preserves the original WoS field organization, which ensures consistency and reproducibility of the review. Metadata are stored as .txt files and were archived as described below.
iv) Sharing policies
All the resources and data are open source. The data do not include PDF files, but links are provided to journal website for access.
v) Policies and provisions for re-use, re-distribution, and production of derivatives
No intellectual property or software are produced in this research.
Materials
The protocol depends on use of the Rayyan tool, which is available at: https:/www.rayyan.ai (Ouzzani et al. 2016)
Troubleshooting
Before start
Details for the nine steps shown in Figure 1 are provided below. Steps 4-8 are specific to this protocol (noted in red in Figure 1). Brief information on the other steps is provided below for completeness. Details on the entire process are described in Koscelny et al.
The process is based on a PRISMA-guided systematic review (Page, McKenzie, et al. 2021; Page, Moher, et al. 2021).
Record search using PRISMA framework
The first step in a PRISMA-guided systematic review is to establish the central question. In this example, the question was:
“Which electrochemical biosensor approaches have been demonstrated to reliably detect E. coli in real or field-relevant matrices applicable to agricultural water, and what analytical performance metrics do they report?”
Next, eligibility
criteria are defined. In the example here, studies were included if they:
- reported an electrochemical biosensor,
- targeted Escherichia coli (any strain or serotype),
- evaluated performance in real samples or realistic matrices, and
- were peer-reviewed articles published in English.
The next step in the PRISMA framework is to set the restrictions on publication date. For this example, the publication window was set from 2015 to 2025, with a last acceptance date of May 1st, 2025.
Next, a plan is established for which types of records are to be included. In this example, non-research items (e.g., reviews, book chapters, conference-only abstracts) were excluded. Only peer reviewed publications were included.
Preliminary Database ScreeningUntitled section
The search engine used for this example was Web of Science (WoS)-All Database, although any search engine that can export results as a .txt file is relevant for this protocol. Search engine filters were applied to refine the results as follows:
- Records were limited to peer-reviewed journal articles in English.
- Records were filtered to only include the top 5 relevant WoS research categories: electrochemistry, nanotechnology, biotechnology, microbiology, environmental sciences, agricultural engineering.
- WoS categories deemed irrelevant were excluded (e.g., physics, business economics, energy fuels, nursing, music) were excluded.
Export bibliographic recordsUntitled section
The selected records from Web of Science- All Database (Clarivate 2025) were exported directly
from the results page using the “Export” function, with the export type set to “Records” in sequentialb batches of 1,000 records (e.g., 1–1,000, 1,001–2,000) until all results had been exported (Figure 2). Files were saved in EndNote Desktop (txt) format.
Figure 2. Web of Science (All Databases) export interface showing record-range selection and
batch export settings.
Important:
- All selected records were exported from Web of Science in a txt file format. Exported fields included: title, abstract, authors, keywords, source, DOI, and metadata.
Record export fields were customized in the “Record Content” dropdown menu to include the following relevant bibliographic information: title, authors, source, citation count, accession number, and abstract. Figure 3 illustrates the interface for custom export selection of records.
Figure 3. Customized export fields in Web of Science.
All exported files were placed in a single project folder, compressed into a single ZIP file, and uploaded to Rayyan using the import function.
Duplicate Removal with Rayyan Desktop Interface
The exported database (ZIP file) was imported into Rayyan (Ouzzani et al. 2016) using the Desktop Interface, where the built-in duplicate detection algorithm was used to identify grouped duplicate records. All duplicate suggestions were manually verified before removal to prevent accidental loss of unique papers (Ouzzani et al. 2016).
Figure 4 shows a screenshot of the Rayyan tool. In this example, 54 articles from a corpus of 12,436 were flagged as duplicates and reviewed manually.
Figure 4. Rayyan duplicate detection view
Critical Step:
Duplicate removal was essential to maintain accurate counts for the PRISMA diagram and to
avoid screening the same record multiple times.
Title and Abstract screening with Rayyan Desktop Interface
Title screening with Rayyan desktop interface used keyword filters to group articles (e.g., “E. coli,” “Escherichia,” “electrochemical,” “biosensor”). Each title was manually reviewed for relevance.
Figure 5 shows a screenshot of the desktop interface (Ouzzani et al. 2016; Rayyan Systems Inc 2025).
Figure 5. Rayyan desktop interface: Grouping, Title and Abstract View
To reduce fatigue, maintain focus, and improve screening speed, two complementary screening approaches were employed depending on the device used: the “flip” method on a mobile phone (see Step 6) and the hotkey-based exclusion system on a desktop computer (see Step 7).
Mobile Gamification for Screening process with Rayyan
Screening decisions were documented using Rayyan Premium (Student Quarterly #SUB-23433) (Ouzzani et al. 2016; Rayyan 2025). A flip-to-include/exclude method was developed for gamification of record inclusion based on Rayyan (See Figure 6).
Participants were informed of the inclusion criteria prior to analysis. Using a mobile phone, participants read the title and then performed a rapid determination as follows:
- Swipe right → Include
- Swipe left → Exclude
- Tap Maybe radio button → Revisit record using Desktop method
This method was designed to increase attention, reduce fatigue, and helped maintain consistent decisions. Using this gamification approach, participants have reported reviewing more than 100 record titles per day.
Uncertain cases were marked as “Maybe” to be revisited using the Desktop
method (shown in next section).
Figure 6. Rayyan Mobile Interface: a) Grouping, Title and Abstract Flip View. The screening of Rayyan’s mobile interface was used strategically to transform screening into a more intuitive and “game-like” task, where swiping left includes a record (panel b) and swiping right excludes a record (panel c).
Additional Screening with Rayyan Desktop Interface (optional)
Recordsnot easily identified with the gamification technique were screened using the Rayyan Desktop tools.
This protocol used a hotkey-based exclusion method with written explanation(s).
Critical step:
The Rayyan premium tool is required for use of the written explanation feature during screening.
Screening decisions were documented using Rayyan Premium’s (Student Quarterly #SUB-23433)“Exclude
with Reasons” tool. This allowed for transparent justification and reproducible exclusion.
Keyboard shortcuts were used to increase speed and maintain consistency during the screening process:
| A | B | C | |
| Hotkey | Meaning | Exclusion Reason Description | |
| n | Not relevant to topic | Used when the study did not involve biosensors or was unrelated to electrochemical detection. | |
| r | Review article | Used for narrative reviews, systematic reviews, or meta-analyses rather than primary research papers. | |
| t | Different target analyte | Used when the analyte was not Escherichia coli (e.g., other bacteria, viruses, or chemical contaminants). | |
| e | No electrochemical method | Used when the study did not use an electrochemical method as a transduction mechanism |
The hotkey codes are applied directly during screening with the Rayyan interface (see Figure 7 for screenshot example). Rayyan automatically assigns each paper to the corresponding exclusion category based on the established hotkey configuration.
Figure 7. Desktop Screening “Exclude with Reasons” Panel: Title and Abstract View.
Rayyan’s AI-powered Prediction Classifier
The AI-powered Predictions Classifier within Rayyan was enabled to assist with prioritizing and cross-validation (See Figure 8a). The tool was used to rank the records based on how likely they are to be included in the review, based on the patterns identified during review (See Figure 8b).
In this protocol, automated predictions were not used to make exclusion decisions, although this is an option with the software (Ouzzani et al. 2016; Rayyan Systems Inc 2025). For this protocol, all records were manually assessed by participants to ensure accuracy.
Figure 8. Rayyan relevance prediction feature showing rating definitions and relevance score output for screened records. (two thumbs up): Most likely to include: Articles with the highest probability of
inclusion. (one thumb up): Likely to include: Articles with a high probability of inclusion. (question mark): No recommendation: The AI doesn't have enough information to make a confident prediction. This is common early in the screening process. (one thumb down): Likely to exclude: Articles with a high probability of exclusion. (two thumbs down): Most likely to exclude: Articles with the highest probability of exclusion.
Detailed analysis of included records
Once the Rayyan screening was complete, each record included in the study was processed in detail by the research team.
To avoid reviewer fatigue (Polanin et al. 2019), a careful plan should be established for detailed analysis of all records included. An example is summarized below.
Each paper is read thoroughly by at least one member of the research team. The analysis goal is two papers per day. Key information is manually extracted from each record, focusing on elements that align with the objectives of the literature review, including:
- Methodology used in the study
- Main findings and conclusions
- Limitations
- Future directions
The inclusion/exclusion data were anaylzed anda PRISMA flowchart prepared for visualizing the systematic review.
Protocol references
Aria, Massimo, and Corrado Cuccurullo. 2017. “Bibliometrix : An R-Tool for Comprehensive Science Mapping Analysis.” Journal of Informetrics 11(4):959–75. doi:10.1016/j.joi.2017.08.007.
Clarivate. 2025. “Web of Science (All Databases).” https://www.webofscience.com/wos/alldb/smart-search.
Ouzzani, Mourad, Hossam Hammady, Zbys Fedorowicz, and Ahmed Elmagarmid. 2016. “Rayyan—a Web and Mobile App for Systematic Reviews.” Systematic Reviews 5(1). doi:10.1186/s13643-016-0384-4.
Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and David Moher. 2021. “The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews.” BMJ n71. doi:10.1136/bmj.n71.
Page, Matthew J., David Moher, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and Joanne E. McKenzie. 2021. “PRISMA 2020 Explanation and Elaboration: Updated Guidance and Exemplars for Reporting Systematic Reviews.” BMJ n160. doi:10.1136/bmj.n160.
Polanin, Joshua R., Terri D. Pigott, Dorothy L. Espelage, and Jennifer K. Grotpeter. 2019. “Best Practice Guidelines for Abstract Screening Large‐evidence Systematic Reviews and Meta‐analyses.” Research Synthesis Methods 10(3):330–42. doi:10.1002/jrsm.1354.
Rayyan. 2025. “Rayyan (Version [Mobile Application]).”
Rayyan Systems Inc. 2025. “Rayyan: AI-Powered Systematic Review Management Platform.” https://www.rayyan.ai/.
Acknowledgements
Funding acknowledgement
This work was supported by the National Science Foundation Office of International Science and Engineering program (PIRE-2230696). ESM was supported by the Agriculture and Food Research Initiative Competitive Grant no. 2023-38821-39796. ESM and DV were supported by NIFA Multistate project NC1194.