Sep 11, 2020
Version 1
Generic Protocol for Environmental Health Systematic Reviews Based on COSTER Recommendations V.1
- 1Lancaster University
- Systematic Reviews
Protocol Citation: Paul Whaley 2020. Generic Protocol for Environmental Health Systematic Reviews Based on COSTER Recommendations. protocols.io https://dx.doi.org/10.17504/protocols.io.biktkcwn
Manuscript citation:
Whaley, Paul, Elisa Aiassa, Claire Beausoleil, Anna Beronius, Gary Bilotta, Alan Boobis, Rob de Vries, et al. 2020. “Recommendations for the Conduct of Systematic Reviews in Toxicology and Environmental Health Research (COSTER).” Environment International 143 (July): 105926.
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: In development
This is the first attempt at developing a generic protocol based on the COSTER recommendations. Feedback is very welcome!
Created: July 15, 2020
Last Modified: September 11, 2020
Protocol Integer ID: 39283
Keywords: systematic review, environmental health, toxicology, protocol,
Abstract
A protocol template to help researchers follow the COSTER recommendations for conduct of systematic reviews. This instance covers the planning steps of a systematic review and will help with writing up the systematic review protocol.
The intent is to convert COSTER from a checklist of things which need to be done into a sequence of actions which can be followed by a research team.
When completing the protocol and either registering it or submitting it to a journal, please cite this instance of the protocol template and the parent manuscript, DOI 10.1016/j.envint.2020.105926.
Image Attribution
Image by Paul Whaley.
Guidelines
Protocols.io has not yet been optimised as a means for reporting what was done in response to complex instructions such as those found in this protocol. Feedback on use of the protocol, and how to develop it to facilitate reporting of planned methods, would be very much appreciated.
Securing capacity, competencies, and tools
Securing capacity, competencies, and tools
Assess the team's combined competence in conduct of a systematic review. Recommendation 1.1.1.
| Competency | Team member(s) (initials) | |
| Information science (for e.g. search strategies) | ||
| Evidence appraisal methods (i.e. risk of bias assessment) | ||
| Statistical methods | ||
| Domain or subject expertise | ||
| Systematic review methods |
Team member competencies
Identify information management practices and tools for each stage of the review. Recommendation 1.1.2.
| Information management component | Tools or packages | |
| Reference manager | ||
| Knowledge management tool | ||
| Systematic review software | ||
| Statistics software and packages | ||
| Aritifical Intelligence support tools (e.g. for screening) |
Information management tools and packages
List the potential conflicts of interest of the authors. Recommendations 1.1.3 and 1.1.4.
This should include both financial and non-financial interests which readers should be aware of in order to understand the motivations of the authors of the review.
By listing the interests as potential, you are confirming that they are not apparent conflicts of interest, i.e. they cannot reasonably be expected to compromise the integrity of the systematic review. People with apparent conflicts of interest should be excluded from decision-making roles in the review.
Interests should be declared using the ICMJE Conflict of Interest Disclosure Forms, as attached. The summary statements generated by the forms for each author can be copy-pasted into the table.
ICMJE COI Disclosure Form.pdf
ICMJE COI Disclosure Form.pdf | Author | ICMJE COI Summary | |
Summary statements of authors declared conflicts of interest.
Setting the research question ("problem formulation")
Setting the research question ("problem formulation")
Demonstrate the need for a new review. Recommendation 1.2.1
Describe the scientific value of the question(s), i.e. why it is important that it be investigated.
Describe the importance to stakeholders of the question(s) being asked.
Summarise relevant existing primary research and evidence syntheses to justify conducting a new systematic review.
Articulate the scientific rationale for each question via development of a theoretical framework. Recommendation 1.2.2. For example, this would describe how the exposure is related to the outcomes of interest if the systematic review is an investigation of an exposure-outcome relationship. The theoretical framework should include discussion of the biological plausibility of the relationship being investigated.
For each research question to be answered by the review, prospectively define a statement of the research objective in terms of Population, Exposure or Intervention, Comparator, Outcome, Study Design, and Target Condition, selected as appropriate. Recommendation 1.2.3.
- Authors may wish to refer to Morgan et al. 2018 for guidance on how to formulate research questions as PECO statements. Conceiving of an ideal study may also help characterise the PECO elements which define what type of study will be informative for your review findings.
CITATION
Define the target Population of interest. These are the objects of investigation, i.e. the entities to which exposures or interventions happen.
| Species | ||
| Sex | ||
| Age | ||
| Health status | ||
| Additional characteristics |
Characteristics of the population of interest. Add rows to cover other population characteristics relevant to the SR question.
Define the target Exposure or Intervention of interest. This concerns the administered or observed change in conditions of the objects of investigation. It should include timing, duration and dose.
| Exposure or intervention | What is the exposure or intervention? | |
| Timing | When does the exposure or intervention happen? | |
| Duration | For how long does the exposure or intervention last? | |
| Dose | What is the dose regimen (amount, frequency)? |
Timing, duration and dose of the exposure / intervention. Add rows to cover other exposure / intervention characteristics relevant to the SR question. Add a new table for each exposure or intervention of interest.
Define the target Comparator of interest. This concerns the characteristics of the exposure or intervention being used as the comparator to which the target exposure or intervention is being compared.
| Comparator | What is the comparator exposure or intervention? | |
| Timing | When does the comparator happen? | |
| Duration | For how long is the comparator administered? | |
| Dose | What is the dose of the comparator (amount, frequency)? |
Timing, duration and dose of the comparator. Add rows to cover other comparator characteristics relevant to the SR question.
Define the target Outcome(s) of interest. This concerns the change being measured in the exposure or intervention group. These should be the primary outcomes of interest to the systematic review which form the hypothesis or hypotheses being tested. Secondary outcomes can also be listed.
| Primary outcome 1 | ||
| Primary outcome 2 | ||
Primary outcomes of interest. Add new rows for each outcome of interest.
| Secondary outcome 1 | ||
| Secondary outcome 2 | ||
Define the Target Condition. This is the object of a test method for diagnosis or detection. It is only necessary for a systematic review of a diagnostic or detection test method.
| Target condition characteristic 1 | ||
| Target condition characteristic 2 | ||
Defining the eligibility criteria and designing the process for screening evidence for inclusion
Defining the eligibility criteria and designing the process for screening evidence for inclusion
Define and justify unambiguous and appropriate eligibility criteria for each component of the objective statement. Recommendation 1.3.1, 1.3.3, 1.3.4, 1.3.5
| PECO element | Description of eligibility criteria | |
| Eligible populations | Include e.g. age, sex, health status, socioeconomic status, occupation etc. | |
| Eligible exposures | Include timing, methods for measurement exposure | |
| Eligible comparators | The populations and exposures against which the exposed populations are being compared | |
| Eligible primary outcomes | Specify the outcome, whether the outcome is apical (whole organism) or intermediate (is a marker of an apical outcome); the acceptable outcome measures (diagnostic criteria, scales, etc.) and timing of outcome measurement | |
| Eligible secondary outcomes | Specify the outcome, whether the outcome is apical (whole organism) or intermediate (is a marker of an apical outcome); the acceptable outcome measures (diagnostic criteria, scales, etc.) and timing of outcome measurement | |
| Eligible study designs | Define eligibile study designs by design features rather than design labels. |
Describe the eligibility criteria for each PECO element. Add additional PECO elements as appropriate.
| PECO element | Description of exclusion criteria | Reasons for exclusion | |
| Excluded populations | |||
| Excluded exposures | |||
| Excluded comparators | |||
| Excluded outcomes |
Describe the criteria for exclusion of studies, according to each PECO element. Add additional PECO elements as appropriate.
Define the points at which screening for eligibility will take place. Recommendation 1.3.2. Will there be screening at title and abstract, full text, or both?
| Points at which screening will take place | |
| Describe whether there will be screening at title and abstract, full text, or both |
Points at which screening will take place
Include all relevant, publicly-available evidence, except for research for which there is insufficient methodological information to allow appraisal of internal validity. Recommendation 1.3.6. Exclude evidence which is not publicly available. Recommendation 1.3.9
| Policy on eligibility of grey literature and unpublished evidence | |
| Describe how grey literature will be handled in the systematic review. If some or all grey literature is to be excluded, explain why and anticipate its implications as a limitation of review methods. |
Policy on grey literature for the systematic review.
Note
COSTER recommends that grey literature (i.e. studies that have not been published in peer-reviewed journals) should be included in systematic reviews. This is because the relevance of evidence is determined by the SR objectives, not by the publication status of that evidence, the language the evidence is in, nor its compatibility with the analyses planned by the reviewers.
Only publicly available information about a study should be eligible for inclusion. If the planned SR will bring into the public domain evidence which was previously inaccessible, this makes the evidence eligible for inclusion.
Studies for which there is insufficient information for risk of bias to be evaluated should be excluded from a SR, to prevent the inclusion in a SR of evidence that is potentially misleading but cannot be identified as such by the reviewers.
Include evidence which is relevant to review objectives irrespective of whether its results are in a usable form. Recommendation 1.3.7
| Policy on eligibility of studies with unusable data | |
| Describe how studies which report their results in a manner incompatible with planned analyses will be handled in the systematic review. |
Policy on usability of study data
Note
COSTER recommends that documents be included in a SR regardless of whether their data fit the analysis plan of the reviewers or they are in a language in which the reviewers are fluent. This is to ensure that study documents which may contain information of potential relevance to the SR’s research objectives are not excluded from the data extraction step of the SR; however, they may be excluded from specific synthesis steps such as meta-analysis.
Include relevant evidence irrespective of language. Recommendation 1.3.8.
| Policy on eligibility of studies based on language | |
| State the language/s in which the systematic review will be written, and how studies not written in that language will be handled. |
Policy on language
| Languages to be included in the systematic review | |
List of included languages in the systematic review
Do not exclude multiple reports of the same research (e.g. multiple publications, conference abstracts etc.); instead collate the methodological information from each of the reports as part of the data extraction process for each unit of evidence. Recommendation 3.4
| Multiple publications policy | |
| Describe how multiple publications derived from the same study will be aggregated. |
Policy on handling of multiple publications from same study
Screening of each piece of evidence for inclusion to be conducted by at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes. Recommendation 3.1
| Team members who will conduct screening | Method for resolving disputes | |
Planned approach to duplicate screening and dispute resolution
Design the PRISMA flow chart for presentation of the results of the screening process. Recommendation 3.2
Pilot test the screening process. Recommendation 1.4.7
Note
A generic protocol for piloting the screening stage of a systematic review is available here:
Defining the strategy for searching for evidence relevant to the review objectives
Defining the strategy for searching for evidence relevant to the review objectives
Design sufficiently sensitive search criteria, so that studies which meet the eligibility criteria of the review are not inadvertently excluded. Document the search methods in sufficient detail to render them transparent and reproducible. Recommendations 1.4.1, 2.6
Search all the key scientific databases for the topic, including national, regional and subject-specific databases. Recommendation 2.1
| List of databases | |
| Database 1 | |
| Database 2 | |
| Database 3 |
List of databases searched in the systematic review
Structure search strategies for each database, electronic and other source, using appropriate controlled vocabulary, free-text terms and logical operators in a manner which prioritises sensitivity. Document the search methods and results in sufficient detail to render them transparent and reproducible. Recommendations 2.3, 2.6
| Database | Search strategy | |
Search strategy for each database in the systematic review
CITATION
2d
Define reproducible strategies for identifying and searching sources of grey literature (databases, websites etc.). Document the search methods and results in sufficient detail to render them transparent and reproducible. Recommendations 2.2 and 2.6
| Grey literature source | Search strategy | Date of search | No. of results | |
Search strategy for each source of grey literature in the review
2d
Search within the reference lists of included studies and other reviews relevant to the topic (“hand-searching”) and consider searching in the reference lists of documents which have cited included studies. Search by contacting relevant individuals and organisations. Recommendations 2.4 and 2.5
| Supplementary search strategies | Indicate if will be used | |
| Hand search references of included studies | ||
| Hand search references of relevant reviews | ||
| Hand search references of studies cited by included studies | ||
| By contacting individuals and organisations | ||
| Other |
Supplementary search strategies
Plan for re-running all searches and screen the results for potentially eligible studies within 12 months prior to publication of the review (screening at least at the level of title plus abstract). Recommendation 2.7
| Timing | When will the searches be updated prior to publication of the review? | |
| Sources | Which sources will be searched again? | |
| Level of screening | What level of screening will be conducted? | |
| Updating findings | How will review findings be updated in context of new studies? |
Policy for updating searches
Methods for synthesising and evaluating the evidence
Methods for synthesising and evaluating the evidence
Design the "characteristics of included studies" table. Recommendation 1.4.2
Design and pilot the data extraction forms. Recommendation 1.4.7
Define the risk of bias assessment methods to be used for evaluating the internal validity of the included research. If observational studies are included, this should cover identification of plausible confounders. Recommendation 1.4.3
Note
Review teams may find the FEAT (Focus-Extent-Application-Transparency) mnemonic to be useful in defining their risk of bias assessment methods.
- Focus: The focus of the tool should be exclusively the internal validity of a study. If other quality constructs are of interest, each should be assessed in a separate process.
- Extent: All the important threats to internal validity should be covered by the tool. If observational studies are being appraised, the threats should include all important confounders.
- Application: The appraisal process should produce consistent, accurate descriptions of the extent to which a study is vulnerable to each identified threat to internal validity. The judgements should be in a form which can be logically incorporated into the evidence synthesis.
- Transparency: The reason for each judgement should be documented, quoting as justification relevant text from the study documentation.
Refer to Section 5 of the COSTER recommendations for detail on how the risk of bias assessment process should be conducted.
Define the tool selection and modification process (how will a suitable tool be identified, and what process will be followed to identify and validate any necessary modifications?) Recommendation 1.4.3
| Tool selected | Studies to which it is applied | Modifications made | Method for validating modifications | |
Selection of tools to be used in systematic review
Risk of bias assessment is to be conducted by at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes. Recommendation 5.3
| Team members conducting risk of bias assessment | Method for identifying and settling disputes | |
Approach to conducting risk of bias assessment
Define the training and piloting process for the risk of bias assessment (how will the review team be trained in use of the tool, and what are the conditions under which the piloting process will be determined satisfactory?) Recommendation 1.4.7
Design the methods for synthesising the included studies, to cover: qualitative and quantitative methods (with full consideration given to synthesis methods to be used when meta-analysis is not possible); assessment of heterogeneity; choice of effect measure (e.g. RR, OR etc.); methods for meta-analysis and other quantitative synthesis; pre-defined, appropriate effect modifiers for sub-group analyses. Recommendations 1.4.4, 6.1
| Synthesis Component | Planned Methods | |
| Qualitative or narrative methods | ||
| Quantitative methods | ||
| Conditions for combining studies in overall and subgroup analyses | ||
| Choice of effect measure | ||
| Assessment of heterogeneity (6.3) and consequences of developing summary results (6.4) | ||
| Effect modifiers for subgroup analysis | ||
| Transformation of scales into common measures (6.2) | ||
| Assessment of publication bias (6.5) | ||
| Impact of the risk of bias assessment on the synthesis (6.6) | ||
| Sensitivity analyses (6.7) | ||
| Other methods |
Methods for synthesising the included evidence
Note
Refer to section 6 of COSTER for detailed recommendations for how evidence should be synthesised in systematic reviews. Popay et al. (2006) attached provides very useful guidance on how to approach the non-quantitive components of the synthesis.
Popay et al. 2006 - Guidance on the Conduct of Narrative Synthesis in Systematic Reviews.pdf Define the methods for determining how, given strengths and limitations of the overall body of evidence, confidence in the results of the synthesis of the evidence for each outcome is to be captured and expressed. (For reviews which include multiple streams of evidence, this may need to be defined for each stream.) Recommendation 1.4.5
Note
The components of assessment of confidence or certainty in the evidence are described in section 7 of COSTER.
Pilot the process for the assessment of confidence in the results of the synthesis of the evidence. How will the review team be trained in use of the tool, and what are the conditions under which the piloting process will be determined satisfactory? Recommendation 1.4.7
For reviews which include multiple streams of evidence (e.g. animal and human studies), define the methods for integrating the individual streams into an overall result. Recommendation 1.4.6
This should include a description of the relative relevance of populations (e.g. species, age, comorbidities etc.), exposures (e.g. timing, dose), and outcomes (direct or surrogate, acute or chronic model of disease, etc.), as appropriate, per which inferences about predicted effects in target populations can be made from observed effects in study populations.
Registering and publishing the protocol
Registering and publishing the protocol
Create a permanent public record of intent to conduct the review (e.g. by registering the protocol in an appropriate registry) prior to conducting the literature search. Recommendation 1.5.1
As appropriate for review planning and question formulation, secure peer-review and public feedback on a draft version of the protocol, incorporating comments into the final version of the protocol. Recommendation 1.5.2
Publish the final version of the protocol in a public archive, prior to screening studies for inclusion in the review. Recommendation 1.5.3
Note
Publication of the protocol in a journal is equivalent to publication in a public archive.
Citations
Step 16.2
Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate.
https://doi.org/10.1002/jrsm.1127Step 6
Morgan RL, Whaley P, Thayer KA, Schünemann HJ. Identifying the PECO: A framework for formulating good questions to explore the association of environmental and other exposures with health outcomes.
https://doi.org/10.1016/j.envint.2018.07.015