Now that the body of literature has been compiled and data has been extracted, it is time to assess quality of the studies reviewed for relevance to the research question.
When reviewing study quality, assessing for bias is necessary to ensure reliability. Biases to assess for include the following, to name a few (Boland, Cherry, & Dickson, 2017):
How generalizable are the results of the study to your target population? Is the sample representative?
How were participants assigned to treatment groups? Is this a predictable assignment, or was it random?
Were participants and/or researchers aware of their treatment group assignment? Were studies single-blind? Double-blind? If participants are aware of their assignment, they may feel an obligation to perform a certain way in alignment with the study's goal.
Were data analyzers and those reviewing study outcomes aware of individual identities/treatment group assignments? This may bias study results.
What portion of participants stopped treatment prior to the end of the study? Did participants dropout of the study, withdraw, or not meet inclusion criteria? The more participants that drop out, the less generalizable the results.
Were all outcomes reported on? Did authors fail to report outcomes that did not hold significance?
In assessing for quality, there are numerous tools available for researcher use. See the following resources for more information:
The following are standard tool used for quality assessment. Note, AMSTAR and PRISMA are standards in assessing systematic reviews, while MOOSE is the standard for assessing meta-analyses.
This checklist notes steps needing to be taken to conduct a systematic review, specifically in the healthcare fields (Shea et. al, 2017). This tool can be used to inform necessary pieces of the review.
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is a resource available for those conducting systematic reviews (Page et. al, 2021). PRISMA offers tools including checklists for authors to ensure all necessary components are complete and flowcharts for illustrating the author's search process. The checklist (most recent version is 2020) provides details regarding every required aspect of a protocol/methodology for a systematic review. This should be treated as a bare minimum for authors to include.
This checklist is for authors and reviewers of meta-analyses (Stroup et. al, 2013).
