Section 10 Quality Assessment

10.1 Why assess study quality?

Low methodological quality can affect internal validity and introduce bias into the results of primary studies. Internal validity refers to the extent to which study results reflect the true cause-effect of an intervention. Different types of bias can influence internal validity (e.g. selection, performance, detection, and attrition biases).

It is not impact. It is not novelty.

Bias in primary studies can lead to an over- or under-estimation of the true intervention effect in both primary studies and systematic reviews. It is important to consider the implications of study quality and validity for interpreting the results from your systematic review and it is often a good idea to incorporate a quality assessment section into your final report.

Study quality characteristics which have been shown to impact the results of preclinical studies include whether animals were randomised to control or treatment groups, and if researchers were blinded to intervention allocation or exposure when assessing outcomes. Read more about allocation and blinding on the NC3Rs Experimental Design Assistant website.

10.2 Reporting quality

You can use a reporting quality checklist on primary studies in your systematic review.

The CAMARADES Checklist
Nature Reporting Checklist. The operationalised checklist is available here.

10.3 Risk of bias / methodological quality

Use a Risk of Bias (RoB) tool to help you evaluate the methodological quality of a primary animal experiment. Tools that have been developed to assess bias and quality in preclinical studies include the SYRCLE RoB tool.

10.4 RoB assessment to inform analysis

The extent to which a study is at risk of bias can hugely impact the findings. Findings from your risk of bias assessment should inform the conclusions of your systematic review.

Conduct sensitivity analysis (quantitatively using meta-analysis or qualitatively)
Exclude studies at high risk of bias from the evidence synthesis (this should be done with extreme caution and prespecified in your protocol to avoid bias)
Reach an overall conclusion for each outcome as to whether the synthesised result is at high risk of bias
Use the overall conclusion to inform the summary assessment of certainty of the evidence using e.g. GRADE approach