Narrative synthesis and synthesis when you can’t perform a statistical meta-analysis
Complexity in a systematic review may arise for a number of reasons. The topic itself may be complex with multiple, and poorly understood, interactions between intervention, mediators, and outcomes. There may be high levels of heterogeneity making it difficult to synthesise the data from the included studies. Heterogeneity may be statistical, or due to methodological characteristics of the included studies, or there may be conceptual or clinical heterogeneity for example diversity in the interventions, populations, or outcomes in the included studies. Methodological developments, such as network meta-analysis and meta-regression, have increased the potential for statistical synthesis to be used in complex systematic reviews. However, there are many instances where heterogeneity means that statistical pooling is not appropriate meaning that quantitative data have to be synthesised narratively. A common criticism of narrative synthesis is that it is difficult to maintain transparency in the interpretation of the data and development of conclusions. This ultimately threatens the value of the synthesis and the extent to which the conclusions can be trusted.
Narrative synthesis is a generic term and there are many other terms used to describe approaches which rely heavily on narrative, rather than statistical, synthesis. For example, thematic synthesis, realist synthesis, meta-ethnography, framework synthesis etc.. Some of these approaches are developed specifically for synthesis of qualitative data. As such these reviews have a different, if often complementary, focus to reviews aiming to establish a quantitative estimate of effect size. Narrative reviews may be used to synthesise ideas and theories. In these cases narrative approaches are used, not because meta-analysis is impractical for the reasons given above, but because the object of interest (the intervention) has not been sufficiently understood. The role of the narrative review in these circumstances is to synthesise the multiple purposes and theories about how and why an intervention might work or not in different circumstances. This kind of review is useful for scoping the landscape in relation to particular interventions.
In addition to developments to improve meta-analytical methods to handle heterogeneity, there have also been valuable developments to promote improved methods for the synthesis of qualitative data. By comparison there is little guidance on narrative synthesis of quantitative data despite a narrative approach being used in around half of all health related systematic reviews (see recently published SWiM reporting guidelines).
Below is an introductory list of resources which may be useful if you are conducting a review where the data cannot be synthesised statistically. It is important to note that authors writing about narrative approaches to synthesising the literature have different views about the best way to conduct such reviews – all argue for robust approaches but the nature of these approaches vary.
Main website for realist methods
Center for Advancement in Realist Evaluation and Synthesis (CARES) https://realistmethodology-cares.org/
Resources for mixed studies reviews
Wiki Toolkit for Mixed Studies Reviews
Reporting standards
SWiM (Synthesis Without Meta-analysis) https://swim.sphsu.gla.ac.uk/
Synthesis of quantitative intervention effect data which does not rely on a meta-analysis of standardised effect sizes.
Guidance on realist synthesis & RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) http://www.ramesesproject.org/
Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Medical Research Methodology. 2012;12(1):181. http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-12-181
Bibliography including methods guidance and tools
Anderson L, Oliver S, Michie S, Rehfuess E, Noyes E, Shemilt I. Investigating complexity in systematic reviews of interventions by using a spectrum of methods.Journal of Clinical Epidemiology.0; http://dx.doi.org/10.1016/j.jclinepi.2013.06.014
Anderson LM, Petticrew M, Rehfuess E, Armstrong R, Ueffing E, Baker P, et al. Using logic models to capture complexity in systematic reviews. Research Synthesis Methods. 2011;2(1):33-42. http://onlinelibrary.wiley.com/doi/10.1002/jrsm.32/abstract
Boaz A et al (2006) ‘A multitude of syntheses: a comparison of five approaches from diverse policy fields’. Evidence & Policy 4(2) 2006
Burford B, Lewin S, Welch V, Rehfuess E, Waters E. Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making.Journal of Clinical Epidemiology.0; http://dx.doi.org/10.1016/j.jclinepi.2013.06.017
Campbell M, Katikireddi S V, Sowden A and Thomson H (2019). "Lack of transparency in reporting narrative synthesis of quantitative data: a methodological assessment of systematic reviews." Journal of Clinical Epidemiology 105: 1-9.
Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, Hartmann-Boyce J, Ryan R, Shepperd S, Thomas J, Welch V, Thomson H. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline BMJ 2020;368:l6890 http://dx.doi.org/10.1136/bmj.l6890
Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Systematic Reviews 2012;1(1):28.
Gough D, Oliver S, Thomas J, editors. An introduction to systematic reviews. London: Sage, 2012.
Ioannidis, J. P. A., N. A. Patsopoulos, et al. (2008). Reasons or excuses for avoiding meta-analysis in forest plots. British Medical Journal336(7658): 1413-1415.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339(jul21_1):b2700-.
McKenzie JE, Brennan SE. Chapter 12: Synthesizing and presenting findings using other methods. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019.
McKenzie JE, Brennan SE, Ryan R, Thomson HJ, Johnston RV. Chapter 9: Summarizing the study characteristics and preparing for synthesis. In Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions. London: Cochrane.
McKenzie JE, Brennan SE, Ryan R, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions. London: Cochrane.
Mozygemba K, Refolo P, Sacchini D, Tummers M, Rehfuess E. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. http://www.integrate-hta.eu/wp-content/uploads/2016/02/Guidance-on-choosing-qualitative-evidence-synthesis-methods-for-use-in-HTA-of-complex-interventions.pdfOgilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health 2005;59(9):804-08.
Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019
Noyes J, Hendry M, Booth A, Chandler J, Lewin S, Glenton C, et al. Current use was established and Cochrane guidance on selection of social theories for systematic reviews of complex interventions was developed. Journal of Clinical Epidemiology. http://www.jclinepi.com/article/S0895-4356%2816%2900005-6/abstract
Pawson R. et al. (2005) ‘Realist review – a new method of systematic review designed for complex policy interventions’. Journal of Halth Services Research & Policy 10 (1): 21-34 doi: 10.1258/1355819054308530
Petticrew M. Why certain systematic reviews reach uncertain conclusions. British Medical Journal 2003;326:756-8.
Petticrew M. Presumed innocent: Why we need systematic reviews of social policies. American Journal of Preventive Medicine 2003;24(1):2-3.
Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ 2001;322(7278):98-101.
Petticrew M, Roberts H (2006) Systematic Reviews in the Social Sciences: A practical guide. Oxford: Blackwell Publishing.
Petticrew M, Anderson L, Elder R, Grimshaw J, Hopkins D, Hahn R, Krause L, Kristjansson E, Mercer S, Sipe T, Tugwell P, Ueffing E, Waters E, Welch V.Complex interventions and their implications for systematic reviews: a pragmatic approach.Journal of Clinical Epidemiology.0; http://dx.doi.org/10.1016/j.jclinepi.2013.06.004
Pigott T, Shepperd S. Identifying, documenting, and examining heterogeneity in systematic reviews of complex interventions.Journal of Clinical Epidemiology.0; http://dx.doi.org/10.1016/j.jclinepi.2013.06.013
Popay J, Roberts H , Sowden A , Petticrew M , Arai L, Rodgers M , Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews: a product of the ESRC methods programme . Lancaster, 2006. http://www.lancaster.ac.uk/shm/research/nssr/research/dissemination/publications.php
Pope, C., N. Mays and J. Popay (2007). Synthesizing qualitative and quantitative health evidence, Open University Press.
Squires J E, Valentine J C & Grimshaw J. Systematic reviews of complex interventions: framing the review question.Journal of Clinical Epidemiology.0; http://dx.doi.org/10.1016/j.jclinepi.2013.05.013
Theory in complex reviews Wiki: http://theoryinreviews.pbworks.com/w/page/85156132/FrontPage
Thomson, H. (2013). "Improving Utility of Evidence Synthesis for Healthy Public Policy: the Three Rs (Relevance, Rigor, and Readability [and Resources])." American Journal of Public Health 103(8): e17-e23.
Thomson H and Thomas S. 2013.The effect direction plot: visual display of non-standardised effects across multiple outcome domains.Research Synthesis Methods.4; 1; 95-10110.1002/jrsm.1060. http://onlinelibrary.wiley.com/doi/10.1002/jrsm.1060/abstract