In this article, Lazaros Belbasis and colleagues explain the rationale for umbrella reviews and the key steps involved in conducting an umbrella review, using a working example.
An umbrella review is a systematic collection and assessment of multiple systematic reviews and meta-analyses on a specific research topic
Umbrella reviews were developed to deal with the increasing number of systematic reviews and meta-analyses in biomedical literature
The validity of umbrella reviews depends on the coverage and quality of both the primary studies and the available systematic reviews and meta-analyses
The key output of umbrella reviews is a systematic and standardised assessment of all the evidence on a broad but well defined research topic (eg, treatment effects of multiple interventions for a particular disease, or adjusted or unadjusted associations of multiple risk factors with a particular disease) based on published systematic reviews and meta-analyses
Currently, clinical researchers have used systematic reviews and meta-analyses (SRMAs) for most clinical and epidemiological questions of interest. Occasionally, researchers might need to examine the evidence not just on a single question but on several different questions on a given topic. Umbrella reviews (ie, a systematic review of SRMAs) could be an appropriate option for these situations.
Umbrella reviews are systematic collections and assessments of multiple SRMAs done on a specific research topic.1 2 The decision to perform an umbrella review depends on the number of available SRMAs (figure 1). An umbrella review is informative when multiple SRMAs have already been published on a specific research topic. When only a trivial number of relevant SRMAs are available, performing a new SRMA is more appropriate and more informative. When multiple outdated SRMAs are available, updating the existing SRMAs is more important. Like all research studies, umbrella reviews have advantages and disadvantages (box 1).
Advantages
They offer a bird eye’s view of multiple interventions for a specific medical condition or multiple epidemiological associations for a specific medical condition (exposure wide approach) or a specific risk factor (phenome wide approach)
They save valuable research resources by avoiding systematic searches from scratch, because they take advantage of existing systematic reviews
They identify the gaps in a specific research field and can inform recommendations for further research
They present an overview of study quality, effect sizes, uncertainty, heterogeneity, and hints of bias across a well defined but broad research field
They present and compare evidence between different interventions or different epidemiological associations, providing a comprehensive picture about the relative strengths and weaknesses of the evidence for each intervention or epidemiological association
Disadvantages
The validity of umbrella review findings depends on the quality of the eligible systematic reviews and meta-analyses
They do not include information for interventions or epidemiological associations that have not been examined in systematic reviews and meta-analyses
Quality problems and biases might also exist in primary studies and in the umbrella review process itself, and these problems and biases could be compounded and difficult to clarify
Decision process regarding whether to perform an umbrella review. SRMA=systematic review and meta-analysis
The two most common applications of umbrella reviews deal with treatment effects of interventions and epidemiological associations of exposures. Umbrella reviews of interventions typically focus on one or more diseases of interest and assess SRMAs on the treatment effects of all interventions for those diseases.3 Umbrella reviews of epidemiological associations often follow either a phenome wide approach or an exposure wide approach. In the phenome wide approach, researchers consider the (adjusted or unadjusted) associations of a particular risk factor with any disease or phenotype.4 In the exposure wide approach, researchers consider the (adjusted or unadjusted) associations of multiple risk factors with a specific disease or phenotype.5–7 Umbrella reviews can also be designed to summarise SRMAs on other types of studies, such as prevalence studies and diagnostic accuracy studies.8 9 From a clinical point of view, the key output of an umbrella review is a comprehensive, systematic, and critical summary of multiple intervention or epidemiological studies (or other types of studies) based on published SRMAs.
As a working example, we will use an umbrella review summarising SRMAs on the non-genetic risk factors for type 2 diabetes mellitus, which included 86 eligible articles (142 epidemiological associations) of SRMAs.10 With so many factors being examined for association with risk of type 2 diabetes mellitus, an umbrella review can obtain a bird eye’s view of the evidence on unadjusted or adjusted effects between particular risk factors and onset of the disorder, in terms of measures such as odds ratios and hazard ratios.
Umbrella reviews have several steps (figure 2), of which four are key: systematic literature search and study selection, data extraction, statistical analysis and grading of evidence, and interpretation of findings.
Key steps in an umbrella review
Researchers need to clearly define the research question of interest and consider which SRMAs are to be included by explicitly stating the eligibility criteria (box 2). A search algorithm must then be constructed to capture all SRMAs that deal with the defined research area. Eligible SRMAs are then selected by independent double screening of the literature search results. When multiple SRMAs on the same topic have partial or complete overlap, criteria are applied to decide which SRMAs to include.11 12 There are no set criteria, but researchers can choose the most recent meta-analysis, the meta-analysis with the largest number of studies, or (for epidemiological associations) the meta-analysis with the largest number of prospective studies. Researchers should also consider the quality of the SRMAs when deciding which to prioritise. In our working example for type 2 diabetes mellitus, the researchers chose the SRMA with the largest number of prospective studies, because prospective studies guarantee temporality in epidemiological associations.
In the definition of eligibility criteria, researchers can follow the PICO characteristics (population, intervention, comparison, and outcomes) for umbrella reviews of interventions. For umbrella reviews of epidemiological associations (either predictive or causal factors), researchers should also define the population(s), risk factor(s), and outcome(s) of interest to consider. By contrast with a single SRMA (systematic review and meta-analysis), umbrella reviews have much broader criteria, but the exact breadth should be carefully defined to ensure that the umbrella review is informative and comprehensive from a clinical or scientific perspective. In our working example, the population of interest was individuals not having type 2 diabetes mellitus at the beginning of the study, the risk factors of interest were any non-genetic factors, and the outcome was the development of the disorder.
For an umbrella review, the search algorithm consists of two parts. The first part aims to identify research articles that are systematic reviews or meta-analyses (eg, using the keywords "systematic review*" OR meta-analys*). Alternatively, other search strings that aim to maximise retrieval of SRMAs could be used. The second part of the search algorithm should capture all the relevant articles about the research question. For this reason, this step should include all the relevant keywords about the research topic of interest; in this task, the inclusion of MeSH terms could facilitate capturing all the relevant terms. In our working example, the researchers used the keyword "diabetes" to capture articles relevant to type 2 diabetes mellitus.10 The final search algorithm is derived by combining the two parts of the algorithm using the boolean operator AND. Recommendations on database combinations to retrieve systematic reviews and meta-analyses based on empirical data have been published.17
In the data extraction process, for systematic reviews without a meta-analysis, the researchers should extract the number of eligible studies, the total sample size and (for binary outcomes) the number of events, the rationale for not performing a meta-analysis, and the descriptive conclusions. For systematic reviews with a meta-analysis, researchers should extract the number of eligible studies, the total sample size and (for binary outcomes) the total number of events, the study specific sample sizes and (for binary outcomes) the study specific numbers of events, the study specific effect estimates with relevant 95% confidence intervals, and the qualitative assessment as presented by the eligible SRMAs (if available).
Once the SRMAs to be included are agreed, two researchers should independently extract the required data from each eligible SRMA using a standardised data extraction form (box 2). With regards to the statistical analysis, researchers should use the study specific data extracted from each SRMA to repeat each meta-analysis separately rather than report the meta-analytical result as presented in the original SRMA. This process is important, because published SRMAs often use inappropriate meta-analytical statistical models, or they do not assess the heterogeneity between studies or the presence of small study effects. By re-running each meta-analysis, researchers can use the same array of methods for all considered meta-analyses and perform various heterogeneity or bias tests. To perform all the statistical analyses, researchers should extract data on study specific effect estimates with the relevant uncertainty estimates and the relevant sample sizes (as reported by the eligible SRMAs). However, some SRMAs offer insufficient information to perform all the desired, standardised analyses; this should be noted and discussed. In that case, researchers might decide to extract the required data from the primary studies.
After running the statistical analyses, researchers should assess the strength of the evidence. For questions about interventions (eg, drug treatments and other interventions in healthcare), researchers can use a validated tool, such as GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), to assess the strength of the evidence.13 For epidemiological associations, researchers can make an assessment of the strength of the evidence by considering several features including amount of evidence, level of significance, extent of heterogeneity between studies, and hints for potential bias (eg, small study effects, and excess significance bias) in each meta-analysis.5 6 An empirical evaluation of 57 umbrella reviews (including 3744 meta-analyses of observational studies) with a set of such criteria was recently published and shows that these criteria provide largely independent, complementary information.14 Researchers can also examine the temporality of epidemiological associations by performing the same assessment focusing only on prospective studies. In the working example for type 2 diabetes mellitus, the researchers graded the epidemiological associations using a predefined set of criteria. They then examined whether the most credible associations maintained their ranking in a sensitivity analysis of prospective studies.
After performing the statistical analyses and grading the strength of the evidence, researchers should report their results. Reporting might be similar to relevant reporting guidelines of systematic reviews for observational or randomised studies (ie, MOOSE (Meta-analysis Of Observational Studies in Epidemiology), and PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)).15 16 The difference is that the building block here is not one primary study, but a systematic review or meta-analysis.
A flowchart of literature search and study selection is helpful. Authors should report the eligible SRMAs identified, and those excluded because of overlap. For systematic reviews without statistical synthesis, researchers could state why meta-analysis was not performed and main conclusions. The findings of an umbrella review can be reported in both tabular and graphical format. Tables summarising all meta-analyses with some key features and results, and the grading of strength of the evidence for assessed interventions or associations are essential (box 3). Furthermore, if some SRMAs present a risk-of-bias assessment using standardised tools (eg, Joanna Briggs Institute critical appraisal tools for observational studies, or Cochrane risk-of-bias tool for randomised clinical trials), researchers can summarise the risk-of-bias assessment in each eligible SRMA using a tabular format. Additionally, visual plots can also facilitate the presentation and interpretation of results, such as the distribution of effect sizes and P values across the primary studies, or the distribution of summary effect sizes, P values, and heterogeneity estimates across the meta-analyses. In the working example on risk factors for the onset of type 2 diabetes mellitus, the researchers presented their results in both tabular and graphical format. They visually presented their results by providing a forest plot of the summary effect estimates for the meta-analyses with the highest strength of evidence, and a Manhattan plot (depicting the distribution of all P values in a −log10 format).10
Several key features and results of each meta-analysis should be reported, as shown below. In the working example of an umbrella review on type 2 diabetes mellitus, all the items listed below were provided in a tabulated manner for all the eligible meta-analyses (a total of 142 epidemiological associations)10:
Total number of cases or events (for binary outcomes)
Total sample size
Number of studies
Effect size metric
Meta-analysis method used (fixed effect or random effects, and related variants)
Summary effect estimate
95% confidence interval
95% prediction interval
P value for the summary effect estimate
Heterogeneity (eg, P value from Cochran’s Q test, I2, or estimate of variance between studies)
Effect size estimate of the largest study with the relevant 95% confidence interval
Suggestions of bias in relevant tests (eg, presence of small study effects and excess significance).
After reporting the results, the next step is interpretation. For umbrella reviews of interventions, interpretation should consider clinical relevance (including absolute risk reductions), potential additional biases in the design and conduct of randomised clinical trials and their meta-analyses, and issues of generalisability. For umbrella reviews of epidemiological associations, traditional considerations of confounding, reverse causality, selection bias, and information bias should be carefully considered either for all examined associations, or for a subset of associations (eg, the ones that seem to have the highest strength of evidence). Causal claims are notoriously difficult and typically only tentative. In our working example, the researchers interpreted the findings of the umbrella review by discussing the biological plausibility of the observed associations, and by systematically collecting published mendelian randomisation studies for type 2 diabetes mellitus.
Conducting an umbrella review has some potential challenges. Umbrella reviews can deal with a topic comprehensively when primary studies and SRMAs have full coverage of the topic, otherwise gaps in the evidence can exist. The validity of an umbrella review depends on the quality of both the primary studies and the existing SRMAs. Cross checking the original reports to confirm whether all the data extraction for all the eligible SRMAs is correct would be impossible. But occasionally, umbrella review authors should go back to original reports to collect additional information (eg, sample size, and number of cases) to allow performing calculations in a standardised way and assessing criteria for strength of the evidence. Moreover, if some data are deemed spurious, the original reports should also be examined to remove errors. Moreover, SRMAs often might use eligibility criteria that deviate from what is intended in the umbrella review. For example, the umbrella review might wish to focus only on randomised trials, but the existing SRMAs might also contain observational studies that should be separated.
Clinicians and other readers should search for specific characteristics indicating a good quality umbrella review. They should explicitly state their eligibility criteria, verifying that these criteria fit with their clinical question; repeat the statistical analyses to estimate all the relevant features about heterogeneity between studies, 95% prediction intervals and related statistical biases; and grade the evidence according to a set of criteria and discuss various other potential biases.
Umbrella reviews can provide a bird eye’s view of the currently available evidence on broad research topics and a thorough assessment of strength of the available evidence, and they can indicate potential priorities for future research. Clinicians and other users should look to umbrella reviews for a systematic and critical summary of the evidence in a broad research topic (eg, multiple risk factors or predictors for a particular disease, multiple health related effects of an exposure, or multiple interventions for a particular disease). From an epidemiological perspective, the findings of an umbrella review can be used to identify which epidemiological associations could get tested further using more sophisticated causal inference methods, such as mendelian randomisation. From a clinical perspective, the findings of an umbrella review can be used by clinicians and trialists to inform the design of preventative or therapeutic interventions through randomised clinical trials.
In our working example, the researchers eventually summarised and assessed the evidence on 142 epidemiological associations.10 By contrast with relevant narrative reviews on risk factors for type 2 diabetes mellitus that selectively report some associations, this umbrella review captured all the relevant SRMAs in a systematic manner. Furthermore, SRMAs usually focus on the presence of a significant effect, whereas the umbrella review example also considered issues related to heterogeneity between studies, confounding, and other biases. In our working example, 116 of 142 epidemiological associations presented a significant effect at P<0.05. However, only 11 presented strong evidence based on a set of criteria that consider level of significance, heterogeneity between studies, 95% prediction intervals, small study effects, and excess significance bias. An important advantage of this umbrella review is that readers can see that specific risk factors have the strongest evidence while others also have strong support, and they can observe the relative magnitude of all the associations.
