Key steps in umbrella reviews
Umbrella reviews have several steps (figure 2), of which four are key: systematic literature search and study selection, data extraction, statistical analysis and grading of evidence, and interpretation of findings.
Figure 2Key steps in an umbrella review
Researchers need to clearly define the research question of interest and consider which SRMAs are to be included by explicitly stating the eligibility criteria (box 2). A search algorithm must then be constructed to capture all SRMAs that deal with the defined research area. Eligible SRMAs are then selected by independent double screening of the literature search results. When multiple SRMAs on the same topic have partial or complete overlap, criteria are applied to decide which SRMAs to include.11 12 There are no set criteria, but researchers can choose the most recent meta-analysis, the meta-analysis with the largest number of studies, or (for epidemiological associations) the meta-analysis with the largest number of prospective studies. Researchers should also consider the quality of the SRMAs when deciding which to prioritise. In our working example for type 2 diabetes mellitus, the researchers chose the SRMA with the largest number of prospective studies, because prospective studies guarantee temporality in epidemiological associations.
Box 2Eligibility criteria, search algorithm, and data extraction in umbrella reviews
Eligibility criteria
In the definition of eligibility criteria, researchers can follow the PICO characteristics (population, intervention, comparison, and outcomes) for umbrella reviews of interventions. For umbrella reviews of epidemiological associations (either predictive or causal factors), researchers should also define the population(s), risk factor(s), and outcome(s) of interest to consider. By contrast with a single SRMA (systematic review and meta-analysis), umbrella reviews have much broader criteria, but the exact breadth should be carefully defined to ensure that the umbrella review is informative and comprehensive from a clinical or scientific perspective. In our working example, the population of interest was individuals not having type 2 diabetes mellitus at the beginning of the study, the risk factors of interest were any non-genetic factors, and the outcome was the development of the disorder.
Search algorithm
For an umbrella review, the search algorithm consists of two parts. The first part aims to identify research articles that are systematic reviews or meta-analyses (eg, using the keywords "systematic review*" OR meta-analys*). Alternatively, other search strings that aim to maximise retrieval of SRMAs could be used. The second part of the search algorithm should capture all the relevant articles about the research question. For this reason, this step should include all the relevant keywords about the research topic of interest; in this task, the inclusion of MeSH terms could facilitate capturing all the relevant terms. In our working example, the researchers used the keyword "diabetes" to capture articles relevant to type 2 diabetes mellitus.10 The final search algorithm is derived by combining the two parts of the algorithm using the boolean operator AND. Recommendations on database combinations to retrieve systematic reviews and meta-analyses based on empirical data have been published.17
Data extraction
In the data extraction process, for systematic reviews without a meta-analysis, the researchers should extract the number of eligible studies, the total sample size and (for binary outcomes) the number of events, the rationale for not performing a meta-analysis, and the descriptive conclusions. For systematic reviews with a meta-analysis, researchers should extract the number of eligible studies, the total sample size and (for binary outcomes) the total number of events, the study specific sample sizes and (for binary outcomes) the study specific numbers of events, the study specific effect estimates with relevant 95% confidence intervals, and the qualitative assessment as presented by the eligible SRMAs (if available).
Once the SRMAs to be included are agreed, two researchers should independently extract the required data from each eligible SRMA using a standardised data extraction form (box 2). With regards to the statistical analysis, researchers should use the study specific data extracted from each SRMA to repeat each meta-analysis separately rather than report the meta-analytical result as presented in the original SRMA. This process is important, because published SRMAs often use inappropriate meta-analytical statistical models, or they do not assess the heterogeneity between studies or the presence of small study effects. By re-running each meta-analysis, researchers can use the same array of methods for all considered meta-analyses and perform various heterogeneity or bias tests. To perform all the statistical analyses, researchers should extract data on study specific effect estimates with the relevant uncertainty estimates and the relevant sample sizes (as reported by the eligible SRMAs). However, some SRMAs offer insufficient information to perform all the desired, standardised analyses; this should be noted and discussed. In that case, researchers might decide to extract the required data from the primary studies.
After running the statistical analyses, researchers should assess the strength of the evidence. For questions about interventions (eg, drug treatments and other interventions in healthcare), researchers can use a validated tool, such as GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), to assess the strength of the evidence.13 For epidemiological associations, researchers can make an assessment of the strength of the evidence by considering several features including amount of evidence, level of significance, extent of heterogeneity between studies, and hints for potential bias (eg, small study effects, and excess significance bias) in each meta-analysis.5 6 An empirical evaluation of 57 umbrella reviews (including 3744 meta-analyses of observational studies) with a set of such criteria was recently published and shows that these criteria provide largely independent, complementary information.14 Researchers can also examine the temporality of epidemiological associations by performing the same assessment focusing only on prospective studies. In the working example for type 2 diabetes mellitus, the researchers graded the epidemiological associations using a predefined set of criteria. They then examined whether the most credible associations maintained their ranking in a sensitivity analysis of prospective studies.
After performing the statistical analyses and grading the strength of the evidence, researchers should report their results. Reporting might be similar to relevant reporting guidelines of systematic reviews for observational or randomised studies (ie, MOOSE (Meta-analysis Of Observational Studies in Epidemiology), and PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)).15 16 The difference is that the building block here is not one primary study, but a systematic review or meta-analysis.
A flowchart of literature search and study selection is helpful. Authors should report the eligible SRMAs identified, and those excluded because of overlap. For systematic reviews without statistical synthesis, researchers could state why meta-analysis was not performed and main conclusions. The findings of an umbrella review can be reported in both tabular and graphical format. Tables summarising all meta-analyses with some key features and results, and the grading of strength of the evidence for assessed interventions or associations are essential (box 3). Furthermore, if some SRMAs present a risk-of-bias assessment using standardised tools (eg, Joanna Briggs Institute critical appraisal tools for observational studies, or Cochrane risk-of-bias tool for randomised clinical trials), researchers can summarise the risk-of-bias assessment in each eligible SRMA using a tabular format. Additionally, visual plots can also facilitate the presentation and interpretation of results, such as the distribution of effect sizes and P values across the primary studies, or the distribution of summary effect sizes, P values, and heterogeneity estimates across the meta-analyses. In the working example on risk factors for the onset of type 2 diabetes mellitus, the researchers presented their results in both tabular and graphical format. They visually presented their results by providing a forest plot of the summary effect estimates for the meta-analyses with the highest strength of evidence, and a Manhattan plot (depicting the distribution of all P values in a −log10 format).10
Box 3Summarising results from multiple meta-analyses in umbrella reviews
Several key features and results of each meta-analysis should be reported, as shown below. In the working example of an umbrella review on type 2 diabetes mellitus, all the items listed below were provided in a tabulated manner for all the eligible meta-analyses (a total of 142 epidemiological associations)10:
Total number of cases or events (for binary outcomes)
Total sample size
Number of studies
Effect size metric
Meta-analysis method used (fixed effect or random effects, and related variants)
Summary effect estimate
95% confidence interval
95% prediction interval
P value for the summary effect estimate
Heterogeneity (eg, P value from Cochran’s Q test, I2, or estimate of variance between studies)
Effect size estimate of the largest study with the relevant 95% confidence interval
Suggestions of bias in relevant tests (eg, presence of small study effects and excess significance).
After reporting the results, the next step is interpretation. For umbrella reviews of interventions, interpretation should consider clinical relevance (including absolute risk reductions), potential additional biases in the design and conduct of randomised clinical trials and their meta-analyses, and issues of generalisability. For umbrella reviews of epidemiological associations, traditional considerations of confounding, reverse causality, selection bias, and information bias should be carefully considered either for all examined associations, or for a subset of associations (eg, the ones that seem to have the highest strength of evidence). Causal claims are notoriously difficult and typically only tentative. In our working example, the researchers interpreted the findings of the umbrella review by discussing the biological plausibility of the observed associations, and by systematically collecting published mendelian randomisation studies for type 2 diabetes mellitus.