Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis

Highlights

• The application of the Grading of Recommendations Assessments, Development, and Evaluation approach to a number of network meta-analyses in the 3 years since the original guidance publication has led to advances that have strengthened the conceptual basis.

• We present, discuss, and illustrate four conceptual advances. These are based on two principles: we need to rate the certainty of the evidence of each pairwise comparison within a network separately and that we need to consider both the direct and indirect evidence contributing to each network estimate.

• Although maximizing the efficiency of the process is desirable, as illustrated in the conceptual advances, use of these strategies requires careful judgment.

Abstract

This article describes conceptual advances of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group guidance to evaluate the certainty of evidence (confidence in evidence, quality of evidence) from network meta-analysis (NMA). Application of the original GRADE guidance, published in 2014, in a number of NMAs has resulted in advances that strengthen its conceptual basis and make the process more efficient. This guidance will be useful for systematic review authors who aim to assess the certainty of all pairwise comparisons from an NMA and who are familiar with the basic concepts of NMA and the traditional GRADE approach for pairwise meta-analysis. Two principles of the original GRADE NMA guidance are that we need to rate the certainty of the evidence for each pairwise comparison within a network separately and that in doing so we need to consider both the direct and indirect evidence. We present, discuss, and illustrate four conceptual advances: (1) consideration of imprecision is not necessary when rating the direct and indirect estimates to inform the rating of NMA estimates, (2) there is no need to rate the indirect evidence when the certainty of the direct evidence is high and the contribution of the direct evidence to the network estimate is at least as great as that of the indirect evidence, (3) we should not trust a statistical test of global incoherence of the network to assess incoherence at the pairwise comparison level, and (4) in the presence of incoherence between direct and indirect evidence, the certainty of the evidence of each estimate can help decide which estimate to believe.

Introduction

In 2014, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group presented guidance to evaluate the certainty of the evidence (confidence in evidence, quality of evidence) from network meta-analysis (NMA). [1] This guidance represented a response to the need for establishing the certainty of the evidence for each paired comparison within an NMA and the desirability of implementing widely used GRADE criteria [2] to inform those judgments [3].

The application of GRADE's approach to rate the certainty of the evidence in an NMA may appear onerous in networks with many interventions. While in the simplest network with only three treatments (e.g., treatments A, B, and C), researchers must undertake the certainty assessment three times (i.e., they must address the direct, indirect, and network estimates for A vs. B, A vs. C, and B vs. C), the requirement in a network with 6 treatments is 15 assessments and in a network with 12 treatments 66 assessments. Moreover, the assessment requires repetition for each outcome of interest.

The application of the GRADE approach to a number of NMAs [4], [5], [6], [7], [8] in the 3 years since the original guidance publication has led to advances that have strengthened the conceptual basis, dealt with challenges that have arisen in applying the approach, and—most relevant to the volume of work required in applying the GRADE approach—may make the process of assessing the certainty of the evidence more efficient. In this article, we describe these advances and their rationale and provide illustrative examples. We focus on guidance for systematic reviewers who aim to rate the certainty of the evidence of all the pairwise comparisons from an NMA, regardless of whether there is high certainty from traditional direct comparisons to inform clinical decision-making. The discussion assumes a familiarity with the basic concepts of indirect evidence and NMA and with the GRADE approach to rating certainty of evidence for bodies of evidence in conventional paired comparison meta-analysis and is restricted to NMAs of randomized trials. This article describes official guidance from the GRADE working group.