Methods
A supplement to our living systematic review and network meta-analysis of drug treatments for covid-19 includes a protocol of our methods.2
Search
The present study uses the search strategy of our living review.2 A supplement to our drug treatment publication includes the full strategy.2 Briefly, we performed daily searches of the WHO covid-19 database—a comprehensive multilingual source of global published and preprint literature on covid-19 (https://search.bvsalud.org/global-literature-on-novel-coronavirus-2019-ncov/). Prior to its merging with the WHO covid-19 database on 9 October 2020, we searched the US Centres for Disease Control and Prevention's covid-19 research articles downloadable database. Our search also included six Chinese databases: Wanfang, Chinese Biomedical Literature, China National Knowledge Infrastructure, VIP, Chinese Medical Journal Net (preprints), and ChinaXiv (preprints). A validated machine learning model facilitated efficient identification of randomised trials.14 We searched WHO information sources from 1 December 2019 to 9 June 2021 and the Chinese literature from conception of the databases to 20 February 2021.
Our team supplemented the search by ongoing surveillance of the Living Overview of the Evidence covid-19 platform by the Epistemonikos Foundation (https://app.iloveevidence.com/loves/5e6fdb9669c00e4ac072701d) and the Norwegian Institute of Public Health's systematic and living map on covid-19 evidence (https://www.fhi.no/en/qk/systematic-reviews-hta/map/). We also included data from two WHO-sponsored prospective meta-analyses.11 15
Study selection
As part of the living systematic review and network meta-analysis,2 pairs of reviewers, following calibration exercises, worked independently and in duplicate to screen titles and abstracts of search records and subsequently the full texts of records determined potentially eligible at the title and abstract screening stage. We linked preprint reports with their subsequent publications based on trial registration numbers, authors, and other trial characteristics. Reviewers resolved discrepancies by discussion, and when necessary, by adjudication with a third party reviewer.
This review included preprint and peer reviewed reports of trials that compared interleukin 6 receptor blockers with standard care, placebo, or corticosteroids or that compared corticosteroids with standard care or placebo in patients with suspected, probable, or confirmed covid-19. We did not set any restrictions on severity of illness, setting, or language of publication.
Data collection
As part of the living systematic review and network meta-analysis,2 for each eligible trial, pairs of reviewers, following training and calibration exercises, independently extracted trial characteristics (trial registration, publication status, study design), patient characteristics (country, age, sex, type of care, severity of covid-19 symptoms), and outcomes of interest (number of participants analysed and number of participants who experienced an event) using a standardised, pilot tested data extraction form. Reviewers resolved discrepancies by discussion and, when necessary, with adjudication by a third party. We updated our data when a study preprint became available as a peer reviewed publication. For this review, we focused on all cause mortality closest to 90 days.
To assess risk of bias, reviewers, following training and calibration exercises, used a revision of the Cochrane tool for assessing risk of bias in randomised trials (RoB 2.0).16 Reviewers resolved discrepancies by discussion and, when necessary, by third party adjudication. A supplement to our drug treatment publication includes our modified risk-of-bias tool.2
Statistical analysis
Our network meta-analysis compared tocilizumab with corticosteroids, tocilizumab without corticosteroids, sarilumab with corticosteroids, sarilumab without corticosteroids, corticosteroids, and standard care or placebo, using a bayesian framework with a plausible prior for the variance parameter and a uniform prior for the effect parameter.17 We summarised the effect of interventions on mortality using odds ratios and corresponding 95% credible intervals.
We classified trials in which all patients randomised to tocilizumab or sarilumab received or did not receive corticosteroids into (1) tocilizumab or sarilumab nodes with corticosteroids or (2) tocilizumab or sarilumab nodes without corticosteroids, respectively. For trials in which some patients received corticosteroids in combination with tocilizumab or sarilumab, we used subgroup data within trials to split trial participants into tocilizumab or sarilumab nodes with corticosteroids and tocilizumab or sarilumab nodes without corticosteroids. The same approach was used for standard care. We grouped patients in the standard care arm who received corticosteroids into the corticosteroid node and patients in the standard care arm who did not receive corticosteroids into the standard care without corticosteroids node. We classified trials that compared corticosteroids with standard care or placebo into corticosteroids and standard care nodes.
We performed network meta-analysis using the gemtc package of R version 3.6.3 (RStudio, Boston, MA) and pairwise meta-analyses using the bayesmeta package. Three Markov chains with 100 000 iterations after an initial burn-in of 10 000 and a thinning of 10 and used node splitting models were used to assess local incoherence and to obtain indirect estimates. We produced network plots using the network map command of Stata version 17.0 (StataCorp, College Station, TX).18
We performed both fixed effect and random effects network meta-analysis. Because estimates from the random effects model proved to have credible intervals that were implausibly wide owing to the uncertainty around the heterogeneity estimate, we presented results from the fixed effect meta-analysis as the primary analysis and random effects meta-analysis as a sensitivity analysis.19
Certainty of evidence
To facilitate interpretation of results, we calculated absolute effects for mortality using baseline risk data from the Centres for Disease Control and Prevention on patients who were admitted to hospital for covid-19.20 21 We assessed the certainty of evidence using a minimally contextualised GRADE approach (grading of recommendations, assessment, development, and evaluations) for network meta-analysis with a null effect as the threshold of importance.22–25 The minimally contextualised approach considers only whether credible intervals include the null effect and does not consider whether plausible effects, captured by credible intervals, include both important and trivial effects. Based on a survey of the authors of our living systematic review and network meta-analysis, to evaluate certainty of no benefit (or no effect), we used a 1% risk difference threshold of the 95% credible interval.
Two reviewers with experience in applying the GRADE approach rated each domain for each comparison and resolved discrepancies by consensus. Reviewers rated the certainty for each comparison and outcome as high, moderate, low, or very low, based on considerations of risk of bias, inconsistency, indirectness, publication bias, intransitivity, incoherence (difference between direct and indirect effects), and imprecision.
Patient and public involvement
Patients were involved in outcome selection, interpretation of results, and the generation of parallel recommendations, as part of the WHO Rapid Recommendations initiative, in partnership with The BMJ and MAGIC Evidence Ecosystem Foundation.13 Our results will be disseminated according to WHO recommendations.