Objective To assess the quality of reporting of adverse events in clinical trials of covid-19 drugs based on the CONSORT (Consolidated Standards of Reporting Trials) harms extension and according to clinical trial design, and to examine reporting of serious adverse events in drug trials published on PubMed versus clinical trial summaries on ClinicalTrials.gov.
Design Systematic review.
Data sources PubMed and ClinicalTrials.gov registries were searched from 1 December 2019 to 17 February 2022.
Eligibility criteria for selecting studies Randomised clinical trials evaluating the efficacy and safety of drugs used to treat covid-19 disease in participants of all ages with suspected, probable, or confirmed SARS-CoV-2 infection were included. Clinical trials were screened on title, abstract, and text by two authors independently. Only articles published in French and English were selected. The Cochrane risk of bias tool for randomised trials (RoB 2) was used to assess risk of bias.
Results The search strategy identified 1962 randomised clinical trials assessing the efficacy and safety of drugs used to treat covid-19, published in the PubMed database; 1906 articles were excluded after screening and 56 clinical trials were included in the review. Among the 56 clinical trials, no study had a high score for quality of reporting of adverse events, 60.7% had a moderate score, 33.9% had a low score, and 5.4% had a very low score. All clinical trials with a very low score for quality of reporting of adverse events were randomised open label trials. For reporting of serious adverse events, journal articles published on PubMed under-reported 51% of serious adverse events compared with clinical trial summaries published on ClinicalTrials.gov.
Conclusions In one in three published clinical trials on covid-19 drugs, the quality of reporting of adverse events was low or very low. Differences were found in the number of serious adverse events reported in journal articles versus clinical trial summaries. During the covid-19 pandemic, risk assessment of drugs in clinical trials of covid-19 drugs did not comply with good practice recommendations for publication of results.
Systematic review registration European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) EUPAS45959.
- Drug-related side effects and adverse reactions
- Clinical trial
- Drug therapy
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
WHAT IS ALREADY KNOWN ON THIS TOPIC
A preliminary review of clinical trials evaluating remdesivir for covid-19 treatment highlighted poor accuracy in the reporting of adverse events
However, no large studies have investigated other drugs used to treat covid-19, including monoclonal antibodies, antiviral agents, and immunomodulators
WHAT THIS STUDY ADDS
This systematic review found inadequacies and inconsistencies in the quality of reporting of adverse events of drugs used for the treatment of covid-19 in published articles
One in three studies had a low or very low score for quality of reporting according to the CONSORT (Consolidated Standards of Reporting Trials) statement
Compared with serious adverse events reported in trial summaries on ClinicalTrials.gov, about 51% of serious events were not reported in published clinical trials
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE, OR POLICY
Use of the new CONSORT harms extension might improve reporting of safety data
Some publications under-reported adverse events and therefore researchers and clinicians should consult several sources of information to correctly establish the safety profile of drugs
The pharmacovigilance of these drugs is affected because assessment of the benefit-risk ratio of these covid-19 drugs based on these clinical trials is not precise
Since its outbreak in December 2019, covid-19 has been a global health emergency. During the covid-19 pandemic, researchers developed several drugs (monoclonal antibodies, antiviral agents, and immunomodulators) to prevent the rapid spread of disease based on an understanding of the mechanism of action of the SARS-CoV-2 virus.1 Monoclonal antibodies developed for the treatment of covid-19 disease (bamlanivimab, bamlanivimab-etesevimab, casirivimab-imdevimab, cilgavimab-tixagevimab, regdanvimab, and sotrovimab) block the entry of the virus into the cell by neutralising the spike protein. Antiviral agents (molnupiravir, nirmatrelvir, PF-07321332 ritonavir, and remdesivir) block multiplication of the virus inside the cell.2 Immunomodulators, used to prevent the inflammatory reaction in the late stages of covid-19 disease, include the interleukin 6 receptor inhibitors, tocilizumab and sarilumab, monoclonal antibodies that neutralise membrane and nuclear interleukin 6 receptors, and the Janus kinase inhibitors, ruxolitinib, tofacitinib, and baricitinib.
The European Medicines Agency or the US Food and Drug Administration, or both, recommended sotrovimab, remdesivir, nirmatrelvir, PF-07321332 ritonavir, and casirivimab-imdevimab as curative treatment for covid-19. For prophylaxis before infection with the SARS-CoV-2 virus, these agencies recommended cilgavimab-tixagevimab, bamlanivimab, and bamlanivimab-etesevimab, and for prophylaxis after infection, casirivimab-imdevimab, regdanvimab, and molnupiravir. The World Health Organization strongly recommended Janus kinase inhibitors, specifically baricitinib, ruxolitinib, and tofacitinib, and interleukin 6 receptor blockers (tocilizumab or sarilumab), in patients with severe and critical covid-19 disease.3
These drugs were evaluated based on the results of clinical trials and subsequently obtained emergency use authorisation from the FDA and marketing authorisation from the European Medicines Agency. Because these drugs are used in large populations with little experience of their use in SARS-CoV-2 infection, evaluation of safety data is crucial, especially to estimate the balance between benefit and risk. Assessment of the quality of reporting of adverse events is therefore important, to improve our knowledge of the safety of these drugs. Adverse events of drugs used to treat covid-19 disease might involve many patients and be potentially serious. For example, the WHO guideline on drugs for the treatment of covid-19 disease recommended monitoring the risk of Janus kinase inhibitors, especially serious infections.3 Although a preliminary review of clinical trials evaluating remdesivir for covid-19 disease highlighted poor accuracy in the reporting of adverse events, no large study has assessed other drugs used for the treatment of covid-19, including monoclonal antibodies, other antiviral agents, and immunomodulators.4
In this systematic review, our main aim was to assess the quality of reporting of adverse events in clinical trials of all drugs used in the treatment of covid-19 disease (except vaccines and glucocorticoids). Secondary objectives were comparison of the quality of reporting of adverse events according to the design of the clinical trial. We also examined reporting of serious adverse events in journal articles of covid-19 drug trials compared with their trial summaries on the ClinicalTrials.gov website.
Materials and methods
This study used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for study design, search protocol, screening, and reporting (online supplemental table 1).
Randomised clinical trials evaluating the efficacy and safety of drugs used to treat covid-19 disease in participants of all ages with suspected, probable, or confirmed SARS-CoV-2 infection were included in our study. We considered monoclonal antibodies (bamlanivimab, bamlanivimab-etesevimab, casirivimab-imdevimab, cilgavimab-tixagevimab, regdanvimab, and sotrovimab), antiviral agents (remdesivir, molnupiravir, and nirmatrelvir-ritonavir), and immunomodulators (tocilizumab, sarilumab, ruxolitinib, tofacitinib, and baricitinib). Eligible randomised clinical trials compared the efficacy and safety of drugs versus placebo, standard of care, or other covid-19 drugs.
Data sources, search strategy, and study selection
The PubMed online bibliographic database was searched from 11 to 17 February 2022 for articles published between 1 December 2019 and 17 February 2022. Online supplemental table 2 shows the search strategy terms. For some drugs, not many clinical trials were published at that time, so the search was performed with keywords (eg, “sotrovimab”) to avoid being too specific and potentially missing relevant studies. Based on the search strategy terms, titles, abstracts, and full texts of articles were identified. For selection of article titles, we searched for the name of the drug and covid-19. For selection of abstracts, we searched in the methods sections for the study design (ie, clinical trial). For the selection of full texts, we searched for the name of the drug and covid-19, and the study design, if this information was not reported in the title or abstract.
The eligibility of the articles was evaluated independently by two reviewers (KM and CF). Articles were selected by screening the title, abstract, and then the full text. Only articles published in French and English were selected. Disagreements between reviewers were resolved by FM. The information extracted and recorded from each of the articles reviewed by the authors (KM and CF) were number of clinical trials for each drug, registration number of the trial, and design of the trial (randomised open label single, double, or triple blind, or no allocation open label) (online supplemental table 3).
Measurement of quality of reporting of adverse events
To assess the quality of reporting of adverse events, we used the CONSORT (Consolidated Standards of Reporting Trials) harms extension statement.5 The CONSORT harms extension statement was developed to provide investigators with good practice guidelines for recording adverse events in randomised clinical trials, but some authors have used these recommendations to assess the quality of reporting of adverse events.4 6 The recommendations of the CONSORT harms extension were subdivided into 19 subcategories because several items could be grouped under one item. All articles on randomised clinical trials of covid-19 drugs included in our study were analysed independently by two reviewers (KM and CF). We developed a data extraction sheet on Excel with the different clinical trials on each drug and the different sub-items of the CONSORT harms extension. Cohen's κ score was calculated to assess the degree of agreement between the reviewers. Disagreements were resolved by discussion between the two authors. For clinical trials evaluating two drugs, we chose to evaluate the drug used in the intervention group and not the drug used in the placebo group. Analysis of adherence of these clinical trials to the CONSORT harms extension was performed based on the article, extended protocol, online supplemental data, and statistical analysis plan of each clinical trial.
For each item, a score of one was assigned if the article followed the recommendation exactly and a score of zero if it did not. For each article analysed, the total score was calculated and a ranking was assigned, as previously described in the literature.4 7 The quality of reporting of adverse events was classified as high (score 15-19), moderate (score 10-14), low (score 5-9), or very low (score 0-4). The quality of reporting of adverse events was also described according to drug class (monoclonal antibodies, antiviral agents, and immunomodulatory drugs). We also calculated the proportion of studies with high, moderate, low, and very low quality of reporting of adverse events according to the design of the clinical trial. Because our sample size was small, the aim of our study was not to generalise the conclusions but to highlight the inadequacies and inconsistencies of reporting adverse events in studies that were published during the peak period of the covid-19 pandemic. Also, we did not calculate confidence intervals because of the small sample of clinical trials.
Secondary objectives were comparisons between the number of serious adverse events reported in published journal articles listed in PubMed and those reported in clinical trial summaries on the ClinicalTrials.gov website. For each clinical trial summary and journal article, we compared the number of serious adverse events and number of patients with serious adverse events in the treated group. The analysis period for ClinicalTrials.gov data was modified from the original protocol and extended by one year. The extra year gave us more time to evaluate the data.
To assess the risk of bias for each study, we used the Cochrane risk of bias tool for randomised trials (RoB 2), which covers sequence generation, allocation concealment, blinding, incomplete outcome data (eg, dropouts and withdrawals), and selective outcome reporting.8 For each domain in the tool, we described the procedures undertaken for each study, including verbatim quotes. A judgment on the possible risk of bias on each of the six domains was made from the extracted information, rated as high risk, low risk, or some concerns. We then compared the type of bias with the quality score for reporting of adverse events for each included study.
Patient and public involvement
We did not involve patients or the public in designing, conducting, or reporting our research, because we used only data from previously published studies. Systematic reviews identify and analyse relevant primary studies to answer a specific research question, but they are not conducted on patients or the public directly. We plan to disseminate our results through open access publication and social media.
Assessment of quality of adverse event reporting
Based on the terms listed in the search strategy (online supplemental table 2), we identified 1962 randomised clinical trials assessing the efficacy and safety of drugs used to treat covid-19, published in the PubMed database. We excluded 1906 studies (seven duplicates, 1814 after screening the title, 65 after screening the abstract, and 20 after screening the full text article; figure 1 and online supplemental table 7). The remaining 56 clinical trials were included in our assessment: 15 clinical trials assessed monoclonal antibodies (two for sotrovimab, five for bamlanivimab, three for bamlanivimab-etesevimab, and five for casirivimab-imdevimab), 17 assessed antiviral agents (11 for remdesivir, five for molnupiravir, and one for nirmatrelvir-ritonavir), and 24 assessed immunomodulatory drugs (14 for tocilizumab, five for sarilumab, two for ruxolitinib, one for tofacitinib, and two for baricitinib).9–57 Table 1, online supplemental table 4, and online supplemental figure 1 show the adequacy of the included clinical trials of covid-19 drugs, based on fulfilling each of the recommendations of the CONSORT harms extension. Cohen's κ for agreement between the two authors for the CONSORT score was 0.92 (95% confidence interval 0.84 to 1.00).
We found that 95% of clinical trials did not provide information on adverse events in the introduction section (CONSORT recommendation 2), 82% did not provide both the number of adverse events and number of patients with adverse events (CONSORT recommendation 8c), 95% did not describe subgroup analysis and exploratory analysis for harms (CONSORT recommendation 9), 91% did not provide a balanced discussion on efficacy and adverse events (CONSORT recommendation 10a), and 97% did not provide the limitations of the study, specifically in relation to adverse events discussed (CONSORT recommendation 10b) (table 1). No clinical trial had a high15–19 score for reporting of adverse events, 60.7% had a moderate10–14 score, 33.9% had a low score, and 5.4% had a very low score (figure 2). Based on our evaluation by drug class, we found that 86.7% of studies that assessed monoclonal antibodies had a moderate score and 13.3% had a low score for quality of reporting of adverse events. For studies that assessed antiviral agents, 64.7% had a moderate score, 29.4% had a low score, and 5.9% had a very low score. For studies that assessed immunomodulators, 50% had a low score, 41.7% had a moderate score, and 8.3% had a very low score (figure 2).
Quality of reporting according to design of clinical trial
We next compared the quality of reporting of adverse events according to the design of the clinical trial to see which design was associated with a high, moderate, low, or very low score. We found that in randomised open label trials, 17.6%, 47.1%, and 35.3% of trials had a very low, low, and moderate quality score, respectively. In randomised single blind trials, 100% of trials had a low quality score. In randomised double blind trials, 19% had a low and 81% had a moderate quality score. In randomised triple blind trials, 50% had a low and 50% had a moderate quality score. In randomised quadruple blind trials, 22.2% had a low and 77.8% had a moderate quality score. In no allocation open label trials, 50% had a low and 50% a had moderate quality score (online supplemental figure 2).
Reporting of serious adverse events in journal articles versus trial summaries
Among the 56 journal articles identified in the review, only 20 trial summaries were available in ClinicalTrials.gov one year after the research period (figure 3). The 20 trial summaries were for eight different drugs: sotrovimab, one journal article and one trial summary; bamlanivimab, three journal articles and trial summaries; remdesivir seven journal articles and trial summaries; molnupiravir two journal articles and trial summaries; nirmatrelvir one journal article and trial summary; tocilizumab, four journal articles and trial summaries; sarilumab one journal article and trial summary; and baricitinib one journal article and trial summary. We found that journal articles under-reported 51% of serious adverse events compared with trial summaries. The differences varied according to the drugs evaluated: 100% for sotrovimab, 41% for bamlanivimab, 62% for remdesivir, 96% for nirmatrelvir, 42% for toclizumab, and 92% for baricitinib. For molnupiravir and sarilumab, however, we found the same number of serious adverse events in the journal articles and trial summaries (table 2). The most frequent serious adverse events under-reported were cardiac disorders, metabolic and nutritional disorders, and respiratory, thoracic, and mediastinal disorders, according to the Medical Dictionary for Regulatory Activities, version 23.0.
Assessment of bias
We assessed the level of bias in each study (online supplemental table 5). Of the 56 studies included in the analysis, 92.9%, 5.4%, and 1.8% had a high, moderate, and low level of bias, respectively. We then compared the level of bias with the quality score for reporting of adverse events for each included study. For studies with a high level of bias, 5.8%, 36.5%, and 57.7% had a very low, low, and moderate quality score, respectively. For studies with a moderate or low level of bias, 100% had a moderate quality score (online supplemental figure 3).
Based on our evaluation of the total harm reporting score, we found that no study had a high score for the quality of reporting of adverse events, most studies had a moderate quality score, and one in three studies had a low or very low score for quality of reporting. We also found that clinical trials with very low scores were those that investigated antiviral agents and immunomodulators (remdesivir, tocilizumab, and sarilumab). Few of the included studies reported information on adverse events in the introduction (item 2 of the CONSORT harms extension), possibly because in general, authors focus mainly on the efficacy of drugs. Also, we found that journal articles under-reported the number of serious adverse events compared with trial summaries of clinical trials of covid-19 drugs (table 2). Information on serious adverse events was lacking in published articles; only two drugs (molnupiravir and sarilumab) had the same number of serious adverse events reported in journal articles and clinical summaries. We found that well designed and mostly double blind randomised controlled trials provided results with a moderate score for the quality of reporting of adverse events compared with other study designs. All clinical trials with a low or very low score for quality of reporting of adverse events had a high level of bias.
Comparison with other studies
Our findings provide new information and support other recent analyses. A preliminary study showed that for clinical trials of remdesivir, the quality of adverse event reporting was low.4 Another study showed that none of the clinical trials that evaluated the effectiveness of hydroxychloroquine or chloroquine for the treatment of covid-19 met the CONSORT criteria in full for reporting harm data.58 Randomised clinical trials on the covid-19 vaccine, however, were reported to be less biased with good quality on reporting harm based on the modified CONSORT harms extension.7
The low quality of adverse event reporting in clinical trials is not limited to covid-19 drugs. Of 1608 serious adverse events in participants treated with six antidepressants reported in trial summaries, 694 (43.2%) did not appear in the associated articles.59 This finding was highlighted in another study that found that reporting was substantially more complete in summaries on the ClinicalTrials.gov website than in published articles for serious adverse events (99% v 63%).60 These safety data are the main scientific sources for researchers and clinicians, and our results indicate that relying on journal articles for information on covid-19 drugs might miss important information on adverse events.
Several studies in our evaluation had moderate-to-low quality reporting of adverse events. This finding might be explained in the context of the pandemic where researchers favoured the evaluation of effectiveness over adverse events. Even in the absence of an epidemic, however, risk assessment of drugs has been reported to be low, possibly because different pharmaceutical companies and countries have different requirements for reporting of adverse events in clinical trials.61–63 Under-reporting of adverse events, however, could be harmful for drug safety and hence for patients. Journals might limit the number of words in titles or abstracts which could lead to a greater focus on effectiveness. The number of adverse events reported in clinical trials is often small because of the low power of clinical trials to detect adverse events, so the recording of adverse events, when they occur, is important for drug safety. Moreover, we found that in many protocols and articles, the CONSORT statement was not mentioned, possibly suggesting unfamiliarity with these guidelines.
Our findings suggest the need to consult several sources of information to correctly establish the safety profile of these drugs from clinical trials. Often published articles only report serious adverse events that occur in >5% of patients. During the covid-19 pandemic, these clinical trials were primarily and rapidly designed to evaluate efficacy, possibly explaining why some clinical trials had a low or very low quality of adverse event reporting. Based on the results of our systematic review comparing the reporting of harms of several covid-19 drugs, and results from previous studies,64 adverse event reporting seemed to be more complete in clinical trial summaries than in journal articles.
Implementing CONSORT recommendations is necessary to correctly measure and report the effectiveness and safety of the intervention.65 Stroehlein et al found insufficient evidence to determine the benefits and harms of vitamin D supplementation as a treatment for covid-19 because of limited safety information, and were concerned about consistency in measuring and recording these outcomes.66 The authors highlighted an urgent need for well designed and adequately powered randomised clinical trials with an appropriate randomisation procedure, comparability of study arms, and preferably double blinding. These findings are in agreement with our results because we showed that the clinical trials that had a very low quality of adverse event reporting were randomised open label trials, and those with a low quality of adverse event reporting were randomised single blind or no allocation open label trials.
The purpose of clinical trials is to evaluate the effectiveness and safety of new drugs. Systematic errors, however, can be made, leading to a variation in results that can result in overestimation or underestimation of the true effect of an intervention. In a study of osteosarcoma and Ewing's sarcoma, the prevalence of a low risk bias was 47.3%, unclear risk domains was 47.8%, and 4.9% of domains had a high risk of bias in randomised clinical trials .64 Domains with the highest risk of bias were blinding of participants or staff, and outcome assessors, followed by randomisation and allocation concealment. In our study, risk of bias was mainly high in 92.9% of the included clinical trials. All clinical trials with low and very low reporting quality were studies with a high risk of bias.
Limitations of this study
Our study had some limitations. In this systematic review, our focus was on published trials and unpublished data were excluded. We also examined only one clinical trials registry (ClinicalTrials.gov).
To deal with the problem of low quality of reporting of adverse events and missing data for covid-19 drugs, editors should impose stricter requirements for submitting journal articles. The recommendations of the CONSORT harms extension should be applied in any submission of results of clinical trials related to drug safety. Journal articles are more accessible and available earlier than clinical trial summaries and therefore CONSORT harms recommendations should be used to improve the quality of reporting of adverse events in randomised controlled trials.67
In one in three clinical trials that assessed the efficacy and safety of covid-19 drugs, we found that the quality of reporting of safety data was low or very low. Also, differences were found in the number of serious adverse events related to covid-19 drugs between trial summaries (ClinicalTrials.gov) and journal articles. Both sources were limited by incomplete reporting. Authors and editors should pay more attention to methods, by better reporting of safety data for covid-19 drugs in clinical trials. Physicians and health professionals should consider published trial reports alongside summaries in clinical trial registries for a more complete understanding of the adverse events of drugs in clinical trials.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
No ethical approval needed.
We thank Marie-Elise Llau for her help and availability in the practical implementation of KM's research project.
Contributors KM, PO, and FM designed the study. KM, CF, and FM screened citations for inclusion and were involved in data extraction and interpretation. KM and CF were involved is the risk of bias assessments. All authors were involved in the interpretation of the data. FM wrote the draft manuscript. KM and FM revised the new version of the manuscript. All authors approved the final and revised versions of the manuscript. FM is the guarantor of this manuscript and accepts full responsibility for the work and the conduct of the study, had access to the data, and controlled the decision to publish. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. Transparency: The lead author (the guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Funding This project was funded by the French National Research Agency (ANR Agence Nationale de la Recherche). FM has received funding under the Vigi-Drugs COVID-19 (No ANR-21-CO17-0001) project from the French National Research Agency for the evaluation of safety data of drugs and vaccines used in the management or prevention of covid-19 (https://anr.fr/Project-ANR-21-CO17-0001). The funder had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
Competing interests All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from the French National Research Agency (ANR Agence Nationale de la Recherche) for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.