Article Text

Availability of results of clinical trials registered on EU Clinical Trials Register: cross sectional audit study
  1. Nicholas J DeVito1,
  2. Jessica Morley1,
  3. James Andrew Smith2,
  4. Henry Drysdale1,
  5. Ben Goldacre1 and
  6. Carl Heneghan1
  1. 1 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  2. 2 Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford Medical Sciences Division, Oxford, UK
  1. Correspondence to Dr Nicholas J DeVito, Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford OX2 6GG, UK; nicholas.devito{at}phc.ox.ac.uk

Abstract

Objective To identify the availability of results for trials registered on the European Union Clinical Trials Register (EUCTR) compared with other dissemination routes to understand its value as a results repository.

Design Cross sectional audit study.

Setting EUCTR protocols and results sections, data extracted 1-3 December 2020.

Population Random sample of 500 trials registered on EUCTR with a completion date of more than two years from the beginning of searches (ie, 1 December 2018).

Main outcome measures Proportion of trials with results across the examined dissemination routes (EUCTR, ClinicalTrials.gov, ISRCTN registry, and journal publications), and for each dissemination route individually. Prespecified secondary outcomes were number and proportion of unique results, and the timing of results, for each dissemination route.

Results In the sample of 500 trials, availability of results on EUCTR (53.2%, 95% confidence interval 48.8% to 57.6%) was similar to the peer reviewed literature (58.6%, 54.3% to 62.9%) and exceeded the proportion of results available on other registries with matched records. Among the 383 trials with any results, 55 (14.4%, 10.9% to 17.9%) were only available on EUCTR. Also, after the launch of the EUCTR results database, median time to results was fastest on EUCTR (1142 days, 95% confidence interval 812 to 1492), comparable with journal publications (1226 days, 1074 to 1551), and exceeding ClinicalTrials.gov (3321 days, 1653 to undefined). For 117 trials (23.4%, 19.7% to 27.1%), however, results were published elsewhere but not submitted to the EUCTR registry, and no results were located in any dissemination route for 117 trials (23.4%, 19.7% to 27.1).

Conclusions EUCTR should be considered in results searches for systematic reviews and can help researchers and the public to access the results of clinical trials, unavailable elsewhere, in a timely way. Reporting requirements, such as the EU's, can help in avoiding research waste by ensuring results are reported. The registry's true value, however, is unrealised because of inadequate compliance with EU guidelines, and problems with data quality that complicate the routine use of the registry. As the EU transitions to a new registry, continuing to emphasise the importance of EUCTR and the provision of timely and complete data is critical. For the future, EUCTR will still hold important information from the past two decades of clinical research in Europe. With increased efforts from sponsors and regulators, the registry can continue to grow as a source of results of clinical trials, many of which might be unavailable from other dissemination routes.

  • Health policy
  • Clinical trial
  • Clinical governance
  • Ethics, medical

Data availability statement

Data are available in a public, open access repository. All data and code for this analysis are available on Github (https://github.com/ebmdatalab/euctr_pub_search) or the project’s Open Science Framework repository (https://osf.io/r3vc5/).

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

  • Clinical trial registries should be a key tool for discovering the results of clinical trials

  • Trial reporting to the European Union Clinical Trials Register (EUCTR) is required by European regulations, and compliance has increased substantially over time

  • Problems with data quality, however, obscure the true value of the registry as a source of clinical trial information

WHAT THIS STUDY ADDS

  • This study examined whether EUCTR offers value to researchers and the public as a repository of information on clinical trials

  • The findings suggested that EUCTR has trials not registered elsewhere, results often appeared first on EUCTR and, at times, were the only results publication for an appreciable number of trials

  • Literature searches for evidence synthesis should strongly consider direct use of EUCTR

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE, OR POLICY

  • Efforts to inform clinical practice are based on a complete view of the evidence, and evidence synthesis, development of clinical guidelines, and clinical practice are compromised when results of clinical trials are withheld

  • By searching EUCTR, interested parties can potentially gain a more complete and systematic view of the latest information on a specific intervention to inform clinical decision making.

Introduction

Clinical trial registries provide transparency into the planning, conduct, and reporting of clinical trials.1 Between 2004 and 2023, clinical trials of investigational medical products (ie, most drugs, biologics, and vaccines), initiated under the EU clinical trial directive, were required to register in the EudraCT (European Union Drug Regulating Authorities Clinical Trials) database.2 In 2011, the European Union Clinical Trials Register (EUCTR) launched providing a public facing, searchable registry that covered most trials in the EudraCT database3. As of February 2023, the EUCTR contains public records for more than 43 000 clinical trials.4

In 2012, EU guidelines were issued requiring trial sponsors of all clinical trials of investigational medical products covered under the clinical trial directive to report results within a year of completion of the trial, expanding on previous requirements to report the results of some paediatric trials within six months of completion.5 6 Results were to be uploaded to the EudraCT system for all trials and then made public on EUCTR for all except some exempted phase 1 trials. This rule applied not only to newly completed trials but also retrospectively to completed trials dating back to the start of the registry.5 After some technical delays following the launch of the EUCTR results section in 2014, these requirements came into full effect in late 2016. Results can be added either in a tabular summary format or as a document upload (eg, a synopsis or journal article) for older trials.

Given longstanding concerns around research waste and publication bias,7 these EU guidelines had substantial potential to increase the transparency of clinical research in Europe. Although no formal enforcement mechanisms were in place, these guidelines preceded more comprehensive reporting requirements as part of the EU Clinical Trial Regulation 536/2014; these new regulations for clinical trials of investigational medical products are being phased into full effect with the launch of a new EU registry, the Clinical Trial Information System (CTIS).8 EUCTR, however, should remain a major archival source of information on the past two decades of European trials of regulated medicines.

Despite its size and growing amount of results,9–13 EUCTR has not been widely studied and is inconsistently used in research.14–23 As of February 2023, the EU TrialsTracker project, which audits compliance with the 2012 guidelines, has identified 19 614 trials due to report, and 16 488 (84.1%) with results; thousands of more trials have also added results to the registry, or should have, but problems with data quality obscure their due dates for automated compliance checks.24 25 Previous investigations have shown that the US ClinicalTrials.gov registry is often the only source of public trial results,26 27 making it a valuable resource for evidence dissemination and synthesis. No similar estimates for EUCTR exist, however, nor the extent of its overlap with the published literature or other registries.28 Therefore, in this study, our aim was to quantify the availability of results of trials on EUCTR compared with other registries and the peer reviewed literature, to generate evidence on its value as a repository for results for the medical community.

Methods

This project was preregistered on the Open Science Framework (https://osf.io/drpc5).29 More details on the methods are available in the protocol and on the Open Science Framework. All data collection, preparation, and analyses were conducted in Python 3 (Python Software Foundation) with data and code publicly available from GitHub30; manual data extraction was conducted in Google Forms.

Data sources

Data from all available EUCTR protocols (ie, details of the trial in each country) and results sections were extracted with custom web scraping software between 1 December 2020 and 3 December 2020.31 32 A trial record on the EUCTR is made up of a protocol for every EU country in which the trial took place, a protocol for non-EU locations of some paediatric trials, and, if available, a results section covering the whole trial. Key fields for this study from EUCTR included trial status (eg, ongoing, completed), date of ethics committee opinion and date of competent authority decision (ie, as proxies for the start of the trial), date of the global end of the trial, and the initial estimate of the duration of the trial from section E.8.9 of the protocol information on EUCTR (online supplemental figure S1).

Supplemental material

Study population and sample

Trials removed from the full December 2020 dataset were those with a status of not authorised or prohibited by competent authority, and those with ethical approval dates from before the launch of the registry or later than the data extraction date. The remaining trials were checked for the latest completion date in the individual trial protocols or in the results section. Completion dates from the results section were preferred when available.

Poor data on completion of a trial are a known concern for EUCTR because many trials lack updated statuses and completion dates.33–35 To ensure that these trials could be captured within our population, and to avoid selection bias for trials with better record keeping, we developed a method to infer a trial completion date when this information was missing. Each trial protocol in the dataset without a completion date was assigned a start date (ie, from the ethics and regulatory start dates provided, whichever was later, because no clear start date field exists on EUCTR) and an expected duration in days calculated from protocol section E.8.9 (online supplemental figure S1).

All protocols from a trial record were then grouped into one record, where the longest estimated duration was added to the latest start date. Another year was conservatively added to this date, to allow for any delays in the start or conduct of the trial, resulting in a final inferred completion date. A Jupyter Notebook detailing this approach and its validation is available on the project's Open Science Framework repository (https://osf.io/r3vc5/).

For trials with an extracted or inferred completion date, we limited our population to those completed at least two years in the past (ie, before 1 December 2018) to allow time for reporting across all dissemination routes.36 37 From this population, a random sample of 500 trials was taken for the analysis by using the .sample() method in the Pandas Python package. This sample size was chosen based on achieving point estimates with a maximum 95% confidence interval of ±5% (online supplemental box S1). The maximum sample needed to achieve this level of precision was 384; a final sample of 500 trials was chosen to allow for greater precision in point estimates of subpopulations. Trials in the sample found, at any point, to have issues that would make it difficult or impossible to discover the results, such as being withdrawn without enrolment, still ongoing, or currently inaccessible on EUCTR, were excluded from the analysis sample and replaced with another randomly chosen trial.

Results search strategy

The search strategy was piloted in 50 trials, with a subset searched in duplicate. We found high percentage agreement on data extraction, and discrepancies were easily resolved through discussion. The pilot informed the decision to only use open, public databases (ie, PubMed, Google Scholar) to allow for greater reproducibility because proprietary databases (ie, Scopus, Ovid) did not give substantially more value in locating results. Details from the pilot searches are available in the protocol and on the project's Open Science Framework repository (https://osf.io/r3vc5/).

Each EUCTR trial record was reviewed for results and information about duplicate registrations or published results. ClinicalTrials.gov and the ISRCTN registry were then searched for potential cross registrations. Both of these registries can host results directly on the registry, and because of their size, geographical focus,38 39 and related regulations,40 41 would be expected to have cross registrations of trials on the EUCTR. After peer review feedback, any remaining trials with no additional registrations located were searched in the International Clinical Trials Registry Platform (ICTRP) database to confirm that no documented cross registrations existed. Registries were first searched with the EUCTR unique trial identifier and then with the trial title, name/acronym, intervention, condition, sponsor, and any additional secondary trial identity numbers. Matching records were searched for additional relevant information; an eligible result on a registry was hosted directly on the registry rather than linking to an external article or resource. Trials that declared, in place of results, that no analysis was possible (eg, because of low enrolment) were counted as having results because enough detail to understand the fate of the trial would be available to interested parties.

Lastly, PubMed and Google Scholar were searched for journal publications with all known trial identity numbers, trial title, acronym, interventions, conditions studied, and any investigator names and affiliations available in the registrations. Searchers could combine these terms, or add more terms, to their searches at their discretion. Results in the literature were included if they were available as a publication in a journal, reported final primary results of the trial, and were >500 words in length (eg, detailed conference abstracts), consistent with previous methods.37 If multiple eligible publications were located, we recorded the earliest. Only results published before the start of searches were included in the final analysis. Article and registry matches were confirmed through comparison of trial identity numbers, study design, indication, intervention, planned enrolment, and registered outcomes.36 No specific threshold was set to define a match between records, but any problems matching registrations and publications were referred to the full study team for more discussion.

One author (ND) searched all trials and 50% were also searched by a second author (JAS, JM, HD) to validate the search strategy and data extraction. The original sample of 500 trials was searched by the lead researcher (ND) between December 2020 and July 2021; secondary searches of half the sample began in April 2021 and concluded in January 2023. Further checks and searches were carried out in 2023 in response to peer review. Any uncertainties or discrepancies were resolved by consensus discussions, with remaining concerns referred to a senior member of the study team (CH) for final adjudication. Problems with extracting publication date because of inconsistencies between sources (ie, PubMed, journal websites) were resolved by the lead author by re-extracting all publication dates, preferring the date on the journal website when available. Online supplemental table S1 details the reliability measures between the searchers.

Outcomes and statistical analysis

The primary outcome was the proportion of trials with results for the examined dissemination routes, and for each route individually. Prespecified secondary outcomes were the number and proportion of unique results, and the timing of results, for each dissemination route. Analysis of the timing of results was altered from prespecification (online supplemental box S2). Differences in reporting statistics between the groups of trials with inferred and extracted completion dates were assessed with a two proportion z test (unadjusted α=0.05, with Holm-Bonferroni corrections). The presence of trial identity numbers in a journal publication was added as a post hoc secondary outcome. This information was extracted during data collection for validation purposes and provides insights into how the published literature linked back to the EUCTR registration.

Exploratory analyses

We performed two prespecified exploratory analyses. For trials with results available in any dissemination route, factors associated with results appearing on EUCTR were examined with univariable and multivariable logistic regression. Prespecified factors were whether the completion date was reported or inferred, trial start year, sponsor type, number of EU protocols registered, final enrolment (intent to treat), and whether the trial was conducted only in the EU or European Economic Area (EEA), in countries inside and outside of the EU and EEA, or entirely outside of the EU and EEA. Trial start year, final enrolment, and location variable were extracted manually by the lead author (ND) from trial registrations and the results located during searches. If contradictions between sources arose, the most recently updated or available source was preferred; if further ambiguity existed, the EUCTR data were preferred. The remaining variables were directly extracted from EUCTR data in code, with missing data coded as unknown. Significance for the univariable and multivariable models was determined with the Holm-Bonferroni method (unadjusted α=0.05).

The second exploratory analysis examined variation in reporting behaviour by sponsor country. The sponsor country field (ie, protocol section B.1.3.4 on EUCTR) was extracted for all country level protocols for each trial, and each trial was assigned the most frequently appearing sponsor country across protocols. In the event of ties, the trial was coded as having multi-country sponsorship. If no sponsor was located, the sponsor country was coded as unknown. The number of trials reporting any results, and reporting results to EUCTR, other registries, and the literature was examined for each sponsor country.

Patient and public involvement

No patients or members of the public were involved in determining the research question, outcome measures, or interpreting the results as this was a doctoral student project without funding to support patient and public involvement. The results of this study will be summarised for the public in a blog post by the first authors on publication, disseminated on the Bennett Institutes for Applied Data Science and TranspariMED websites to their relevant audiences, and publicised on social media.

Results

Study population and sample

As of 1 December 2020, 98 622 individual country protocols were registered to EUCTR for 38 566 trials. After exclusions, the final overall population included 66 833 individual protocols for 27 241 trials. During searches, we found evidence indicating that 22 trials in our sample were withdrawn without enrolment (n=16), were still ongoing (n=3), or were inaccessible on EUCTR (n=2), and were replaced according to our protocol. One other trial seemed to have mistakenly been registered on EUCTR because it was an observational study and the sponsor noted that they did not plan to upload results (2013-001141-14); this trial was also replaced because results could not appear on EUCTR. Table 1 shows the characteristics of the final sample of 500 trials from this population.

Table 1

Characteristics of final sample of 500 trials

For the whole of EUCTR, we extracted an end date for 21 766 trials (56.4%) and inferred an end date for 16 051 (41.6%) trials; 749 (1.9%) trials had insufficient information about completion or had not received authorisation. After excluding trials with completion dates of less than two years in the past (n=10 576), 27 241 trials formed the population to be sampled: 19 182 (70.4%) trials had extracted end dates and 8059 (29.6%) had inferred end dates. Our final random sample of 500 trials contained 354 (70.8%) trials with extracted completion dates and 146 (29.2%) with inferred completion dates (figure 1). EUCTR was the only registration available for 138 trials (27.6%, 95% confidence interval 23.7% to 31.5%) (online supplemental figure S2). For trials with known extracted completion dates, trials were completed a median of 3040 days (interquartile range 1884-4137; range 745-5755), or 8.3 years, before the start of the searches; trials with inferred completion dates were expected to have been completed for a median of 2804 days (interquartile range 1788-3513; range 791-5937), or 7.7 years, before the start of the searches (online supplemental figure S3). Moods median test showed no difference in median follow-up between these two samples (P=0.14).

Figure 1

Flowchart of selection of the sample of included trials. EUCTR=European Union Clinical Trials Register

Trial registration and results reporting

Table 2 shows the availability of results for each of the examined dissemination routes for the 500 trials in our sample. A total of 694 results were located for 383 (76.6%, 95% confidence interval 72.9% to 80.3%) trials across all dissemination routes; EUCTR had results for 266 (53.2%, 48.8% to 57.6%) registered trials. Of the 383 trials with results, 55 (14.4%. 10.9% to 17.9%) were only available on EUCTR. Results on EUCTR were most commonly provided only in the registry’s tabular format (n=117, 43.9%); for unique results, the most common format was clinical study report synopsis documents (n=30, 54.6%) (online supplemental table S2). The most common dissemination route overall was publishing only in a peer reviewed journal (n=108); 117 (23.4%, 19.7% to 27.1%) trials had results elsewhere but not on EUCTR, and 117 (23.4%, 19.7% to 27.1%) had no results disseminated anywhere, both indicating a failure to report under EU guidelines.

Table 2

Availability of results overall and by dissemination route

For the 293 journal articles located, six (2%) were conference abstracts that met the length criteria for inclusion (ie, >500 words). Another 24 abstracts were located that seemed to be matched to EUCTR registrations but did not meet the length criteria for inclusion; seven of these would have represented the only results available if they had been included. Also, of the 266 trials with results on EUCTR, five (1.8%) were included that provided a statement on EUCTR declaring that no analysis was possible because of low enrolment in place of results. Based on enrolment values extracted for the exploratory analyses, the 117 unreported trials enrolled, or planned to enrol, 33 673 participants. Figure 2 shows an UpSet plot visualising all combinations of availability of results for all of the dissemination routes.42

Figure 2

Availability of results for the four dissemination routes. EUCTR=European Union Clinical Trials Register. Dark dots represent the combination of dissemination routes measured in the bar chart above them with the lighter dots excluded from that set. For instance, the first column of dots represents trials with no results in any route, and the last column represents trials with results on all routes examined

Trials with a known completion date were significantly more likely to have results available in any dissemination route compared with those with an inferred end date (88.1% v 48.6%, P<0.001) and among all individual dissemination routes except ISRCTN (table 2). Trial results outside EUCTR were also more likely for trials with extracted versus inferred end dates (72.9% v 48.0%, P<0.001). Only one trial with an inferred end date had results available on EUCTR but this finding was not considered unusual because the methods used to extract completion dates often relied on the availability of a results section. When results were grouped by the status of the sponsor (ie, commercial v non-commercial), as requested during peer review, commercial sponsors consistently showed higher rates of reporting and higher rates of known completion dates, indicating better management of registry data on EUCTR (online supplemental table S3).

Timing of results reporting

For 291 trials that could have appeared in ClinicalTrials.gov, EUCTR, or a journal article, results most commonly first appeared in a journal article (n=156, 53.6%). The results section of ClinicalTrials.gov did not launch until October 2008, however, and the results section of EUCTR did not launch until March 2014. When accounting for these differences, first availability of trial results in this sample changed from strongly favouring journal articles (n=64, 73.6%) over ClinicalTrials.gov (n=23, 26.3%) before the launch of the results section of EUCTR, to EUCTR (n=84, 43.5%) having relative parity with journal articles (n=89, 46.1%) as the earliest dissemination route compared with ClinicalTrials.gov (n=20, 10.4%) (online supplemental table S4).

Figure 3 shows cumulative incidence curves examining time from extracted completion date to reporting for each dissemination route for trials with a first result after the launch of EUCTR. Median time to a result was similar for EUCTR (1142 days, 95% confidence interval 812 to 1492) and a journal publication (1226 days, 1074 to 1551) and much faster than the median time to results on ClinicalTrials.gov for trials that could appear there (3321 days, 1653 to undefined). EUCTR also continued to add new results long after trial completion whereas both journal publication and ClincialTrials.gov results began to plateau at about 2000 days from completion with few new results added. Online supplemental figure 4 shows the distribution of start years for the trials with no results in any dissemination route and no results on EUCTR, with the proportion of missing results remaining relatively consistent over time.

Figure 3

Time to reporting from extracted completion date, after the launch of the results section of European Union Clinical Trials Register (EUCTR)

Reporting of trial identity numbers

Online supplemental table 5 shows the proportion of journal articles, matched to a trial registration, that contained the trial identity number of that registry. Only 22.5% (95% confidence interval 17.7% to 27.3%) of journal articles had a linked EUCTR identity number, substantially less than identity numbers for ClinicalTrials.gov (83.3%, 78.4% to 88.2%) or ISRCTN (62.5%, 43.1% to 81.9%).

Exploratory analyses

In the exploratory risk factor analysis, we excluded whether the trial had an inferred or extracted end date as a potential risk factor, despite prespecification, because this factor was nearly a perfect predictor of non-reporting of results to EUCTR (table 2). In the univariable models, being a commercial sponsor (odds ratio 14.40, 95% confidence interval 8.47 to 24.49), having more EU protocols (1.57, 1.32 to 1.87), and with sites inside and outside of the EEA (reference EEA only: 6.13, 3.50 to 10.73) increased the likelihood of results appearing on EUCTR. In the fully adjusted multivariable analysis, only commercial sponsorship remained significant (adjusted odds ratio 9.75, 95% confidence interval 5.04 to 18.85, P<0.001). Online supplemental table S6 shows all of the results of the model.

The second exploratory analysis examined dissemination by sponsor country. Online supplemental table S7 includes the reporting practice of all sponsor countries identified. We found substantial heterogeneity in the use of different dissemination routes for results. In half of the 12 countries with >10 sponsored trials, dissemination rates were highest to EUCTR. The two lowest reporting countries to EUCTR, Italy (17.4%) and Spain (13.9%), also had the lowest availability of results for all dissemination routes among the most active sponsoring countries (Italy 54.4%, Spain 50.0%).

Discussion

Summary of results

In our sample of 500 trials, availability of results on EUCTR (53.2%, 95% confidence interval 48.8% to 57.6%) was similar to the peer reviewed literature (58.6%, 54.3% to 62.9%) and exceeded the proportion of results available on other registries with matched records. Among the 383 trials with results, 55 (14.4%, 10.9% to 17.9%) were only available on EUCTR. Also, after the launch of the results section of the EUCTR database, median time to results was fastest on EUCTR (1142 days, 95% confidence interval 812 to 1492), comparable with the literature (1226 days, 1074 to 1551) and exceeding ClinicalTrials.gov (3321 days, 1653 to undefined). For 117 trials (23.4%, 19.7% to 27.1%), however, results were disseminated elsewhere but were not submitted to EUCTR, and no results were reported in any dissemination route for 117 trials (23.4%, 19.7% to 27.1%).

Strengths and limitations

This analysis provides a comprehensive assessment of the fate of the results of trials registered on EUCTR. We searched the literature as well as other large, common registries for EU registered trials, which improved our ability to locate linked results and compare dissemination across various routes. We also saw high agreement from the search strategy between searchers.

The extent of missing completion information on EUCTR, detailed in previous research, influenced the design of this study.24 35 The choice to include trials with both explicit and inferred completion dates has strengths and weaknesses. Although this method might introduce some error in the analysis, mainly by including ongoing or withdrawn trials with incorrect registry data that cannot be identified, it also provides a comprehensive real world view of reporting by avoiding selection bias for only those trials with the best managed data. Overall, this method seemed successful because the existence of most trials (67%) with inferred dates were confirmed by the availability of results or other registrations that also did not list the trial as withdrawn. Reporting disaggregated results for these two subpopulations gives further context to the findings.

Ideally, whether a trial never occurred or was ongoing would be clear from the trial registration. If the comprehensive search strategy used in this analysis could not identify the status of these trials, a search strategy for a systematic review is also unlikely to do so, unless the authors are contacted directly. Because of time and resource constraints, we did not contact investigators as part of this assessment. Contact information on EUCTR is frequently missing, and therefore would have added a substantial burden to data extraction as well as a lengthy process of managing outreach. Uncertainty about the current status of a trial could lead to wasted effort and time by reviewers in trying to locate expected results that do not exist.43

Results in context

Hwang and colleagues44 assessed whether some paediatric trials registered on EUCTR had results, compared with ClinicalTrials.gov and the published literature. They found a slightly higher rate of overall reporting (85%) with less reliance on the literature for unique results than in our sample.44 Deane and colleagues45–47 examined the reporting of trials supporting drug approvals by the European Medicines Agency, from 2009 to 2013; they found that by including EUCTR registrations, 11% of records were added to the most recent analysis. EUCTR did not include results at the time of this analysis, however, and de-duplicated record counts by registry were not reported.45–47

Speich and colleagues48 examined the reliability of registry information for 360 studies with ethics approval from Switzerland, the UK, Canada, and Germany. These authors found higher rates of availability of results for both ClinicalTrials.gov (57%) and EUCTR (69%) registrations compared with our findings, with none of 20 studies reporting to ISRCTN. Also, 69% of trials registered on both EUCTR and ClinicalTrials.gov with a result on one registry, also had a result on the other (v 55% in our study). The study of Speich and colleagues was more focused in both geography and time compared with our analysis, however, and so the different reporting rates are not surprising.48

Other research focused on the availability of results for registered trials in the EU did not include EUCTR or did not search the published literature. An examination of reporting by top European non-commercial sponsors focused only on EUCTR results,9 and studies of trial reporting at German and Polish academic medical centres did not include EUCTR as a results source.37 49 50 Other similar studies often included populations of trials that overlapped with EUCTR, but explicit search of EUCTR was rare.14–23 Past research by staff of ClinicalTrials.gov showed that in a sample of 380 completed studies with results, 69% had results available on the registry within 48 months, compared with just 40% in the literature, further supporting that registries can disseminate results more efficiently.27 Future research should assess the quality of results on EUCTR compared with the literature. This research would allow greater understanding of the value of hosted results and expand on previous work examining results on ClinicalTrias.gov which consistently showed that some areas, most notably adverse events, were better reported to the registry than to matched journal publications.51–54

Our work also gives important context to audits of reporting practice on EUCTR. EU TrialsTracker is a live audit tool that necessarily uses a conservative definition of when a trial is due to report. The wider population of trials considered for results searches in this study showed that despite considerable progress in reporting tracked by the EU TrialsTracker over time,9 10 problems with data quality affect precise assessments. In December 2020, when the data for this study were extracted, the reporting rate on the EU TrialsTracker was 68%, but in our sample, the reporting rate was just over half of the examined trials (53%). Only one trial with an inferred end date had a result on the registry, in the form of a clinical study report synopsis, despite results existing on other routes for nearly half (48%) of these trials.

Although the EU TrialsTracker provides a valuable public audit and feedback service, and longitudinal data on performance over time, the justifiably conservative methods to minimise false positive “due” trials will overestimate the true level of compliance because of missing and incomplete data. Also, any attempts to manually or automatically link the published literature to a matching EUCTR registration is complicated by the fact that EUCTR trial identity numbers are rarely attached to the resulting publication. Failure to include trial identity numbers in manuscripts and research databases is a persistent problem,55–61 and reporting guidelines and editorial guidance should be amended to clarify that all relevant trial registration numbers, not just one, should be included in publications, abstracts, and metadata.62 63

Lastly, although our exploratory analyses were not designed to provide definitive assessments, the results were consistent with previous work. Commercial sponsorship was the only significant (P<0.001) predictor of availability of results on EUCTR in the adjusted risk factor analysis, which matches the strong associations seen in previous work on reporting under US and EU guidelines.10 64 This finding is likely a result of better resourcing and processes for compliance at pharmaceutical companies. Also, very low reporting to the registry by sponsors in Italy and Spain was similarly shown in an analysis of the reporting of major non-commercial sponsors throughout Europe.9

Implications for policy and practice

Clinical trial registries are an increasingly important source for results, with the potential to help systematic reviews and other evidence searches,51–54 but despite recommendations65 66 they are often underutilised.67–76 The 2012 EU reporting guidelines, which governed reporting of results to EUCTR, operated under a soft requirement approach, where reporting was mandatory but compliance was voluntary, with no sanctions for failing to report.77 This approach led to a well reported discrepancy between commercial and non-commercial reporting compliance.9 10 The most successful effort to improve non-commercial reporting was pressure on public sponsors by the UK Parliament before Brexit.9 78 Other actions from the European Medicines Agency,79 national competent authorities,80 and transparency advocates81 have helped in increasing reporting by non-commercial sponsors to varying degrees. These efforts have substantially increased the number of results available on EUCTR and therefore its value as a research tool and public repository for results. Our findings indicated that EUCTR already has value as a unique repository of results, and therefore efforts to improve the completeness of the registry data should continue.

As the EU transitions to a new regulatory regimen for clinical research, phasing out the use of EUCTR, our findings highlight the need for increased vigilance of the reporting of results to the new CTIS. Member states must have a proactive role in ensuring compliance with reporting requirements to the CTIS by using the powers delegated to them under the regulations.82 83 Denmark and Belgium have already implemented some policies targeting non-reporting sponsors under these provisions but more direct and widespread audit and enforcement provisions are likely needed.84 85 This approach will ensure continued access to the results of most European trials of medicines. Also, the European Medicines Agency, their affiliated national authorities, and trial sponsors must ensure that the quality of the data on the CTIS is better managed and maintained than currently is the case with EUCTR. These problems obscure the true status of a clinical trial, including whether it even occurred, which complicates independent public audit and undermines the purpose of the registry.24 A continued culture of lax standards for data quality and non-existant enforcement of requirements of reporting of results would create many of the same problems for CTIS that prevented EUCTR from reaching its full potential and have lessened the effectiveness of US regulations.83 86 The use of EUCTR can also continue to grow, however, as various groups put pressure on institutions to improve their reporting practices for older trials.13 81

Even with the introduction of CTIS, EUCTR should not be neglected or forgotten as an important source of information on clinical trials. The European Medicines Agency should continue to promote reporting to the registry, publicise its use, and ensure it remains a supported, publicly available, open source database and archive of details of trials. Currently, the position of the European Medicines Agency is to continue to accept results submitted to the EudraCT/EUCTR system into the future, even after transition to CTIS is complete.87 Although it cannot match the size of ClinicalTrials.gov, and problems with data remain, EUCTR has data on nearly two decades of research, covering more than 40 000 trials and 20 000 results, many on treatments in wide use today. Hence this database should continue to be a relevant resource well into the future for researchers, clinicians, and the public; the launch of CTIS will not lessen its importance.

Improvements to the completeness of the EUCTR results database is achievable; 117 trials in this study had results available elsewhere that could be included in EUCTR, adding to its completeness. The other unreported 117 trials, with no results located in any dissemination route, should also be managed. Two of the 500 trials in this sample had disclosures in EUCTR that the results were lost and permanently unavailable. Increased attention to transparency processes from regulators and sponsor institutions can help ensure that future results are not lost and that an accounting of their fate is made available.

Guidance on searching EUCTR directly as part of systematic searches is available.88 The Cochrane Handbook recommends searching ClinicalTrias.gov and ICTRP, but notes that the search function of the ICTRP database can often miss results from ClinicalTrias.gov compared with direct searches.28 65 89 Our findings suggest that resources like the Cochrane Handbook might consider recommending direct searches of EUCTR, and in the future CTIS, as part of a systematic review of medicinal products to locate studies and results.

Conclusions

EUCTR can help researchers and the public access the results of clinical trials in a timely way, and its reporting requirements can help avoid research waste from non-dissemination of results. Although its full potential is unrealised because of inadequate compliance with EU guidelines and problems with data quality that complicate the routine use of the registry, we showed that results often appeared first on EUCTR and results on EUCTR might be the only results available across common dissemination routes. Therefore, EUCTR should be recommended as a resource for systematic literature searchers because it has the results of thousands of trials from the past two decades of clinical research in Europe and will likely continue to grow in the coming years.

Data availability statement

Data are available in a public, open access repository. All data and code for this analysis are available on Github (https://github.com/ebmdatalab/euctr_pub_search) or the project’s Open Science Framework repository (https://osf.io/r3vc5/).

Ethics approval

This project exclusively used publicly available, non-sensitive data and did not include human subjects, therefore no ethical approval was required.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @ndevito1

  • Contributors ND conceived the study with input and supervision from BG and CH. ND led the design, data collection, analysis, and writing. JM, HD, JAS, and CH contributed to the data collection and analysis. All authors contributed to the discussion and interpretation of the results, which secured the intellectual content of the manuscript. All authors approved the final version for submission. ND is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. Transparency: The lead author (the guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

  • Funding This work was supported by the Naji Foundation who provided ND with a doctoral studentship, and by the Good Thinking Foundation, who supported the TrialsTracker Project. The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

  • Competing interests All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from the Naji Foundation and the Good Thinking Foundation for the submitted work; ND has received additional funding from the Fetzer Franklin Memorial Trust, German Federal Ministry of Education and Research (BMBF), and World Health Organization; BG has received research funding from the Laura and John Arnold Foundation, NHS National Institute for Health and Care Research (NIHR), NIHR School of Primary Care Research, NIHR Oxford Biomedical Research Centre, Mohn-Westlake Foundation, NIHR Applied Research Collaboration Oxford and Thames Valley, Wellcome Trust, Good Thinking Foundation, Health Data Research UK, Health Foundation, World Health Organization, UK Research and Innovation (UKRI), Asthma UK, the British Lung Foundation, Elsevier, and the Longitudinal Health and Wellbeing strand of the National Core Studies programme; BG also receives personal income from speaking and writing for lay audiences on the misuse of science and is a co-founder of the AllTrials Campaign; JM is the recipient of a doctoral studentship from the Wellcome Trust; CH holds grant funding from the NIHR School of Primary Care Research and the World Health Organization, and sits on the International Clinical Trials Registry Platform (ICTRP) Advisory Group; JM receives expenses and fees for his media work, and expenses for teaching evidence based medicine; JAS is currently head of global strategy at Alvea and was previously funded by the NIHR Oxford Biomedical Research Centre; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.