Availability of results of clinical trials registered on EU Clinical Trials Register: cross sectional audit study

Data sources

Data from all available EUCTR protocols (ie, details of the trial in each country) and results sections were extracted with custom web scraping software between 1 December 2020 and 3 December 2020.31 32 A trial record on the EUCTR is made up of a protocol for every EU country in which the trial took place, a protocol for non-EU locations of some paediatric trials, and, if available, a results section covering the whole trial. Key fields for this study from EUCTR included trial status (eg, ongoing, completed), date of ethics committee opinion and date of competent authority decision (ie, as proxies for the start of the trial), date of the global end of the trial, and the initial estimate of the duration of the trial from section E.8.9 of the protocol information on EUCTR (online supplemental figure S1).

Study population and sample

Trials removed from the full December 2020 dataset were those with a status of not authorised or prohibited by competent authority, and those with ethical approval dates from before the launch of the registry or later than the data extraction date. The remaining trials were checked for the latest completion date in the individual trial protocols or in the results section. Completion dates from the results section were preferred when available.

Poor data on completion of a trial are a known concern for EUCTR because many trials lack updated statuses and completion dates.33–35 To ensure that these trials could be captured within our population, and to avoid selection bias for trials with better record keeping, we developed a method to infer a trial completion date when this information was missing. Each trial protocol in the dataset without a completion date was assigned a start date (ie, from the ethics and regulatory start dates provided, whichever was later, because no clear start date field exists on EUCTR) and an expected duration in days calculated from protocol section E.8.9 (online supplemental figure S1).

All protocols from a trial record were then grouped into one record, where the longest estimated duration was added to the latest start date. Another year was conservatively added to this date, to allow for any delays in the start or conduct of the trial, resulting in a final inferred completion date. A Jupyter Notebook detailing this approach and its validation is available on the project's Open Science Framework repository (https://osf.io/r3vc5/).

For trials with an extracted or inferred completion date, we limited our population to those completed at least two years in the past (ie, before 1 December 2018) to allow time for reporting across all dissemination routes.36 37 From this population, a random sample of 500 trials was taken for the analysis by using the .sample() method in the Pandas Python package. This sample size was chosen based on achieving point estimates with a maximum 95% confidence interval of ±5% (online supplemental box S1). The maximum sample needed to achieve this level of precision was 384; a final sample of 500 trials was chosen to allow for greater precision in point estimates of subpopulations. Trials in the sample found, at any point, to have issues that would make it difficult or impossible to discover the results, such as being withdrawn without enrolment, still ongoing, or currently inaccessible on EUCTR, were excluded from the analysis sample and replaced with another randomly chosen trial.

Results search strategy

The search strategy was piloted in 50 trials, with a subset searched in duplicate. We found high percentage agreement on data extraction, and discrepancies were easily resolved through discussion. The pilot informed the decision to only use open, public databases (ie, PubMed, Google Scholar) to allow for greater reproducibility because proprietary databases (ie, Scopus, Ovid) did not give substantially more value in locating results. Details from the pilot searches are available in the protocol and on the project's Open Science Framework repository (https://osf.io/r3vc5/).

Each EUCTR trial record was reviewed for results and information about duplicate registrations or published results. ClinicalTrials.gov and the ISRCTN registry were then searched for potential cross registrations. Both of these registries can host results directly on the registry, and because of their size, geographical focus,38 39 and related regulations,40 41 would be expected to have cross registrations of trials on the EUCTR. After peer review feedback, any remaining trials with no additional registrations located were searched in the International Clinical Trials Registry Platform (ICTRP) database to confirm that no documented cross registrations existed. Registries were first searched with the EUCTR unique trial identifier and then with the trial title, name/acronym, intervention, condition, sponsor, and any additional secondary trial identity numbers. Matching records were searched for additional relevant information; an eligible result on a registry was hosted directly on the registry rather than linking to an external article or resource. Trials that declared, in place of results, that no analysis was possible (eg, because of low enrolment) were counted as having results because enough detail to understand the fate of the trial would be available to interested parties.

Lastly, PubMed and Google Scholar were searched for journal publications with all known trial identity numbers, trial title, acronym, interventions, conditions studied, and any investigator names and affiliations available in the registrations. Searchers could combine these terms, or add more terms, to their searches at their discretion. Results in the literature were included if they were available as a publication in a journal, reported final primary results of the trial, and were >500 words in length (eg, detailed conference abstracts), consistent with previous methods.37 If multiple eligible publications were located, we recorded the earliest. Only results published before the start of searches were included in the final analysis. Article and registry matches were confirmed through comparison of trial identity numbers, study design, indication, intervention, planned enrolment, and registered outcomes.36 No specific threshold was set to define a match between records, but any problems matching registrations and publications were referred to the full study team for more discussion.

One author (ND) searched all trials and 50% were also searched by a second author (JAS, JM, HD) to validate the search strategy and data extraction. The original sample of 500 trials was searched by the lead researcher (ND) between December 2020 and July 2021; secondary searches of half the sample began in April 2021 and concluded in January 2023. Further checks and searches were carried out in 2023 in response to peer review. Any uncertainties or discrepancies were resolved by consensus discussions, with remaining concerns referred to a senior member of the study team (CH) for final adjudication. Problems with extracting publication date because of inconsistencies between sources (ie, PubMed, journal websites) were resolved by the lead author by re-extracting all publication dates, preferring the date on the journal website when available. Online supplemental table S1 details the reliability measures between the searchers.

Outcomes and statistical analysis

The primary outcome was the proportion of trials with results for the examined dissemination routes, and for each route individually. Prespecified secondary outcomes were the number and proportion of unique results, and the timing of results, for each dissemination route. Analysis of the timing of results was altered from prespecification (online supplemental box S2). Differences in reporting statistics between the groups of trials with inferred and extracted completion dates were assessed with a two proportion z test (unadjusted α=0.05, with Holm-Bonferroni corrections). The presence of trial identity numbers in a journal publication was added as a post hoc secondary outcome. This information was extracted during data collection for validation purposes and provides insights into how the published literature linked back to the EUCTR registration.

Exploratory analyses

We performed two prespecified exploratory analyses. For trials with results available in any dissemination route, factors associated with results appearing on EUCTR were examined with univariable and multivariable logistic regression. Prespecified factors were whether the completion date was reported or inferred, trial start year, sponsor type, number of EU protocols registered, final enrolment (intent to treat), and whether the trial was conducted only in the EU or European Economic Area (EEA), in countries inside and outside of the EU and EEA, or entirely outside of the EU and EEA. Trial start year, final enrolment, and location variable were extracted manually by the lead author (ND) from trial registrations and the results located during searches. If contradictions between sources arose, the most recently updated or available source was preferred; if further ambiguity existed, the EUCTR data were preferred. The remaining variables were directly extracted from EUCTR data in code, with missing data coded as unknown. Significance for the univariable and multivariable models was determined with the Holm-Bonferroni method (unadjusted α=0.05).

The second exploratory analysis examined variation in reporting behaviour by sponsor country. The sponsor country field (ie, protocol section B.1.3.4 on EUCTR) was extracted for all country level protocols for each trial, and each trial was assigned the most frequently appearing sponsor country across protocols. In the event of ties, the trial was coded as having multi-country sponsorship. If no sponsor was located, the sponsor country was coded as unknown. The number of trials reporting any results, and reporting results to EUCTR, other registries, and the literature was examined for each sponsor country.

Patient and public involvement

No patients or members of the public were involved in determining the research question, outcome measures, or interpreting the results as this was a doctoral student project without funding to support patient and public involvement. The results of this study will be summarised for the public in a blog post by the first authors on publication, disseminated on the Bennett Institutes for Applied Data Science and TranspariMED websites to their relevant audiences, and publicised on social media.

Study population and sample

As of 1 December 2020, 98 622 individual country protocols were registered to EUCTR for 38 566 trials. After exclusions, the final overall population included 66 833 individual protocols for 27 241 trials. During searches, we found evidence indicating that 22 trials in our sample were withdrawn without enrolment (n=16), were still ongoing (n=3), or were inaccessible on EUCTR (n=2), and were replaced according to our protocol. One other trial seemed to have mistakenly been registered on EUCTR because it was an observational study and the sponsor noted that they did not plan to upload results (2013-001141-14); this trial was also replaced because results could not appear on EUCTR. Table 1 shows the characteristics of the final sample of 500 trials from this population.

For the whole of EUCTR, we extracted an end date for 21 766 trials (56.4%) and inferred an end date for 16 051 (41.6%) trials; 749 (1.9%) trials had insufficient information about completion or had not received authorisation. After excluding trials with completion dates of less than two years in the past (n=10 576), 27 241 trials formed the population to be sampled: 19 182 (70.4%) trials had extracted end dates and 8059 (29.6%) had inferred end dates. Our final random sample of 500 trials contained 354 (70.8%) trials with extracted completion dates and 146 (29.2%) with inferred completion dates (figure 1). EUCTR was the only registration available for 138 trials (27.6%, 95% confidence interval 23.7% to 31.5%) (online supplemental figure S2). For trials with known extracted completion dates, trials were completed a median of 3040 days (interquartile range 1884-4137; range 745-5755), or 8.3 years, before the start of the searches; trials with inferred completion dates were expected to have been completed for a median of 2804 days (interquartile range 1788-3513; range 791-5937), or 7.7 years, before the start of the searches (online supplemental figure S3). Moods median test showed no difference in median follow-up between these two samples (P=0.14).

Request permission

Trial registration and results reporting

Table 2 shows the availability of results for each of the examined dissemination routes for the 500 trials in our sample. A total of 694 results were located for 383 (76.6%, 95% confidence interval 72.9% to 80.3%) trials across all dissemination routes; EUCTR had results for 266 (53.2%, 48.8% to 57.6%) registered trials. Of the 383 trials with results, 55 (14.4%. 10.9% to 17.9%) were only available on EUCTR. Results on EUCTR were most commonly provided only in the registry’s tabular format (n=117, 43.9%); for unique results, the most common format was clinical study report synopsis documents (n=30, 54.6%) (online supplemental table S2). The most common dissemination route overall was publishing only in a peer reviewed journal (n=108); 117 (23.4%, 19.7% to 27.1%) trials had results elsewhere but not on EUCTR, and 117 (23.4%, 19.7% to 27.1%) had no results disseminated anywhere, both indicating a failure to report under EU guidelines.

For the 293 journal articles located, six (2%) were conference abstracts that met the length criteria for inclusion (ie, >500 words). Another 24 abstracts were located that seemed to be matched to EUCTR registrations but did not meet the length criteria for inclusion; seven of these would have represented the only results available if they had been included. Also, of the 266 trials with results on EUCTR, five (1.8%) were included that provided a statement on EUCTR declaring that no analysis was possible because of low enrolment in place of results. Based on enrolment values extracted for the exploratory analyses, the 117 unreported trials enrolled, or planned to enrol, 33 673 participants. Figure 2 shows an UpSet plot visualising all combinations of availability of results for all of the dissemination routes.42

Request permission

Figure 2

Availability of results for the four dissemination routes. EUCTR=European Union Clinical Trials Register. Dark dots represent the combination of dissemination routes measured in the bar chart above them with the lighter dots excluded from that set. For instance, the first column of dots represents trials with no results in any route, and the last column represents trials with results on all routes examined

Trials with a known completion date were significantly more likely to have results available in any dissemination route compared with those with an inferred end date (88.1% v 48.6%, P<0.001) and among all individual dissemination routes except ISRCTN (table 2). Trial results outside EUCTR were also more likely for trials with extracted versus inferred end dates (72.9% v 48.0%, P<0.001). Only one trial with an inferred end date had results available on EUCTR but this finding was not considered unusual because the methods used to extract completion dates often relied on the availability of a results section. When results were grouped by the status of the sponsor (ie, commercial v non-commercial), as requested during peer review, commercial sponsors consistently showed higher rates of reporting and higher rates of known completion dates, indicating better management of registry data on EUCTR (online supplemental table S3).

Timing of results reporting

For 291 trials that could have appeared in ClinicalTrials.gov, EUCTR, or a journal article, results most commonly first appeared in a journal article (n=156, 53.6%). The results section of ClinicalTrials.gov did not launch until October 2008, however, and the results section of EUCTR did not launch until March 2014. When accounting for these differences, first availability of trial results in this sample changed from strongly favouring journal articles (n=64, 73.6%) over ClinicalTrials.gov (n=23, 26.3%) before the launch of the results section of EUCTR, to EUCTR (n=84, 43.5%) having relative parity with journal articles (n=89, 46.1%) as the earliest dissemination route compared with ClinicalTrials.gov (n=20, 10.4%) (online supplemental table S4).

Figure 3 shows cumulative incidence curves examining time from extracted completion date to reporting for each dissemination route for trials with a first result after the launch of EUCTR. Median time to a result was similar for EUCTR (1142 days, 95% confidence interval 812 to 1492) and a journal publication (1226 days, 1074 to 1551) and much faster than the median time to results on ClinicalTrials.gov for trials that could appear there (3321 days, 1653 to undefined). EUCTR also continued to add new results long after trial completion whereas both journal publication and ClincialTrials.gov results began to plateau at about 2000 days from completion with few new results added. Online supplemental figure 4 shows the distribution of start years for the trials with no results in any dissemination route and no results on EUCTR, with the proportion of missing results remaining relatively consistent over time.

Request permission

Reporting of trial identity numbers

Online supplemental table 5 shows the proportion of journal articles, matched to a trial registration, that contained the trial identity number of that registry. Only 22.5% (95% confidence interval 17.7% to 27.3%) of journal articles had a linked EUCTR identity number, substantially less than identity numbers for ClinicalTrials.gov (83.3%, 78.4% to 88.2%) or ISRCTN (62.5%, 43.1% to 81.9%).

Exploratory analyses

In the exploratory risk factor analysis, we excluded whether the trial had an inferred or extracted end date as a potential risk factor, despite prespecification, because this factor was nearly a perfect predictor of non-reporting of results to EUCTR (table 2). In the univariable models, being a commercial sponsor (odds ratio 14.40, 95% confidence interval 8.47 to 24.49), having more EU protocols (1.57, 1.32 to 1.87), and with sites inside and outside of the EEA (reference EEA only: 6.13, 3.50 to 10.73) increased the likelihood of results appearing on EUCTR. In the fully adjusted multivariable analysis, only commercial sponsorship remained significant (adjusted odds ratio 9.75, 95% confidence interval 5.04 to 18.85, P<0.001). Online supplemental table S6 shows all of the results of the model.

The second exploratory analysis examined dissemination by sponsor country. Online supplemental table S7 includes the reporting practice of all sponsor countries identified. We found substantial heterogeneity in the use of different dissemination routes for results. In half of the 12 countries with >10 sponsored trials, dissemination rates were highest to EUCTR. The two lowest reporting countries to EUCTR, Italy (17.4%) and Spain (13.9%), also had the lowest availability of results for all dissemination routes among the most active sponsoring countries (Italy 54.4%, Spain 50.0%).

Summary of results

In our sample of 500 trials, availability of results on EUCTR (53.2%, 95% confidence interval 48.8% to 57.6%) was similar to the peer reviewed literature (58.6%, 54.3% to 62.9%) and exceeded the proportion of results available on other registries with matched records. Among the 383 trials with results, 55 (14.4%, 10.9% to 17.9%) were only available on EUCTR. Also, after the launch of the results section of the EUCTR database, median time to results was fastest on EUCTR (1142 days, 95% confidence interval 812 to 1492), comparable with the literature (1226 days, 1074 to 1551) and exceeding ClinicalTrials.gov (3321 days, 1653 to undefined). For 117 trials (23.4%, 19.7% to 27.1%), however, results were disseminated elsewhere but were not submitted to EUCTR, and no results were reported in any dissemination route for 117 trials (23.4%, 19.7% to 27.1%).

Strengths and limitations

This analysis provides a comprehensive assessment of the fate of the results of trials registered on EUCTR. We searched the literature as well as other large, common registries for EU registered trials, which improved our ability to locate linked results and compare dissemination across various routes. We also saw high agreement from the search strategy between searchers.

The extent of missing completion information on EUCTR, detailed in previous research, influenced the design of this study.24 35 The choice to include trials with both explicit and inferred completion dates has strengths and weaknesses. Although this method might introduce some error in the analysis, mainly by including ongoing or withdrawn trials with incorrect registry data that cannot be identified, it also provides a comprehensive real world view of reporting by avoiding selection bias for only those trials with the best managed data. Overall, this method seemed successful because the existence of most trials (67%) with inferred dates were confirmed by the availability of results or other registrations that also did not list the trial as withdrawn. Reporting disaggregated results for these two subpopulations gives further context to the findings.

Ideally, whether a trial never occurred or was ongoing would be clear from the trial registration. If the comprehensive search strategy used in this analysis could not identify the status of these trials, a search strategy for a systematic review is also unlikely to do so, unless the authors are contacted directly. Because of time and resource constraints, we did not contact investigators as part of this assessment. Contact information on EUCTR is frequently missing, and therefore would have added a substantial burden to data extraction as well as a lengthy process of managing outreach. Uncertainty about the current status of a trial could lead to wasted effort and time by reviewers in trying to locate expected results that do not exist.43

Results in context

Hwang and colleagues44 assessed whether some paediatric trials registered on EUCTR had results, compared with ClinicalTrials.gov and the published literature. They found a slightly higher rate of overall reporting (85%) with less reliance on the literature for unique results than in our sample.44 Deane and colleagues45–47 examined the reporting of trials supporting drug approvals by the European Medicines Agency, from 2009 to 2013; they found that by including EUCTR registrations, 11% of records were added to the most recent analysis. EUCTR did not include results at the time of this analysis, however, and de-duplicated record counts by registry were not reported.45–47

Speich and colleagues48 examined the reliability of registry information for 360 studies with ethics approval from Switzerland, the UK, Canada, and Germany. These authors found higher rates of availability of results for both ClinicalTrials.gov (57%) and EUCTR (69%) registrations compared with our findings, with none of 20 studies reporting to ISRCTN. Also, 69% of trials registered on both EUCTR and ClinicalTrials.gov with a result on one registry, also had a result on the other (v 55% in our study). The study of Speich and colleagues was more focused in both geography and time compared with our analysis, however, and so the different reporting rates are not surprising.48

Other research focused on the availability of results for registered trials in the EU did not include EUCTR or did not search the published literature. An examination of reporting by top European non-commercial sponsors focused only on EUCTR results,9 and studies of trial reporting at German and Polish academic medical centres did not include EUCTR as a results source.37 49 50 Other similar studies often included populations of trials that overlapped with EUCTR, but explicit search of EUCTR was rare.14–23 Past research by staff of ClinicalTrials.gov showed that in a sample of 380 completed studies with results, 69% had results available on the registry within 48 months, compared with just 40% in the literature, further supporting that registries can disseminate results more efficiently.27 Future research should assess the quality of results on EUCTR compared with the literature. This research would allow greater understanding of the value of hosted results and expand on previous work examining results on ClinicalTrias.gov which consistently showed that some areas, most notably adverse events, were better reported to the registry than to matched journal publications.51–54

Our work also gives important context to audits of reporting practice on EUCTR. EU TrialsTracker is a live audit tool that necessarily uses a conservative definition of when a trial is due to report. The wider population of trials considered for results searches in this study showed that despite considerable progress in reporting tracked by the EU TrialsTracker over time,9 10 problems with data quality affect precise assessments. In December 2020, when the data for this study were extracted, the reporting rate on the EU TrialsTracker was 68%, but in our sample, the reporting rate was just over half of the examined trials (53%). Only one trial with an inferred end date had a result on the registry, in the form of a clinical study report synopsis, despite results existing on other routes for nearly half (48%) of these trials.

Although the EU TrialsTracker provides a valuable public audit and feedback service, and longitudinal data on performance over time, the justifiably conservative methods to minimise false positive “due” trials will overestimate the true level of compliance because of missing and incomplete data. Also, any attempts to manually or automatically link the published literature to a matching EUCTR registration is complicated by the fact that EUCTR trial identity numbers are rarely attached to the resulting publication. Failure to include trial identity numbers in manuscripts and research databases is a persistent problem,55–61 and reporting guidelines and editorial guidance should be amended to clarify that all relevant trial registration numbers, not just one, should be included in publications, abstracts, and metadata.62 63

Lastly, although our exploratory analyses were not designed to provide definitive assessments, the results were consistent with previous work. Commercial sponsorship was the only significant (P<0.001) predictor of availability of results on EUCTR in the adjusted risk factor analysis, which matches the strong associations seen in previous work on reporting under US and EU guidelines.10 64 This finding is likely a result of better resourcing and processes for compliance at pharmaceutical companies. Also, very low reporting to the registry by sponsors in Italy and Spain was similarly shown in an analysis of the reporting of major non-commercial sponsors throughout Europe.9

Implications for policy and practice

Clinical trial registries are an increasingly important source for results, with the potential to help systematic reviews and other evidence searches,51–54 but despite recommendations65 66 they are often underutilised.67–76 The 2012 EU reporting guidelines, which governed reporting of results to EUCTR, operated under a soft requirement approach, where reporting was mandatory but compliance was voluntary, with no sanctions for failing to report.77 This approach led to a well reported discrepancy between commercial and non-commercial reporting compliance.9 10 The most successful effort to improve non-commercial reporting was pressure on public sponsors by the UK Parliament before Brexit.9 78 Other actions from the European Medicines Agency,79 national competent authorities,80 and transparency advocates81 have helped in increasing reporting by non-commercial sponsors to varying degrees. These efforts have substantially increased the number of results available on EUCTR and therefore its value as a research tool and public repository for results. Our findings indicated that EUCTR already has value as a unique repository of results, and therefore efforts to improve the completeness of the registry data should continue.

As the EU transitions to a new regulatory regimen for clinical research, phasing out the use of EUCTR, our findings highlight the need for increased vigilance of the reporting of results to the new CTIS. Member states must have a proactive role in ensuring compliance with reporting requirements to the CTIS by using the powers delegated to them under the regulations.82 83 Denmark and Belgium have already implemented some policies targeting non-reporting sponsors under these provisions but more direct and widespread audit and enforcement provisions are likely needed.84 85 This approach will ensure continued access to the results of most European trials of medicines. Also, the European Medicines Agency, their affiliated national authorities, and trial sponsors must ensure that the quality of the data on the CTIS is better managed and maintained than currently is the case with EUCTR. These problems obscure the true status of a clinical trial, including whether it even occurred, which complicates independent public audit and undermines the purpose of the registry.24 A continued culture of lax standards for data quality and non-existant enforcement of requirements of reporting of results would create many of the same problems for CTIS that prevented EUCTR from reaching its full potential and have lessened the effectiveness of US regulations.83 86 The use of EUCTR can also continue to grow, however, as various groups put pressure on institutions to improve their reporting practices for older trials.13 81

Even with the introduction of CTIS, EUCTR should not be neglected or forgotten as an important source of information on clinical trials. The European Medicines Agency should continue to promote reporting to the registry, publicise its use, and ensure it remains a supported, publicly available, open source database and archive of details of trials. Currently, the position of the European Medicines Agency is to continue to accept results submitted to the EudraCT/EUCTR system into the future, even after transition to CTIS is complete.87 Although it cannot match the size of ClinicalTrials.gov, and problems with data remain, EUCTR has data on nearly two decades of research, covering more than 40 000 trials and 20 000 results, many on treatments in wide use today. Hence this database should continue to be a relevant resource well into the future for researchers, clinicians, and the public; the launch of CTIS will not lessen its importance.

Improvements to the completeness of the EUCTR results database is achievable; 117 trials in this study had results available elsewhere that could be included in EUCTR, adding to its completeness. The other unreported 117 trials, with no results located in any dissemination route, should also be managed. Two of the 500 trials in this sample had disclosures in EUCTR that the results were lost and permanently unavailable. Increased attention to transparency processes from regulators and sponsor institutions can help ensure that future results are not lost and that an accounting of their fate is made available.

Guidance on searching EUCTR directly as part of systematic searches is available.88 The Cochrane Handbook recommends searching ClinicalTrias.gov and ICTRP, but notes that the search function of the ICTRP database can often miss results from ClinicalTrias.gov compared with direct searches.28 65 89 Our findings suggest that resources like the Cochrane Handbook might consider recommending direct searches of EUCTR, and in the future CTIS, as part of a systematic review of medicinal products to locate studies and results.