Comparison of published core outcome sets with outcomes recommended in regulatory guidance from the US Food and Drug Administration and European Medicines Agency: cross sectional analysis

Selection of core outcome sets

We examined all core outcome sets for research (including those intended for both research and practice) that involved patients in the consensus process and were published between 1 January 2015 and 31 December 2019. Selection of only those core outcome sets published in the past five years (at the time of beginning our study) that involved patients probably increased the number of COS-STAD standards (Core Outcome Set Standards for Development2) that are met by this sample of core outcome sets. To maximise relevance to regulatory guidance, we restricted the study sample to those sets relating to any intervention or specifically to drug, device, or gene therapy interventions; we excluded sets relating exclusively to surgical interventions because such procedures are not subject to EMA or FDA regulatory oversight.

Identification of matching regulatory guidance documents

For each eligible core outcome set, we identified EMA/FDA guidance documents that addressed similar health conditions, published up to April 2021. Because we are not aware of a readily searchable electronic database for guidance documents, we searched websites of the EMA19 and FDA,20 using the key clinical terms and synonyms as search terms. We refined these searches using Google’s site specific search capability. For example, if searching for guidance documents relating to diabetes, we searched for "diabetes site: fda.gov" on the FDA website and for "diabetes site: ema.europa.eu" on the EMA website. We considered addendums to guidance documents as separate guidance documents because they usually refer to different target populations and/or interventions from those in the original documents. A core outcome set could be matched to more than one guidance document, and vice versa. For all core outcome sets, two investigators (from among IJS, SD, RF, SLG, DAH, PJ, JJK, and DT, each of whom is experienced in core outcome set development or methodology) independently ran searches for guidance documents and resolved disagreements in search results through discussion, consulting SD and PRW as needed.

Assessing overlap in scope between core outcome sets and matched guidance documents

We considered the scope of a given core outcome set as the reference and any given regulatory guidance document to match if they were at least generally matched in terms of both clinical condition/disease and intervention. We used a previously developed framework by Saldanha et al to assess the overlap (figure 1, top panel).9 We considered core outcome sets and regulatory documents to be matched only in scenarios represented by scenarios A-C, E-G, or I-K (ie, those corresponding to at least a general match in both intervention and population between the core outcome set and guidance document; figure 1, top panel). Table 1 provides examples of these scenarios.

Request permission

Extracting information

For each core outcome set, we used an existing database of previously extracted information regarding recommended core outcomes. This database contains all published core outcome sets and is compiled using data extracted for an annually updated systematic review of published core outcome sets.7 24–27 We also reviewed the articles describing each core outcome set (also obtained from the COMET Database) to confirm the list of core outcomes. In the one instance where an article recommended additional outcomes but specified that these were not part of the core outcome set (adverse events in a core outcome set for haemophilia),28 we included those outcomes when comparing the core outcome set to the regulatory document because those outcomes were recommended. We also assessed whether the core outcome set referred to the corresponding regulatory document.

To facilitate standardised data extraction across the team, all investigators participated in an initial pilot exercise using two pairs of core outcome sets and regulatory documents. After that, for the remaining data extraction, two investigators (from among IJS, SD, RF, SLG, DAH, PJ, JJK, and DT) independently extracted all outcomes from each pair of core outcome set and regulatory guidance document. We extracted outcomes regardless of the outcome’s location in the document or the outcome’s status as primary, secondary, or other. Disagreements between pairs of extractors were resolved by discussion among the pairs (and, when needed, with SD).

Matching of outcomes between core outcome set and relevant guidance documents

We focused on the outcome domains (eg, pain for the "what" of an outcome). We did not examine whether the "how" of an outcome (eg, one pain measurement instrument versus another) matched. We considered matching of outcomes separately for each pair of core outcome set and guidance document.

Consistent with previous work,9 29 we considered an outcome in a core outcome set and an outcome in a guidance document to be matched if they were generally or specifically related. Outcome pairs were thus matched generally if one document specified a broad outcome (eg, "disease activity"), while the other was more explicit (eg, "joint damage"); or matched specifically if both documents specified the same explicit outcome (eg, "overall survival" and "all-cause mortality").

Measurement instruments (eg, multi-component questionnaires on quality of life) recommended in guidance documents deserve special mention. If the outcome instrument recommended in a guidance document overlapped with an outcome in a core outcome set, we considered the outcomes as generally matched if the guidance outcome included an overall summary measure that covered more domains (eg, health related quality of life) than just the core outcome set's outcome (eg, physical functioning); or as specifically matched if the instrument covered only the core outcome set's outcome. For all generally matched pairs of outcomes, we also assessed which of the two outcomes was broader.

Statistical analysis

We calculated descriptive statistics (percentages and medians with interquartile ranges) for core outcome sets and relevant guidance documents. We calculated the median percentages of outcomes in core outcome sets that were specific matches, general matches, and non-matches with outcomes in relevant guidance documents overall as well as separately for the EMA and FDA. We constructed scatter plots and used kernel (non-parametric) smoothing to depict potential relations between percentage matching (ie, percentage of outcomes in the core outcome sets that were matched to outcomes in the guidance documents) and the number of outcomes in the core outcome sets. For the kernel smoothing, we used the rule-of-thumb method to calculate window sizes. We conducted all analyses using Stata version 16 (College Station, TX). We estimated the medians (and interquartile ranges) of the proportion of outcomes that overlapped between the core outcome sets and guidance documents.

Patient and public involvement

Patients and the public were not involved in the design, conduct, reporting, or dissemination plans of this research, as this was a methodological study.

Included core outcome sets

Based on a search of the COMET Database, a total of 108 core outcome sets for research, with patients included in the consensus process, were published between 1 January 2015 and 31 December 2019 (figure 2). We excluded 10 core outcome sets relating to specific interventions other than drugs, devices, or gene therapy.

Request permission

Among the remaining 98 core outcome sets, we identified at least one regulatory guidance document for 38 (39%) sets. Table 2 summarises these 38 core outcome sets. Just under a third (32%) of these 38 sets were published in 2017. The most frequent topic areas were neurology (18%), cancer (13%), gastroenterology (13%), and pain (11%). Most (82%) core outcome sets were not developed for a specific type of intervention (ie, they were developed for any intervention). The 38 core outcome sets included a median of eight outcomes (interquartile range 6-11) per set. None of the core outcome sets referred to their corresponding regulatory documents.

Overlap in timing and scope

We found 29 matching EMA guidance documents for 35 core outcome sets (figure 2). These EMA guidance documents were published between 2005 and 2021. Of the 44 pairs of core outcome sets matched with EMA documents, the core outcome set was published before the EMA document in 15 (34%) pairs, published in the same year in five (11%) pairs, and published after the EMA document in 24 (55%) pairs. Figure 1 (middle panel) shows these 44 pairs. The core outcome set was very likely to be relevant for 21 (48%) pairs and may have been relevant for 23 (52%) pairs. The most frequent was scenario C (figure 1, top panel; ie, the core outcome set's population was narrower than the population outlined in the EMA guidance document, but the core outcome set described a broader intervention or set of interventions than the EMA guidance document; 21 (48%) pairs). Both the population and the intervention in the core outcome set and the EMA guidance were exact in scope (ie, scenario F in figure 1 (top panel)) in only one (2.3%) pair.

We found 21 matching FDA guidance documents for 24 core outcome sets (figure 1). These FDA guidance documents were published between 1981 and 2020. Of the 30 pairs of core outcome sets matched with FDA documents, the core outcome set was published before the FDA document in 12 (40%) pairs, in the same year in four (13%) pairs, and after the FDA document in 14 (47%) pairs. Figure 1 (bottom panel) shows these 30 pairs. The core outcome set was very likely to be relevant for 22 (73%) pairs and may have been relevant for eight (27%) pairs. For one in five pairs (20%), both the population and the intervention were exact in scope (ie, scenario F in figure 1, top panel).

Overlap in outcomes

For the 44 pairs of core outcome sets and EMA guidance documents, a median of 70% of outcomes in the core outcome sets (interquartile range 48-86%) were specific or general matches to outcomes in corresponding guidance documents (table 2). A median of 46% (27-68%) were specific matches. Among general matches in outcomes, the EMA outcome was broader in 83% and the COS outcome was broader in 17%. The scatter plot in online supplemental figure 1A suggests a generally inverse relation between the number of outcomes in the core outcome set and the percentage match between the outcomes in the core outcome set and the EMA guidance document. This relation was true for all matches (blue smoothed curve) as well as for specific matches in particular (green smoothed curve).

For the 30 pairs of core outcome sets and FDA guidance documents, a median of 52% of outcomes in the core outcome sets (interquartile range 33-77%) were specific or general matches to outcomes in corresponding guidance documents (table 3). A median of 26% (18-46%) were specific matches. Among general matches in outcomes, the FDA outcome was broader in 75% and the COS outcome was broader in 25%. The scatter plot in online supplemental figure 1B suggests a generally inverse relation between the number of outcomes in the core outcome set and the percentage match between the outcomes in the core outcome set and the FDA guidance document. This relation was true for all matches (blue smoothed curve) as well as for specific matches in particular (green smoothed curve).

Examples of matching

Table 4 provides examples of specific matches, general matches, and non-matches between pairs of outcomes in EMA/FDA guidelines and outcomes recommended in corresponding core outcome sets. For generally matched outcomes, table 4 also includes our assessment of whether the outcome was narrower, broader, or neither in the core outcomes sets than in the matched regulatory documents.

Summary of findings

In this analysis, just under 40% of recent (2015-19) core outcome sets were on topics with relevant EMA or FDA regulatory guidance documents. A median of 70% of outcomes in core outcome sets were specific or general matches to outcomes in EMA guidance documents, and almost half (46%) were specific (ie, exact) matches. Corresponding proportions for pairs of core outcome sets and FDA guidance documents were about half (52%) and quarter (26%), respectively. When outcomes were generally matched, the outcome from the core outcome set was more frequently narrower than the guidance outcome.

Numbers of outcomes per core outcome set

Our analysis found a generally inverse association between the number of outcomes in a core outcome set and the percentage of those outcomes that were either specific or general matches to outcomes recommended in corresponding guidance documents, both EMA and FDA. This finding contradicts the lack of a relation previously shown by us between the number of outcomes in core outcome sets and the percentage of overlap with outcomes in systematic reviews on the same topic.9 We believe that because of the purpose that regulatory guidance documents (examined in the current analysis) serve, they can focus on the most important outcomes of interest for the research question. Core outcome sets with fewer outcomes are also more likely reflect developers’ efforts to prioritise the most important outcomes. These factors might explain why core outcome sets with fewer outcomes in the current analysis had a greater percentage overlap with outcomes in regulatory guidance. It is possible that developers of core outcome sets who recommend a greater number of outcomes adopt a more inclusive view of what a core outcome set should constitute. Another potential reason for the difference between our current and previous findings (from a similar analysis comparing core outcome sets and systematic reviews)9 could relate to the different purposes and intended audiences of regulatory guidance documents and systematic reviews.

Regulatory document purposes

When regulatory bodies, such as the EMA and the FDA, issue guidance on study design (including outcome choice), the guidance is intended to guide the design, analysis, and reporting of studies that evaluate regulated products.30 But, regulatory bodies approve claims made in product labels, not the products themselves. So, regulators are often flexible about which outcomes are included in trials because the general focus of regulators is on determining whether the evidence presented supports the claim about clinical benefit that the manufacturer wants included on the product label.

In the context of regulatory document purposes, surrogate (ie, intermediate) outcomes are also worth discussing. Because trials might not be powered for, or might not have long enough follow-up for, longer term clinical outcomes, it is plausible that regulatory guidance for such studies is more accepting of surrogate outcomes than are developers of corresponding core outcome sets (who generally first prioritise outcomes based on importance rather than feasibility). Kalf and colleagues conducted a cross sectional analysis and suggested that although practices vary across regulators, regulators generally accept surrogate outcomes.31 However, the issue of surrogate outcomes in recent FDA guidance on emerging disease modifying treatments for dementia32 has been a source of concern33 and merits careful consideration and clearer methodological guidance. In the current analysis, we did not evaluate the extent to which surrogate outcomes were recommended and how that might vary comparing regulatory documents and core outcome sets. This area has potential for future research.

Implications for developers of core outcome sets

At the outset, developers should consider the eventual uptake of the core outcome set being developed. Developers should appraise related regulatory guidance documents in the topic area when planning a core outcome set; none of the included sets in the current analysis reported doing this. We also agree with Aiyegbusi and colleagues, who conducted a systematic review of core outcome sets and regulatory guidance and noted that regulators might be relevant stakeholders in outcomes recommended in core outcome sets (especially for those sets related to drugs, devices, and gene therapy); greater collaboration between developers and regulators is warranted; and regulators should participate in the development of core outcome sets.34 Regulator participation might be important because it could enable regulators' preferences (along with those of others) to be considered in the development process for core outcome sets. However, bureaucratic restrictions could preclude regulator participation in core outcome sets. How best to engage regulators in the development and adoption process of core outcome sets remains to be explored. Nevertheless, greater adoption of core outcome sets in regulatory guidance documents has the potential to push clinical trialists, especially those funded by industry, to measure and report outcomes from these sets.

Implications for regulators

The processes of outcome selection for inclusion in regulatory guidance documents generally includes an initial selection by agency staff, followed by input obtained through meetings with clinical experts, patients, and various stakeholders, and engagement with the public.35 Although this input might sometimes include alerting regulators about relevant core outcome sets, the extent to which the sets are being consistently considered in this process is unclear.

Regulators are, however, increasingly recognising the importance of engaging patients in the regulatory process.36 37 As part of its efforts for Patient-Focused Drug Development, the FDA has developed a set of methodological guidance documents to help various stakeholders "collect and submit patient experience data and other relevant information from patients and caregivers for medical product development and regulatory decision making."38 All the core outcome sets examined in the current analysis (and almost 40% of those sets developed by 20197 and over 90% of ongoing sets (unpublished work)) have involved patients in the development process. Regulators should capitalise on this engagement of patients (and various other stakeholders), which is a great strength of the development process for core outcome sets.

Regulators should also be reassured that the overlap in outcomes between core outcome sets and existing corresponding regulatory documents is good. The current analysis finds that as many as 70% of outcomes in core outcome sets were matched to outcomes in corresponding EMA guidance documents and 52% to outcomes in corresponding FDA guidance documents. Thus, core outcome sets could serve as a useful resource for regulators when recommending outcomes for studies evaluating the regulated products.

The majority of core outcome sets in this analysis were published in the same year or after the matched guidance document (66% of COS-EMA pairs and 60% of COS-FDA pairs). Nevertheless, our results support the two main actions that have been suggested for regulators.12 Firstly, when drafting a new or updated guidance document, regulators should review the COMET Database for evidence about relevant, high quality, core outcome sets for the scope of the guidance, and consider those outcomes when developing regulatory guidance. Secondly, regulators should engage with the development process of core outcome sets to help identify barriers and facilitators early on.12

Challenges in conducting this analysis

We encountered some challenges during this analysis that are worth discussing. Firstly, we included core outcome sets for research, recognising that these are not only intended for randomised trials but also for non-randomised studies. However, this choice was reasonable because regulatory guidelines issued by the EMA or FDA target both randomised trials and non-randomised studies. Secondly, in some instances, we had to make particularly careful decisions regarding the match in scope between the core outcome set and the guidance document. For example, one eligible core outcome set addressed interventions for patients on haemodialysis. We found an EMA guidance document addressing primary prevention of chronic kidney disease in groups at risk or secondary prevention (ie, early interventions to prevent worsening of kidney function). We did not consider the core outcome set and the guidance document to be a match because haemodialysis is an example of a treatment for established and advanced chronic kidney disease (ie, haemodialysis is a form of tertiary prevention). Thirdly, when extracting outcomes from guidance documents, it was not always clear whether the document was truly recommending a particular outcome. This extraction required judgment in discerning potentially ambiguous language—for example, that the outcome "should," "could," "might," or "may" be considered, or that the outcome "is important." Some of this ambiguity in language could have arisen because guidance from regulatory bodies regarding outcome choices in studies of regulated products generally is non-binding.

Limitations of the study

The current analysis has some limitations. Firstly, the core outcome sets we analysed were restricted to those with patient involvement in consensus generation, which is what is recommended by COS-STAD.2 However, these sets might not be representative of all core outcome sets. Among all core outcome sets published by 2020, those that involved patients included a somewhat larger number of outcomes (median 8, interquartile range 5-12) than those that did not involve patients (6, 3-8; COMET, personal communication). Therefore, given the inverse relation between the number of outcomes in a core outcome set and the percentage overlap between core outcome sets and regulatory outcomes (documented in this paper), the overlap may have been higher if we looked at a sample of core outcome sets without patient involvement. The core outcome sets in the current analysis represented a recent five-year sample, and the level of adherence to COS-STAD standards (including patient involvement) has generally been improving.7 24–27

Secondly, although we applied our systematic search methods for guidance documents consistently, some guidance documents from the EMA or FDA could have been missed. Thirdly, for regulatory guidance documents, we restricted our search to the EMA and FDA. Although the EMA and FDA are two prominent healthcare regulatory bodies, we recognise that there are other regulatory bodies in other regions and countries. As such, our findings should be inferred to be informed by guidance produced by the EMA and FDA.

Conclusions

We found sizeable overlap between outcomes in core outcomes sets and in corresponding EMA and FDA guidance documents. We encourage developers to involve regulators in core outcome set development and to consider outcomes recommended in regulatory guidance documents. We encourage regulators to engage with the development process of core outcome sets (to help identify barriers and facilitators early on) and, when drafting a new or updated guidance document, to review the COMET Database for relevant high quality core outcome sets.