Performance of polygenic risk scores in screening, prediction, and risk stratification: secondary analysis of data in the Polygenic Score Catalog

Reported performance metrics

By April 2022, the Polygenic Score Catalog had curated 13 828 performance metric estimates for 2194 polygenic scores (unique polygenic score codes), for 544 diseases or traits (unique experimental factor ontology identifiers), reported in 303 unique publications. We removed polygenic scores for continuous traits and those with implausible values (167 instances where the hazard ratio or odds ratio for one standard deviation was recorded as <1, two instances where the area under the receiver operating characteristic curve was <0.5, and one instance where the C index was recorded as 632), leaving 3915 performance metric estimates for 926 polygenic risk scores with 310 unique binary outcomes (mainly diseases). The reported performance metrics were odds ratio for one standard deviation in 1216 instances, hazard ratio for one standard deviation in 378, area under the receiver operating characteristic curve in 2077, and C index in 244 instances (online supplemental file 1).

Converting reported metrics to useful performance metrics

Polygenic risk scores have a gaussian distribution with the same standard deviation in affected and unaffected groups (online supplemental file 2). We used these properties together with the metrics reported in the Polygenic Score Catalog to mathematically derive measures that are more useful for judging the performance of polygenic risk scores in their intended applications.

The first step in this calculation is to use the reported metrics to calculate the difference in mean values for polygenic risk scores between affected and unaffected groups. The calculations have been described previously23–25 and are explained in detail in online supplemental file 2. We used the calculated difference in mean values to determine the overlap in the distributions of the polygenic risk score between the affected and unaffected groups. This method allows calculation of the detection rate and false positive rate, which are the percentage of people with a polygenic score above a particular cut-off value (positive test) among those who are later affected or remain unaffected by disease, respectively. For simplicity and consistency, we set the polygenic score cut-off value at the 95th centile for the unaffected group (1.645 standard deviation units from the mean). This cut-off value defines a 5% false positive rate, and the corresponding detection rate (DR₅) is the detection rate for a 5% false positive rate.23 25 The work sheets in online supplemental file 1 can be used to enter the odds ratio or hazard ratio for one standard deviation, area under the receiver operating characteristic curve, or the C index and return the DR₅ value. A risk screening converter that returns the detection rate for any user defined false positive rate value is available at https://www.medicalscreeningsociety.com/rsc.asp.

The likelihood ratio in screening is the ratio of the detection rate/false positive rate. In individual risk prediction, the likelihood ratio is the ratio of the heights of the gaussian distribution curves for affected and unaffected individuals at a particular polygenic risk score centile. In risk stratification, the likelihood ratio is the ratio of areas under the relative frequency distributions for affected and unaffected individuals in each polygenic score quantile (eg, each fifth of the polygenic score distribution; figure 1 and online supplemental file 2). In each case, multiplying the likelihood ratio by the background odds of disease for the whole population gives the corresponding odds of becoming affected for the individual or group of interest. When we discuss a polygenic score centile or quantile, we are referring to the distribution in the unaffected group. When referring to a particular polygenic risk score, we used the Polygenic Score Catalog identifier number.

Request permission

We reanalysed tabular data taken from two original sources5 26 to quantify the extent to which the addition of information on polygenic risk score to data on conventional risk factors improved the prediction of coronary artery disease and stroke. For this analysis, we extracted the reported counts of affected and unaffected individuals above and below the 10 year risk cut-off values recommended in guidelines for starting treatment with statins. We did this analysis separately for the counts reported by the authors for conventional risk factor models, and for the conventional risk factor models with the addition of polygenic risk scores (online supplemental file 1). We used these counts to calculate the detection rate and false positive rate with and without information on the polygenic risk score. We then calculated the number of individuals who need to be genotyped (and a polygenic score calculated) to detect or prevent one additional coronary artery disease event or stroke with the authors own assumptions about the risk reduction from treatment with statins. We refer to this value as the number needed to genotype (table 1). We modelled the use of a breast cancer polygenic risk score to prioritise mammographic screening at age 40 rather than from the currently recommended age of 50.

Patient and public involvement

Patients and the public were not involved in the design, or conduct, or reporting of this research. We plan to work with patients and the public to disseminate the findings through the patient and public representative groups of the Multimorbidity Mechanisms and Therapeutics Research Collaborative and the UCL Hospitals NIHR Biomedical Research Centre.

Performance of polygenic risk scores in screening

For all diseases studied, median DR₅ based on all polygenic risk scores was 11% (interquartile range 8-18%); that is, 89% (82-92%) of patients were missed. Median DR₅ values for polygenic risk scores whose performance was reported with odds ratio or hazard ratio for one standard deviation were 9% (6-12%) and 8% (7-10%), respectively. For polygenic risk score performance reported based on area under the receiver operating characteristic curve or the C index, median DR₅ values were 14% (10-22%) and 19% (13-25%), respectively. Figure 2 shows median DR₅ values for polygenic risk scores for 28 common diseases, including coronary artery disease and breast cancer.

Request permission

Coronary artery disease

Median DR₅ from performance metrics for 27 polygenic risk scores for coronary artery disease was 12% (interquartile range 9-20%; 88% of patients missed) (figure 3), corresponding to a likelihood ratio of 2.4. Applied at age 50, with a background 10 year risk of coronary artery disease of 5% (odds 1:19), the odds of becoming affected given a positive result were 2.4:19 or about 1:8; that is, false positive results outnumbered true positive results by about eight to one. Reducing the cut-off value to reduce the false positive rate to 1% reduced the detection rate to 3%, with 97% of patients missed. Retaining a 5% false positive rate but applying the test in a population with a risk of coronary artery disease of one in 56 (about 2%), over the same period (background odds 1:55; eg, at about age 40 years of age), gave an odds of becoming affected given a positive result of 1:23, with false positive results outnumbering true positive results by just over 20 to one.

Request permission

Figure 3

Relative polygenic risk score distributions among those later affected or not by coronary artery disease and breast cancer. Mean value of polygenic risk score distribution in those later affected was shifted 0.48 standard deviation units to the right of the mean of the distribution for those who remained unaffected by coronary artery disease, and 0.37 standard deviation units to the right for breast cancer. Also shown are corresponding values for detection rate for a 5% false positive rate (DR₅) and for odds ratios (rounded to the nearest whole number) for comparisons of top and bottom 1%, 5%, 10%, 20%, and 25% of unaffected polygenic risk score distribution

Performance of polygenic risk scores in individual risk prediction

The overlap in distributions of polygenic scores derived from the metrics in the Polygenic Score Catalog allowed calculation of the likelihood ratio for an individual which, together with the background odds of the disorder for the population, can be used to calculate the odds of becoming affected for that individual (online supplemental file 2).25

Coronary artery disease

The odds of developing coronary artery disease in the next 10 years were 1:54, 1: 29, 1:15, and 1:8 with a polygenic risk score at the 2.5th, 25th, 75th, and 97.5th centiles, respectively, at age 50 (when the background odds was 1:19) (figure 4), and 1:157, 1:85, 1:45, and 1:24, respectively, at age 40 (when the background odds was 1:55).

Request permission

Figure 4

Likelihood ratios and 10 year odds of coronary artery disease and breast cancer for people aged 50 with a polygenic risk score result corresponding to 2.5th, 25th, 75th, and 97.5th centiles of the corresponding distribution

Performance of polygenic risk scores in risk stratification

Coronary artery disease

Figure 5 shows the overlapping distributions for affected and unaffected individuals applied to a hypothetical cohort of 100 000 men aged 50 years grouped into polygenic risk score quintile groups. The 10 year odds of coronary artery disease were reduced from the average of 1:19 for all men to 1:41 for those in the lowest quintile group, and increased to 1:11 in the highest quintile group. Figure 6 focuses on those at the highest risk, and the 10 year odds were 1:7 for the upper 2.5% of the polygenic risk score distribution for coronary artery disease, but this group contributed only 7% of patients.

Request permission

Figure 5

Likelihood ratios, odds, and number of affected and unaffected individuals for each quintile group in a hypothetical population of 100 000 individuals with a background 10 year odds of coronary artery disease of 1:19, and women with a 10 year odds of breast cancer of 1:41

Request permission

Figure 6

Likelihood ratios and 10 year odds of coronary artery disease and breast cancer for people aged 50 comparing highest and lowest 2.5% of the unaffected polygenic risk score distributions

Polygenic risk scores for screening along with conventional risk factors or tests

Breast cancer

Using polygenic risk scores to prioritise the use of established screening tests for cancer has also been proposed.3 One suggestion is that younger women should undergo mammographic screening if their risk of breast cancer, determined with a polygenic risk score, exceeds that of an average woman aged 50, the age when mammography is offered to all women. Figure 6 shows that women aged 40 at or above the unaffected 97.5th centile of a breast cancer polygenic risk score distribution have an odds of breast cancer of 1:19, higher than the average 10 year odds at age 50 of 1:41.30 Figure 7 shows that by using the breast cancer polygenic risk score (Polygenic Score Catalog identifier PGS000004) as a stage 1 screen in 100 000 women aged 40, applying the unaffected 97.5th centile as a cut-off value would result in 2570 women with a high risk polygenic score being offered mammography, of whom 108 would be affected and 2462 unaffected (odds of becoming affected given a positive result 1:23). Assuming 100% uptake and a DR₈ value of 75%,31 mammography would then correctly identify 81 of the 108 affected individuals but miss 27 patients with breast cancer. However, 1430 patients with breast cancer (over 10 times as many) are estimated among the 97 430 women aged 40 with a polygenic risk score below the unaffected 97.5th centile who would not be offered mammography.

Request permission

Figure 7

Estimated number of patients with breast cancer detected and missed, number of false positive results, and number of additional mammograms for a two stage screening test with a polygenic risk score (Polygenic Score Catalog identifier PGS000004) with a cut-off value at the unaffected 97.5th centile. Estimates are based on a hypothetical cohort of 100 000 women aged 40 with a background 10 year odds of breast cancer of 1:41. Performance of mammography in the detection of breast cancer uses estimates from the literature31

Principal findings

Our results showed the poor performance of polygenic risk scores in population screening, individual disease prediction, and population risk stratification. This finding is not obvious from the metrics reported in the Polygenic Score Catalog but is clear based on the appropriate metrics used in this study. Our conclusion is consistent with that of others,16 17 19 but is insufficiently recognised. The findings are relevant to consumers, patients, doctors, those involved in preventive medicine and public health, as well as funders and policy makers.

Polygenic risk score distributions overlapped substantially for all conditions studied, and this extensive overlap constrained their performance in each of their intended applications, whether used alone or in combination with conventional risk factors or screening tests. For example, achieving a clinically useful performance in population screening, such as an 80% detection rate for a 5% false positive rate (DR₅=80%) requires an odds ratio for one standard deviation of 12 or higher (compared with the median observed value of 1.31) or an area under the receiver operating characteristic curve of 0.96 (compared with the median observed value of 0.65). Only 11.4% of the area under the curve values in the Polygenic Score Catalog exceeded 0.8, which equates to a DR₅ of 32%, with most of these resulting from large effect variants at the HLA locus in a few autoimmune diseases (figure 1 and online supplemental file 1).

Study implications

When a risk factor has a monotonic relation with risk of disease,32 more instances arise among the majority with near average risk factor values than among the few with more extreme values, termed the prevention paradox.33 34 In this respect, polygenic risk scores are similar to some non-genetic risk factors, such as blood pressure and low density lipoprotein cholesterol, which although causal, are poor predictors of coronary artery disease.16 35 That the performance of polygenic risk scores in the prediction of coronary artery disease is sometimes compared favourably with that of blood pressure and cholesterol26 is to benchmark one poor predictor against another.

Where safe and inexpensive preventive interventions are available (eg, statins and blood pressure lowering drugs for prevention of coronary artery disease and stroke), broadening rather than limiting eligibility for such interventions gives greater public health benefits.36 Prevention of coronary artery disease and stroke has been achieved in effect by the progressive lowering of the 10 year risk cut-off value for prescription of statins in primary prevention. The cut-off value was reduced from a 10 year risk of coronary artery disease in the UK in 1997 of 30%,37 to 10% for the 10 year risk of coronary artery disease or stroke in the UK from 201627 and to 7.5% in the US from 2019.38 The reduction in the risk cut-off value resulted from reduced drug acquisition costs through patent expiry, and by accumulating evidence on long term safety. Eligibility could be extended even further and simplified by using age alone to guide prescription of statins for primary prevention, preventing coronary artery disease and stroke in many more patients.28 In contrast, retaining the same 10 year risk cut-off value and adding information on polygenic risk score to conventional risk factor models has a much weaker effect. Based on recently reported data,5 26 we showed that several thousand individuals need to be genotyped and a polygenic risk score calculated to prevent one additional vascular event.

Identifying a minority of individuals at very high risk (with genetics or other means) might be justified if a preventive intervention is costly, resource limited, or has substantial harms.39 With breast cancer as an example, however, we showed that identifying those at high risk requires testing in all and, apart from missing the many more patients among those at average risk, generates many false positive results. This finding could have substantial downstream resource implications for healthcare systems if, for example, genetic risk stratification was followed by a confirmatory screening test, such as mammography for breast cancer.40 In this case, reducing the age cut-off value for mammography for all women without determining their polygenic risk score might be more sensible.

The enthusiasm surrounding polygenic risk scores might have been encouraged by pressure on academia to demonstrate a tangible health effect after decades of research investment in human genomics and by commercial opportunity. Unrealistic expectations have probably been raised by use of uninformative metrics. Publications on polygenic risk scores often illustrate comparisons between mutually exclusive groups (eg, those in opposite ends of a polygenic score distribution).41 This finding is relevant in aetiological studies but is not relevant in screening. Figure 3 shows seemingly impressive odds ratios of 13, 7, 5, 4, and 3 for comparisons of the top versus the bottom 1%, 5%, 10%, 20%, and 25%, respectively, of the polygenic risk score distribution for coronary artery disease, all reduced to a DR₅ of only 12%. What is relevant in screening is the risk of an event in a group compared with that of the whole population, which is achieved with the calculation of the detection rate for a specified false positive rate.

Policy implications

Our findings are relevant to commercial providers of genetic tests and to researchers working on polygenic risk scores. Commercial providers could communicate individual test results to customers with greater clarity and relevance to performance in disease prediction; for example, by presenting the overlapping distributions of polygenic risk scores among those later affected and unaffected and by presenting an absolute measure of risk for an individual or group, which requires additional information on population average risk at a particular age over a specified time. At the same time, as already suggested,42 policy makers might wish to consider stricter regulation of commercial genetic tests based on polygenic risk scores, with a focus on clinical performance and not just assay performance (as indicated by the Royal Statistical Society Diagnostic Tests Working Group Report43), to protect the public from unrealistic expectations and already stretched public health systems from becoming overburdened by the management of false positive results. Researchers reporting studies on polygenic risk scores should present as a minimum: mean and standard deviation values for polygenic risk scores among later affected and unaffected individuals; overlap in their distributions; relevant performance metrics, such as the detection rate for a specified false positive rate (eg, DR₅), avoiding the need to make this calculation indirectly23; and performance of polygenic risk scores with and without the inclusion of other variables so that users can judge the incremental benefit provided by the polygenic risk score itself.

Although our analysis showed the poor performance of polygenic risk scores in screening, prediction, and risk stratification, these scores might be useful in other situations. For example, polygenic scores might explain the variable penetrance of rare mutations in monogenic diseases (eg, hypertrophic cardiomyopathy or familial hypercholesterolaemia), and be used to help detect patients. Other predictive applications of genotyping also exist, for example in pharmacogenetic testing to optimise the efficacy and safety of medicines. Genotyping might also be of value in blood and tissue matching. Because genetic variation is transmitted from parents to offspring through a randomised process (like treatment allocation in a clinical trial), and is unaltered by disease, an important translational application arising from genomic discoveries could be providing evidence on disease causation and targets for pharmaceutical intervention.44

Conclusion

Use of the appropriate metrics showed poor performance of polygenic risk scores in population screening, individual risk prediction, and population risk stratification. The wide scope and analytical approach of our study might help to resolve the debate on the value of polygenic risk scores, and avoid unjustified expectations about their role in the prediction and prevention of disease.