Evaluation of evidence supporting NICE recommendations to change people's lifestyle in clinical practice: cross sectional survey

Study design, definitions, and eligibility criteria

In this cross sectional survey, we systematically reviewed all recommendations of IOLIs provided in NICE guidelines. An IOLI was defined as any non-drug intervention provided by healthcare personnel to any adult (over age 18 years) with the aim to improve health by changing one or more of the following lifestyle habits: physical activity or exercise, diet, tobacco use, and alcohol use. Examples of such interventions are motivational interviewing to promote physical activity or healthy diet; prescriptions for physical exercise; enrolment in weight loss or diet programmes; or brief advice from healthcare personnel on diet, exercise or physical activity, tobacco cessation, or alcohol use. Interventions targeting specific symptoms (eg, exercises to treat low back pain or urinary incontinence, dietary advice to reduce gallbladder attacks, treatment of substance abuse disorders) or those delivered outside clinical settings (eg, workplace) were not eligible.

Data sources and selection process

We systematically searched NICE Pathways, an online resource, on 26 August 2020 to identify all IOLI recommendations using the following search string: lifestyle OR physical activity OR exercise OR diet OR smoking OR alcohol OR tobacco. We searched NICE Pathways to optimise the sensitivity to detect IOLI recommendations in the guidelines. One author (MJ) screened all NICE Pathways for any candidate IOLI recommendations, while a second author (MR) verified these judgments. Two authors (MJ and MR) then independently, and in duplicate, assessed each guideline for eligibility. Any disagreements were resolved by discussion with a third author (LA). Search results and reasons for exclusions are available in online supplemental appendix 1,13 and a PRISMA flowchart is available in online supplemental appendix 3.

Data extraction and analysis

Two authors (MJ and LA), working in duplicate and independently, extracted data from included guidelines into a standardised and prepiloted data extraction form. Disagreements were solved through discussion with a third author and consensus. We estimated interobserver agreement using raw agreement and chance adjusted agreement using the phi (φ) statistic.14 Because the agreement levels were very high (>92% for all judgments, mean 96%), we report only the raw agreement levels, which are easier to interpret.

The full data extraction form, including all extracted data, is available in online supplemental appendix 2.13 The information extracted included (but was not limited to) the targeted lifestyle habit; the targeted population; the nature of the cited evidence to support the recommendation (see definition below); the reported certainty of the evidence to support the recommendation; whether psychosocial harms, physical harms, treatment burden, and opportunity costs were considered in any way; and whether the specific aspects related to opportunity costs were considered or reported. These aspects included the time needed to perform the intervention per individual, the relevant category of healthcare personnel, the eligible fraction of the population, the human resources needed to fully implement the intervention, the financial resources needed to fully implement the intervention, and a potential for redistribution of resources from other interventions or from vulnerable to privileged groups.

We planned to rely on the judgments made by the guideline authors about the certainty of the evidence (ie, GRADE—grading of recommendations assessment, development, and evaluation).15 However, when piloting the data extraction form, we found that the certainty of the evidence was often either not assessed or not reported, unclear, or inconsistently reported; and that the relevance of the evidence for the specific recommendation was often unclear (eg, the certainty was assessed for studies investigating the effect on biochemical markers for participants who changed behaviour—while the recommendation was advice to change behaviour). We concluded that to only extract the certainty of the evidence as reported in the guidelines would provide insufficient information.

We considered performing a GRADE assessment for each recommendation but found that the necessary information was often not available in the guidelines. We therefore chose instead to include the following in our data extraction (referred to as the nature of the cited evidence to support the recommendation): whether no studies were cited to support the recommendation; whether the only cited studies were observational studies; whether the cited studies differed importantly from the recommendation in terms of the intervention, the population or the outcome; and whether the cited studies investigated the recommended intervention but suggested only a short term beneficial effect or no beneficial effect. The choice of these categories was based on the findings during piloting, as well as existing knowledge within the author team. We judged the cited evidence to be of questionable relevance if it referred only to observational studies of correlation between a lifestyle habit and worse outcomes; studies suggesting no beneficial effect of the recommended intervention; or evidence that was not directly relevant to either the recommendation’s target population, the intervention, or the intended outcome (see examples in the results section below).

We followed the guidance provided by GRADE16 to judge whether an outcome was a surrogate or a patient relevant outcome. Of note, we classified weight loss, smoking cessation rates, and increases in physical activity to be surrogate outcomes (see limitations section in the discussion section below).

Regarding psychosocial harms, physical harms, and treatment burden, we judged these harms to be considered if they were mentioned in any way in the guideline. For example, if guideline authors reported that no adverse effects were reported in the cited studies, we would judge that this guideline considered harms. Further, for all guidelines who considered harms beyond stating that the cited studies did not report any harms, we extracted how harms were considered and list the exact quotes in online supplemental appendix 3.

Descriptive analyses were used to summarise the results. Online supplemental appendix 2 presents all extracted data.13 The included NICE evidence syntheses supporting the recommendations are stored as pdf documents and are available on request.

Patient and public involvement

A patient advocate (HB) has been involved in the design, conduct, reporting, and dissemination plans of this research as an author.

Characteristics of IOLI recommendations

Of the 379 included IOLIs, 112 (30%) were on tobacco cessation, 74 (20%) on diet, 60 (16%) on physical activity or exercise, 21 (6%) on alcohol use, and 111 (29%) recommended multiple lifestyle interventions (eg, both diet and physical activity). Almost all (n=374; 99%) recommended the lifestyle intervention and five (1%) recommended against it. Eight (2%) recommended the intervention only when judged appropriate by healthcare personnel, while 366 (98%) recommended the intervention for all applicable patients (eg, dietary advice for all people with a body mass index over 30).

Evidence underpinning the recommendations

Table 1 and figure 1 describes the evidence cited in support of the IOLI recommendations. Inter-reviewer agreement was >92% for all judgments (mean 96%). Of the 379 recommendations, 123 (32%) did not cite any supportive evidence. For 181 (48%) recommendations, the evidence cited was judged to be of questionable relevance to support a beneficial effect. Of these recommendations, 71 (19%) referred only to evidence from observational studies (eg, people with obesity have worse outcomes than people without obesity); 61 (16%) referred to evidence that was not directly relevant to either the recommendation’s target population (eg, population of the research evidence was pregnant women only while the target population of the recommendation was all women of fertile age) or to the intervention (eg, research evidence was on intensive and complex interventions while the intervention in the recommendation was brief advice), or the cited evidence had limited relevance to support that the recommended intervention will lead to the intended outcome (eg, evidence suggested weight loss leads to beneficial outcomes while the recommendation assumes dietary advice would cause weight loss); and 49 (13%) cited evidence suggesting no beneficial effect of the recommended intervention but recommended it anyway (eg, studies of brief advice on physical activity did not show a clinically relevant effect on the measured outcomes).

Request permission

Of the 379 recommendations, 75 (20%) referred to research evidence suggesting a beneficial effect of the recommended lifestyle intervention. Of these 75 recommendations, 74 (20% of total) reported a beneficial effect on surrogate outcomes, and 14 (4% of total) reported a beneficial effect on patient relevant outcomes (table 1). Of the 74 recommendations who reported an effect on surrogate outcomes, 13 reported a beneficial effect based on high or moderate certainty evidence (3% of total), of which seven became non-significant with long term follow-up. Of the 14 recommendations who reported an effect on patient relevant outcomes, seven reported a beneficial effect based on high or moderate certainty evidence (2% of total), of which three became non-significant with long term follow-up.

Consideration of harms for the individual

Possible psychosocial harms from the intervention were considered in 19 (5%) recommendations, of which 13 did so only by including quality of life as an outcome (ie, harms were not specifically considered); of these, 10 did not find any studies reporting on quality of life. Possible physical harms were considered in 32 (8%) recommendations, of which 31 did so only by stating that no adverse events were reported in the cited evidence. None of the 379 recommendations considered the treatment burden placed on the patient because of the recommendation. Altogether, three recommendations considered any harm to the individual beyond what was reported in the cited studies. All of these recommendations are listed in online supplemental appendix 3.

Consideration of clinical opportunity costs

For one of the 379 recommendations, the opportunity costs for the healthcare system of implementing IOLIs was considered (online supplemental appendix 3). None of the recommendations considered redistribution of human and financial resources from other interventions with greater beneficial effect, or from vulnerable to privileged groups. Likewise, none of the recommendations presented estimates of the proportion of the population for which the recommendation would be applicable, or the human or financial resources that would be needed if the intervention were to be fully implemented in eligible individuals.

Meaning of the study

The findings of our study do not contest the beneficial effects of healthy lifestyle habits. On the contrary, high certainty evidence support the association between a healthy lifestyle and improved health. However, evidence for beneficial effects of healthy habits should not be confused with evidence for beneficial effects of lifestyle interventions. In other words, just because losing weight is beneficial, it does not mean that advising people to lose weight will also be beneficial because the advice may not help people change behaviour.

Further, our findings do not rule out that IOLIs have beneficial effects, merely that the evidence cited to support recommendations of lifestyle interventions made by one of the most respected guideline institutions in the world is very limited. It could be argued that the need for high certainty evidence to support recommendations of IOLIs is lower than for drug or surgical interventions, since the risk of physical harms is likely to be lower and since the evidence to support beneficial effects of healthy lifestyle habits is uncontested. Yet, other factors should be considered.

At the individual level, to target individuals with unhealthy habits might cause harm.11 12 People with unhealthy lifestyles should arguably be able to see their clinician for a specific health concern without receiving unrequested and unrelated lifestyle advice. The patient’s agenda might not be fully considered in encounters in which the healthcare professional must deliver IOLIs, detracting time from patient centred care. IOLIs can place a substantial burden on the individual and their caregivers, particularly among those who are eligible for multiple interventions and already carry a substantial burden of treatment related to chronic multimorbidity.12 17

At the healthcare system level, the time, energy, and attention needed for healthcare personnel to implement IOLI recommendations must be carefully considered given their existing workload, available resources, and all the other tasks assigned to them. Given the large fraction of general populations around the world that have unhealthy lifestyle habits, and that IOLIs often require considerable clinician time, IOLI recommendations have potential to cause substantial opportunity costs within health systems. At the societal level, the observed global increase in costs of healthcare services draws resources from other societal sectors capable of improving the health and wellbeing of the population,18 19 a situation currently augmented by the covid-19 pandemic. A redistribution of resources from societal interventions to individually oriented interventions within the healthcare system could counteract a rational use of community resources.19 20

Given these concerns, it seems reasonable to expect IOLI recommendations to be supported by evidence that the interventions at least succeed in achieving the change in behaviour on which the health goal is predicated. Yet, behaviour change has generally been shown to be very difficult to achieve for the individual, particularly when living under the influence of political, corporate, and socioeconomic determinants of health.

An additional ad hoc finding of our study was that the rigour by which the beneficial effects of lifestyle interventions was evaluated was inconsistent; in many cases, the cited studies did not investigate the effect of the recommended intervention. The same methodological rigour might currently not be applied to recommendations on lifestyle interventions as for recommendations on other medical interventions. While a limited evidence base is not the fault of the guideline panels, we believe that these findings highlight a need for a closer adherence of guideline panels to frameworks for how to interpret and evaluate evidence when issuing recommendations on IOLIs.21

Based on our findings, we have identified some issues that could help bring the presented aspects into sharper relief: (1) low certainty evidence of potential short term beneficial effects on surrogate outcomes were sometimes assumed to automatically translate into long term beneficial effects on patient relevant outcomes; (2) evidence of a correlation between a lifestyle habit and better outcomes, or evidence of a beneficial effect from behaviour change, were sometimes interpreted as direct evidence that advice on lifestyle would have a beneficial effect (ie, it was assumed that the intervention will make people change behaviour despite no evidence to support this); (3) healthcare personnel’s time was considered (at best) as a financial cost only and not as a finite resource to be carefully prioritised; (4) each recommendation was evaluated separately without considering the implications of implementing all recommendations combined; (5) the resource use required for the recommended intervention was (at best) considered in relation to interventions within healthcare only, and not in relation to other types of societal interventions aimed at improving health and wellbeing of the population.

Limitations and strengths of the study

Our results rely on subjective judgments about the quality and scope of the underlying evidence. Although the inter-reviewer agreement was high (between 92% and 100%, mean 96%; table 1), using duplicate assessors, discussion, and consensus might not have fully controlled for investigator biases. In general, we sought to err on making favourable judgments. For example, we classified a recommendation as considering psychosocial harm when its text simply mentioned that no studies reported on the IOLI’s effect on quality of life.

The variable degree of explicitness of reporting in the guidelines, however, further complicated their evaluation. For example, the relevance of the cited evidence (as well as the GRADE assessments, when available) to evaluate a beneficial effect of the recommended intervention was often questionable. We therefore developed post hoc categories to evaluate the relevance of the cited evidence more fully. Furthermore, because many of the guidelines did not use an explicit method for evidence rating, our judgments could have underestimated the proportion of recommendations that were based on moderate to high certainty evidence. Since only 4% of the recommendations were underpinned by any evidence of a beneficial effect from the recommended intervention on patient relevant outcomes, this prospect seems unlikely.

We classified weight loss, smoking cessation rates, and increases in physical activity as surrogate outcomes. These outcomes themselves can be argued as patient relevant outcomes (ie, outcomes that patients can experience and value). Yet, even if we were to reclassify them, our conclusions remain unchanged because high to moderate certainty evidence of a beneficial effect on any surrogate outcomes underpinned only 3% of the recommendations.

The central strength of our review was in the complete sample of recommendations from a well respected guideline authority, transparency and adherence to prespecified methods detailed in a registered protocol, and full reporting of all raw data extracted (online supplemental appendices 1–3). Future analyses of the recommendations from other guideline authorities can help determine how our findings can apply to other settings. The high quality of the process followed by NICE to formulate recommendations, however, suggests that our results might represent a best case scenario.

Conclusions

We found that recommendations of individually oriented lifestyle interventions in state-of-the-art guidelines were rarely supported by evidence of a beneficial effect, and that harms and burdens at the individual level, and opportunity costs at the healthcare system level, were not explicitly considered. The findings of our study do not contest the beneficial effects of healthy lifestyle habits. However, guidelines recommending clinicians to try to change people’s lifestyle need to be reconsidered given the substantial uncertainty about the effectiveness, harms, and opportunity costs of such interventions.