Annals of Public Health Reports

Patient Perspectives on the Acceptability of Emergency Admission Risk Prediction: A Focus Group Study

Abstract

Background: Emergency admission risk stratification (EARS) tools predict admission risk for a general practice patient population. Policy has encouraged targeting higher risk patients with proactive care approaches, in partnership with patients. Previously published stakeholder views on the acceptability and use of EARS tools have been limited to professionals.

Objective: Our objective was to explore the views of patients on acceptability, benefits, challenges and risks of communicating emergency admission risk scores to patients.

Design, setting and participants: We undertook an in-person focus group with a geographically diverse group of patient and public members in Wales, UK. Participants brought experience of multiple health providers. All had chronic conditions and/or recent experience of emergency admission to hospital. We coded and thematically analyzed the transcript.

Results: Participants supported the use of EARS where it was underpinned by communication of risk scores and direct involvement of patients. Participants expressed a desire to receive their own risk scores. They felt EARS use was well suited to holistic approaches to care, and as a stimulus to self-management.

They recognized capacity and cost challenges related to general practitioner use but saw potential for other primary and community staff to be involved. The security and integrity of EARS data was deemed important.

Conclusions: This study provides a rare insight from a stakeholder group that has largely been excluded from debates around EARS use. These findings identify issues that are deserving of further exploration to improve our understanding of the potential role and effectiveness of EARS and other risk approaches in healthcare.

Patient or public contribution: This study was conceived following discussion with public contributors to a trial of emergency admission risk stratification (PRISMATIC) [1], and a subsequent informal workshop with eight members of a patient, carer and public member group aligned to research around chronic conditions management in Wales, UK [2]. This preliminary work emphasized the importance of patient involvement in debates around EARS and contributed to the study reported here and to the information sheet and topic guide. JD, a public contributor, was a co-applicant on PRISMATIC, took part in a preliminary workshop, and is a co-author of this paper.

Keywords

Primary health care, Health services research, Emergency health services, Risk stratification, Clinical prediction rule, Chronic disease

Introduction

Emergency admission risk stratification (EARS) tools identify patients at high risk of hospitalization for targeted care. They have been strongly advocated in the UK and internationally [3,4] as a mechanism to support efforts to reduce emergency admissions and as a mechanism for patient-centered care that takes a holistic rather than disease specific approach [5,6].

EARS tools use clinical and demographic data (patient’s records) to calculate and stratify risk of emergency admission for patients within a population, typically in primary care. Most risk tools produce individualized patient scores that correspond to the likelihood of an emergency admission within one year. EARS tools have been widely implemented in the UK, with use recommended in NICE guidance [7], and in GP contracts, notably the £480 million unplanned admission enhanced service in England [8]. New tools continue to be introduced [9].

Despite the background and rhetoric around patient centredness, patient and public views on EARS are unknown. Research is dominated by development and validation studies of tool performance [10,11]. Qualitative studies with stakeholders other than patients also feature including interviews with general practitioners, allied health professionals, commissioners, providers, and managers [12-16]. These studies indicate support for EARS in principle but highlight challenges to implementation around (inter alia) integration with clinical records, capacity in primary and community care, and (a lack of) guidance over risk discussions with patients [17-20]. It is notable that patient/clinician discussion of EARS risk is infrequently considered in the literature. Opportunities may be lost to involve patients in their care decisions, and to fully realize their potential as a resource in self-management of their health and wellbeing. A fundamental knowledge gap is whether EARS (and use of patient data for individualized risk scoring) is acceptable to patients. Acceptability to providers and recipients is considered a necessary condition for intervention effectiveness [21-23], increasing the likelihood of patient engagement, intervention fidelity, treatment adherence and improved clinical outcomes [24-26].

How do patients feel about emergency admission risk scores? How might they react? Do they want to know their own risk scores? These are all important questions relating to EARS and add to the compelling argument for closer scrutiny, That this study begins to address.

Our aim was to explore patient views about acceptability, benefits, challenges and risks of communicating information on predicted emergency admission risk, during patient and clinician encounters.

Method

Design and setting

We employed a qualitative study design, undertaking a focus group with patient and public individuals to explore their views. The group was held in Cardiff, Wales, UK in November 2019. We convened a focus group (rather than interviews) because they provide a forum that supports participants to share experiences and opinions, to react and respond to others; potentially offering views or comments that might not surface during individual interviews [27,28]. These dynamics also make focus groups well suited to exploratory work and idea generation, and to exploring beliefs about health and disease [29].

This paper accords with the consolidated criteria for reporting qualitative research (COREQ) guidance [30].

Participants and sampling

Participants were sampled opportunistically on a theoretical sampling basis-where the aim was not to achieve a representative sample but to choose cases that could contribute to theory generation [31]. Participants were members of an existing public involvement group (SUPER - Service Users for Primary and Emergency care Research) [32] which brings together representatives from across Wales and is not linked to any particular care service. This was appropriate in the context of exploring the issues from a general rather than local or service specific perspective.

To facilitate involvement, the focus group was conducted within a scheduled SUPER meeting, and in the same location (a hotel). All meeting participants (n = 22) were invited through email and provided with an information sheet and consent form in advance. Eligible participants were those with one or more chronic conditions or recent experience of an emergency hospital admission. This ensured a degree of familiarity with the emergency admission concept, and discussions around management of chronic conditions, multi morbidity and admission risk. The aim was to include six to ten participants to achieve a judicious balance between information discussion and manageability [27,33]. Participation was voluntary. As per good practice an honorarium was provided to cover participants’ time and costs of attending [34].

Data collection

The focus group was facilitated by MK, an experienced qualitative researcher, using a semi-structured topic guide (Box 1). The discussion lasted 56 minutes. At the start of the focus group MK provided an overview of EARS tools, and how they could be used to support patient care as outlined in NHS documentation [3,4,8]. The focus group was digitally recorded and transcribed verbatim by an external agent. Comments were anonymised within the transcription, with each participant assigned a code based on gender and an incremental number e.g., M3.

Box 1: Focus group semi-structured topic guide

Analysis

We used an inductive approach and analysed the transcript thematically using printouts and NVivo (version12) software. We followed the six stages of thematic analysis as described by Braun and Clarke, which include familiarization with the data, coding, generating themes and interpretation [35]. MK led analysis with AP supporting key stages of coding, generating themes and interpretation, thus encouraging a critical stance to test and confirm findings [36]. Quotations are used for illustration, to reinforce themes and to maintain authenticity of patient voice.

Results

Nine participants took part, with most residents in Wales- (Table 1). The group demographics were consistent with the wider SUPER group profile [32].

The results are presented in five themes as determined from the data:

a. Attitudes to EARS use in patient care.

b. Professional users of EARS.

c. Alignment with health and wellbeing.

d. Role of EARS in patient/clinician interaction.

e. Data quality and security.

a) Attitudes to EARS use in patient care

Participants recognized potential value in the use of EARS in primary care, where EARS information was shared with patients and used to stimulate discussion or action. When asked, all participants expressed a wish to know their own risk score; not merely out of curiosity but as a stimulus to positive behaviour change and risk modification. The group debated who might be best placed to benefit from risk scores. A repeated concern was that it may not be practical or effective to target those at the very highest risk: Those at lower risk may have more modifiable risk factors. One participant described the focus on the top of the pyramid [as used in much of the GP contracts] as ‘simplistic’, as in the (highest) risk level may be ‘a lot of elderly, frail people with comorbidities’ (M2) -for which preventative work may not realize benefits. Members of the group proposed an approach that focused on those at lower risk- to prevent downstream issues and ameliorate risk. The following passage further illustrates this emphasis on engaging with patients who could benefit most from application of EARS score:

M2: The GP could do more in actually telling people in advance [of reaching top risk level] rather than just pulling in the top cohort. If you’re in the top cohort, you’re a bit late to do changes.

F5: Because changing your life style takes time, doesn’t it?

The extract is also noteworthy because it emphasizes two further observations consistent across the transcript. Firstly, that participant consistently valued risk scores as new information that could potentially support patient self-management of health. Secondly, the language used was indicative of a positioning of the patient as owner of their health and as the active actor in modifying relevant lifestyle behaviours. As per the extract however, realizing this role would rely on patients being informed of their risk.

Age was a recurring concern for participants, particularly in relation to the appropriateness of patient identification and intervention strategies based on age. While some participants saw value in targeting younger patients, others cautioned over care that might discriminate purely on age.

b) Professional users of EARS

There was debate about who might be the most appropriate clinicians to use and communicate risk score information. This included discussion of the potential use of EARS within annual reviews undertaken by practice nurses (e.g., for asthma patients or those over a certain age). Although there was support for this approach, some participants reported that the provision of such reviews was variable. Another participant (F5) wondered if a ‘care navigator’ position might be best suited–someone who knows patients and the services that they might have access to. There was awareness however that asking GPs to use the risk tools and follow up with patients may be challenging in the context of high demand for their services, and their cost:

GPs are under so much pressure, they’re not using this information in a proactive manner, in the sense of how often are they running the reports? And x percentage of their patients, we need to invite them in. Who’s ever been invited in by their GP? Generally, it doesnot happen, it’s patient driven as opposed to GP driven. (M3)

And do you need a sophisticated GP, expensive GP?…practice nurses all have degrees these days. They can interpret and help you analyze the information. It doesn’t have to be the GP. (F1)

c) Alignment with health and well being

A recurring theme from the group was that EARS tool use should align with a broad perspective of patient’s health and wellbeing -in line with the general and non-disease- specific risk (of emergency admission) that tools identify. The group expressed concern that the risk models ‘seem very focused on medical factors’ (M4) and were keen for non-medical factors to be part of the consideration in understanding individual patient risk. Factors such ‘as mental illness’, ‘housing conditions’, ‘loneliness’, and ‘alcohol’ were cited (M5 and M15). Although the group recognized that these may not be routinely captured in many current risk models there was a hope that practitioners would use their additional understanding of patients, to combine risk data with their own clinical and social insight for a more nuanced determination of which patients to follow up with.

A GP and staff might well take those into account, even if they’re not in the [EARS] system. (M8)

But whilst the health centre is primarily promoted as being the medical place that you go to, the predictability tool there has got huge benefits but the answers to the issues that it comes up with will not always be medical. (M3)

However, as one participated reflected, a broad model of risk may present a challenge in determining the most appropriate, or any, risk mitigation:

What are you at risk off? Because if they’re frail and they’re at risk of falling, is there something that can be done by the GP practice to prevent that from happening.

Where as if they’re a smoker and they’re at risk of having an asthma attack because they smoke and they have asthma, then that’s something that perhaps they can tackle more effectively. (F6)

The importance of non-medical factors to the patient group was not limited to patient selection but applied equally to potential risk mitigation. To illustrate this view, one participant drew on an analogy from a service for regular 999 callers which had found that “for a lot of these people, the services they need are not in emergency departments” (M4). In relation to community-based intervention there were suggestions that EARS use would fit well where there was a focus on ‘wellbeing’ (M2) or ‘social prescribing’ (F5). But this approach was not deemed typical in current practice:

If we moved to a broader concept of wellbeing centres, which actually start to de-medicalise some aspects of people’s lives-because currently, the expectation is most people going to their GPs, they’re going to walk out with a prescription, and the reality is they perhaps don’t need a prescription, they need something else…the patient’s expectation will always be of a medical intervention. GPs don’t get paid to de- medicalise patients. (M4)

d) Role of EARS in patient/clinician interactions

Most participants reported no direct experience of risk tools, and no one reported that they knew of an EARS tool in use in their general practices. They were, however, familiar with the concept of risk and recognized that GPs regularly made decisions or recommendations based on a balance of risk and benefits, albeit that this was not always enunciated:

I have many GP consultations about all sorts of things, and I think that he’s always got risk in the back of his mind, but he doesn’t ever say it. (F1)

There was also a view that GPs were less likely, and less willing to discuss risk with patients than some other professionals, perhaps due to the complexity of risk discussions.

I find - my personal experience of GPs is, when I ask them about risk levels, they prescribe a drug for me for a condition. They’re extremely reluctant to talk about risk. They won’t really talk about it. Hospital surgeons, for example, will give me figures without asking. So, my personal experience is GPs don’t like talking about risk levels. It’s a very difficult are a anyway, and you can’t necessarily give a straight answer. (M8)

Two participants confirmed experience of disease-specific risk scores in primary care - for colon cancer and heart disease respectively. In discussing the former, there was a view that cancer risk was a regular consideration for clinicians and would likely lead to predetermined pathways of care.

It’s all laid out by the NICE recommendations, and if you fit certain criteria then you’re straight off on the urgent cancer pathway. (F5)

From the recipient of the heart disease risk, there was a sense of frustration with the response to their assessment-which resulted in a prescription-but not further discussion or information:

Well, actually, in the next three to five years, you could have a heart attack,” and, you know, you could actually die from one heart attack, bang, you know. What would have been very useful, I think, is sharing with them some information to actually look-when you’ve gone away from the surgery, to make sure you can consider what the kind of options would be, as opposed to it being, “Well, there’s the conversation, here’s your prescription. Thank you very much, off you go.” (M4)

The examples prompted further reflection on the use of risk scores, and other participants expressed concern over meaningful discussion of the next steps for care following a risk assessment. Participants were keen that risk scores should not be used bluntly nor the scope of care limited to medical intervention. They also wanted patients to be involved in the decision making about their care.

…a bit like [participant M4], you know, the response to that was, “Oh, you need to go on statins,” as opposed to, “Well, actually, are there any life style changes that you can do, you know, weight loss or other factors?” There was an immediate medical response to a statistic, as opposed to, “Well, actually, are there any other interventions?” (M3)

I don’t want somebody to do a risk assessment and then bully, force, whatever me into pursuing a certain course of action. I mean, if I don’t want to take statins or not have a flu jab. (M8)

When asked (at the end of the discussion) if they would want to know their own risk score, the group were unanimous that they would -although this was tempered a little with one view that “unless something’s going to be done with it, I don’t think there is much point” (F1). In terms of intervening, the participants emphasized the patient role in any follow-on care management- with patients able to decide to respond (or not) to the information and or advice shared with them.

F6: It’s great to have somebody do something about it, if they’re going to give me that and say - that means I’ll do something about it.

M3: Or I won’t do something about it [all laugh].

F6: But it’s a decision not to then, Isn’t it?

F7: Yeah, I would suggest it empowers you to do something about your life style to a certain extent.

In relation to the potential communication approaches to sharing EARS risk with patients, one participant noted that the PRISM risk tool (of which screenshots were shared) was intended for GPs to access, and by implication, may not be appropriate to share with patients.

Some of the terminologies are not exactly how you would like your GP to regard mental health problems…So, that’s one other factor, if you’re going to say to people, “Would you like to see your record?”. (F5)

e) Data quality and security

Several participants had reservations over the use of routine data, chiefly regarding accuracy, access and security. In respect of the first of these, participants understood that accuracy was an important attribute of the risk models, but felt that issues may emerge given the scale and complexity of the data, and the absence of patient scrutiny:

…in all the millions of records, there must be errors, and it’s making sure errors don’t cause problems. (M8)

And if you can’t check it, you can’t challenge what’s there about you. I mean, I’m thinking, I’ve got an electronic patient record, and I haven’t got any of this [in the record]. (F6)

None of the group were previously aware of the use of EARS tools in primary care, and they were surprised (with audible gasps) about the scale and scope of EARS tool roll out and the costs of the unplanned admissions enhanced service in England. This prompted discussion over ownership, patient engagement, and the use of data without patient awareness.

M3: I think it’s my data and I should know what’s being done with it, and that is my first principle.

M8: I think there’s a scandal that there isn’t more known about this publicly, because this is the first I’ve heard about it…

M3: It’s like it’s secret almost.

There was however also recognition that there are aspects of data used in the NHS that patients would not routinely be aware of - rightly or wrongly.

I think that the GP contracts contain all sorts of information that we’d be interested in, but we don’t look at them, see them, aren’t aware of. (M6)

The concern for wider awareness was also acknowledged in relation to data flows, given expectations that some private companies may be involved in supporting risk stratification (e.g. as suppliers of risk tools).

It’s within the NHS, but they subcontract to a private company, and that’s fine…But we need to know what’s happening. (F5)

While there were no concerns raised over GP practice staff having access to the risk score and data, reservations were expressed about the potential access by commercial organizations such as pharmaceutical or insurance companies.

Discussion

Summary

This study identified and articulated a range of patient/public views on EARS. In doing so it provides a rare insight from a stakeholder group that has largely been excluded from debates around EARS. The peculiarity of that exclusion has been reinforced by the passion and enthusiasm with which the participants engaged with this topic.

Overall, the participants were cautiously positive about EARS and its potential role in primary care. Strikingly, all wanted to know their own risk score. They saw emergency admission risk information as a useful overview of patient health, providing an additional, personalized perspective. Through risk stratification and discussion of personal risk, the group saw potential for subsequent risk mitigation activity, with patients playing a major role. This perspective was founded on two key principles. Firstly, That patients were made aware of their risk score, facilitating patient involvement in decisions on next steps. Secondly that the generalized, non-condition specific risk of admission may need a general (or holistic) response, not limited to medical and drug interventions but considerate of social or lifestyle factors too. There were few specific concerns over the principles of risk stratification for emergency admissions, but data safeguarding was important. It may well be that the use of EARS tools is acceptable to most patients - but further exploration is needed.

Some respondents had experience of other risk scores/tools in their care but cautioned that subsequent care decisions were often applied as a fait accompli and lacked meaningful patient/clinician dialogue. This served as a warning for the use of EARS, which participants saw as a nuanced measure, requiring reflection and discussion to inform care. The groups were cognisant of pressures in primary care and recognized that discussions of EARS risk and shared decision making may not be straightforward or swift. They felt that GPs may struggle to implement models of care based on proactive patient identification. Such approaches in primary care were not a familiar aspect of the group’s experiences of primary care - hence the rhetorical question of ‘Who’s ever been invited in by their GP?’ In this context, there was a general sense that other professionals may have a role in supporting those identified following risk tool use.

Participants noted some limitations of the variables included in EARS models, particularly regarding the absence of non-clinical factors e.g., around social determinants of health. Participants felt the question of who would benefit most from intervention was important - and linked potential answers with interventions addressing social and life style aspects and patient self-management.

Strengths and limitations

The primary strength of this study is in providing patient & public insights into a topic dominated by numerical analyses and the experience and views of managers, clinicians, commissioners, and other stakeholders. The study complements these perspectives and the evidence around EARS. The chief limitation of the study is its scale, with one focus group and nine participants. However, this was an exploratory study conducted to generate theory and identify issues of importance, rather than conclusive results [31,37].

A potential limitation of the sampling frame was that participants were members of a pre-existing primary and emergency care research group (SUPER) [32]. They were therefore likely to be more health and research literate than a general patient population. However, they were also varied, in terms of age, gender, clinical history, and all brought experience of chronic conditions and/or emergency admissions. They lived in different general practice areas with diverse health and wellbeing experiences, and thus brought multiple perspectives of care. Some participants knew each other, which helped support a lively discussion.

Comparison with literature

While this study covers new ground in providing patient insights into EARS use in primary care, some of these views resonate with those of stakeholders reported elsewhere. This was true of the comments that point to the use of a more nuanced approach to identifying patients than use of a risk tool alone i.e., supplementing the data with the knowledge and insight of clinicians. For example, a qualitative study with German physicians undertaking case management for patients identified from EARS (Freund, et al., 2012) found that the clinicians reported ‘care sensitivity’ -patient’s willingness and ability to participate- as a key determinant of selection for the intervention [13]. Similarly, a study of managers and clinicians in an integrated care multi-disciplinary team (MDT) initiative, found movement away from a ‘formal data-driven approach’ to a more ‘fluid, active process of selection via iteration and consensus’ - to target those patients with ‘unmet and actionable needs, rather than simply quota filling’ [16]. The authors advocate involvement in the MDT of those with the capability to address these needs (although made no mention of patients). The same study also provided clinician views on the most appropriate patients to target. These views were consistent with the patient feedback here - advocating a focus on patients of moderately high (rather than highest) risk and with social rather than medical complexities [16]. Those findings overlap with a further qualitative study of primary care clinicians and service users regarding cancer risk assessment tools. In this study participants had concerns about the tool fitting with everyday practice, and the need for training. Whereas some practitioners were sceptical, the service users were more encouraged by the potential of the tool to support decision making, self-management and personalized care [38].

We found that some participants had reservations over data quality, privacy and security, and potential involvement of commercial interests-all of which are common reservations expressed about the application of ‘big data’ in healthcare [39,40]. The use of personal health data for EARS overlaps with wider debates about public trust in science and healthcare, including reservations over the use of such data without explicit consent [41].

Patients consistently linked the perceived effectiveness of EARS to the involvement of clinicians and patients in decisions about care (i.e., shared decision making). This aligns with the UK policy emphasis on inclusion of patients as active partners in healthcare decisions [42]. It also correlates with the growing body of evidence supporting risk communication and the premise that the exchange of information about risk leads to better understanding and better decisions about clinical management [43,44]. Although risk communication is not straightforward as risks can be challenging to understand and communicate appropriately [45,46], it has been described as an essential prerequisite for shared decision making [47]. This in turn can support patients to manage their health and improve outcomes, including greater adherence to treatment and improved satisfaction with the process of care [45,48,49].

…providing evidence-based risk and benefit information to patients and ensuring adequate understanding and accurate perception of such information by patients, forms the cornerstone of informed decision making [45].

It is widely accepted that a mass cultural shift away from clinicians as decision makers, and towards shared decision making has yet to materialize [42], with the implication that many patients are not fully engaged in discussions of treatment choices or their personal preferences [42,50].

The unanimous agreement of participants that they would want to know their risk scores may seem surprising. While it may reflect their interest in the topic, it is consistent with findings that “most patients want more health information than they are usually given”, and that many “express disappointment about the lack of opportunities to participate in decisions about their care” [42].

Issues of equity in patient selection were apparent, with participants raising ethical concerns over the potential for patient selection based on age. Meeting the challenges of scarce resources and rising demand necessitates decisions about the most appropriate allocation of health resources - a central tenet of health economics. This is equally true of risk stratification, and there is merit in views that interventions should be offered to those who are the strongest responders and most likely to benefit from the intervention [51,52], especially if publicly funded. However, it may also be difficult to draw a line between strong, moderate and marginal responders and biases in selection may differentially affect patients from disadvantaged groups [52,53].

Implications for research and practice

While small scale, this exploratory study has reinforced the importance of involving patients and public members in debates about the use of EARS. Further patient focused study is critical to our understanding and needs to address broad principles around acceptability as well as focused work on EARS application in specific contexts, including individual patient/clinician interactions. Ideally, this patient input would be prior to the introduction of new EARS initiatives, in line with the Medical Research Council’s framework for developing and evaluating complex interventions which emphasises that the feasibility and acceptability of interventions can be improved by engaging potential intervention users in advance [23]. Further exploration of acceptability could be underpinned by the Theoretical Framework of Acceptability [21] which features seven component constructs (attitude, burden, ethicality, intervention coherence, opportunity costs, perceived effectiveness, self-efficacy) that can be explored at different stages of implementation (prior to, during and post) [21].

The feedback reported here furthers the argument for closer inspection of risk communication, and the potential of EARS to support shared decision making and patient self-management. In the context of nearly 20 years of data driven EARS tools, the development of EARS communication guidance (with both clinical users and patients as active participants) is long overdue. Subsequent evaluation of any guidance would add to the body of evidence from the disease-specific risk communication field, both in terms of approach and efficacy; and could have relevance to approaches for other generalized, holistic, or multi-morbid risks such as frailty [54] or Covid-19 [55].

Conclusion

This qualitative study adds patient perspectives to our evolving understanding of EARS in primary care. The findings reinforce the importance of understanding patient views. Nine patient representatives found EARS broadly acceptable and were optimistic about the potential for EARS risk communication to promote shared decision making and stimulate positive patient behaviours. They saw challenges in introducing EARS and proactive care to overstretched care practitioners and settings. The participants were active and engaged in the topic and many patients want similar involvement in decisions about their care. To improve our understanding of the potential role of EARS and similar risk stratified approaches in healthcare, further patient input is needed and overdue.

Contributors

MRK conceived and designed the study with input from co-authors. MRK and AP led analysis. MRK drafted the first manuscript and made the decision to publish. MRK is the guarantor. All authors reviewed drafts, assisted in redrafting, and have approved the final version.

Funding Statement

This study was supported through infrastructure funding from the Wales Centre for Primary and Emergency Care Research (PRIME).

Ethics Approval Statement

UK Research Ethics Committee Approval was not required. Written patient consent was obtained prior to participation.

References

1. Snooks H, Bailey-Jones K, Burge-Jones D, et al. (2018) Predictive risk stratification model: A randomised stepped-wedge trial in primary care (PRISMATIC). Southampton (UK): NIHR Journals Library.
2. Evans BA, Porter A, Snooks H, et al. (2019) A co-produced method to involve service users in research: The SUCCESS model. BMC Med Res Methodol 19: 34.
3. Lewis G, Curry N, Bardsley M (2011) Choosing a predictive risk model: A guide for commissioners in England. Nuffield Trust.
4. NHS England Operational Research and Evaluation Unit (2017) New care models: Risk stratification: Learning and Impact Study. London: NHS England.
5. NHS England (2015) Using case finding and risk stratification: A key service component for personalised care and support planning. NHS England.
6. NHS England (2018) Breaking down barriers to better health and care. London.
7. NICE (2016) Multimorbidity: clinical assessment and management: NG56: NICE.
8. NHS England (2015) Enhanced Service Specification: Avoiding unplanned admissions: proactive case finding and patient review for vulnerable people.
9. NHS England (2023) AI to help South West NHS spot people at risk of emergency admission: NHS England.
10. Wallace E, Stuart E, Vaughan N, et al. (2014) Risk prediction models to predict emergency hospital admission in community-dwelling adults: A systematic review. Med Care 52: 751-765.
11. Girwar S-AM, Jabroer R, Fiocco M, et al. (2021) A systematic review of risk stratification tools internationally used in primary care settings. Health Sci Rep 4: e329.
12. Müller-Riemenschneider F, Holmberg C, Rieckmann N, et al. (2010) Barriers to routine risk-score use for healthy primary care patients: Survey and qualitative study. Arch Intern Med 170: 719-724.
13. Freund T, Wensing M, Geissler S, et al. (2012) Primary care physicians' experiences with case finding for practice-based care management. Am J Manag Care 18: e155-e161.
14. Ross RL, Sachdeva B, Wagner J, et al. (2017) Perceptions of risk stratification workflows in primary care. Healthcare (Basel, Switzerland) 5: 78.
15. Evans BA, Dale J, Davies J, et al. (2022) Implementing emergency admission risk prediction in general practice: A qualitative study. Br J General Pract 72: e138-e147.
16. Stokes J, Riste L, Cheraghi-Sohi S (2018) Targeting the 'right' patients for integrated care: Stakeholder perspectives from a qualitative study. J Health Serv Res Policy 23: 243-251.
17. Wagner J, Hall JD, Ross RL, et al. (2019) Implementing risk stratification in primary care: Challenges and strategies. J Am Board Fam Med 32: 585-595.
18. Arce RS, De Ormijana AS, Orueta JF, et al. (2014) A qualitative study on clinicians’ perceptions about the implementation of a population risk stratification tool in primary care practice of the Basque Health Service. BMC Family Practice 15: 150.
19. Kingston MR (2010) Initial uses of the Prism risk stratification tool in CCM Demonstratorsites: A qualitative study. Llanharen: National Leadership and Innovation Agency for Healthcare.
20. Kingston M, Porter A, Evans B, et al. (2014) PP51 Implementation of a predictive risk tool in primary care: Examining understanding and engagement among practitioners. Journal of Epidemiology and Community Health 68: A67-A.
21. Sekhon M, Cartwright M, Francis JJ (2017) Acceptability of healthcare interventions: An overview of reviews and development of a theoretical framework. BMC Health Services Research 17: 88.
22. Diepeveen S, Ling T, Suhrcke M, et al. (2013) Public acceptability of government intervention to change health-related behaviours: A systematic review and narrative synthesis. BMC Public Health 13: 756.
23. Skivington K, Matthews L, Simpson SA, et al. (2021) A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 374: n2061.
24. Hommel KA, Hente E, Herzer M, et al. (2013) Telehealth behavioral treatment for medication nonadherence: A pilot and feasibility study. Eur J Gastroenterol Hepatol 25: 469-473.
25. Borrelli B, Sepinwall D, Ernst D, et al. (2005) A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol 73: 852-860.
26. Proctor E, Silmere H, Raghavan R, et al. (2011) Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 38: 65-76.
27. Tausch AP, Menold N (2016) Methodological aspects of focus groups in health research: Results of qualitative interviews with focus group moderators. Glob Qual Nurs Res 3: 2333393616630466.
28. Stewart DW, Shamdasani PN (2014) Focus groups: Theory and practice: Sage publications.
29. Bowling A (2002) Research methods in health. Maidenhead: Open University Press.
30. Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int J Qual Health Care 19: 349-357.
31. Jupp V (2006) The SAGE Dictionary of Social Research Methods. SAGE Publications, Ltd.
32. Evans BA, Gallanders J, Griffiths L, et al. (2020) Public involvement and engagement in primary and emergency care research: The story from PRIME Centre Wales. Int J Popul Data Sci 5: 1363.
33. Leung F-H, Savithiri R (2009) Spotlight on focus groups. Can Fam Physician 55: 218-219.
34. NIHR (2021) Briefing notes for researchers-public involvement in NHS, health and socialcare research. NIHR.
35. Braun V, Clarke V (2021) Thematic analysis: A practical guide. Sage.
36. Marshall C, Rossman GB (2014) Designing qualitative research. Sage publications.
37. Swedberg R (2020) Exploratory research. In: Elman C, Mahoney J, Gerring J, The production of knowledge: Enhancing progress in social science. Strategies for Social Inquiry. Cambridge: Cambridge University Press 17-41.
38. Akanuwe JNA, Black S, Owen S, et al. (2021) Barriers and facilitators to implementing a cancer risk assessment tool (Q Cancer) in primary care: A qualitative study. Prim Health Care Res Dev 22: e51.
39. Ancker JS, Silver M, Miller MC, et al. (2013) Consumer experience with and attitudes toward health information technology: A nationwide survey. J Am Med Inform Assoc 20: 152-156.
40. Goytia CN, Kastenbaum I, Shelley D, et al. (2018) A tale of 2 constituencies: Exploring patient and clinician perspectives in the age of big data. Med Care 56: S64-S69.
41. Kalkman S, van Delden J, Banerjee A, et al. (2022) Patients’ and public views and attitudes towards the sharing of health data for research: A narrative review of the empirical evidence. J Med Ethics 48: 3-13.
42. Coulter AC, Alf Collins (2011) Making shared decision-Making a reality: No decision about me, without me. London.
43. Ahl AS, Acree JA, Gipson PS, et al. (1993) Standardization of nomenclature for animal health risk analysis. Rev Sci Tech 12: 1045-1053.
44. Hargraves I, Montori VM (2014) Decision aids, empowerment, and shared decision making. BMJ 349: g5811.
45. Naik G, Ahmed H, Edwards AG (2012) Communicating risk to patients and the public. Br J Gen Pract 62: 213-216.
46. Dodson S, Good S, Osborne R (2015) Health literacy tool kit for low and middle-income countries: A series of information sheets to empower communities and strengthen health systems.
47. Edwards A, Elwyn G (2001) Understanding risk and lessons for clinical risk communication about treatment preferences. Qual Health Care 10: i9-i13.
48. O'Connor AM, Bennett CL, Stacey D, et al. (2009) Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev Cd001431.
49. Zolnierek KB, Dimatteo MR (2009) Physician communication and patient adherence to treatment: A meta-analysis. Med Care 47: 826-834.
50. Fagerlin A, Zikmund-Fisher BJ, Ubel PA (2011) Helping patients decide: Ten steps to better risk communication. J Natl Cancer Inst 103: 1436-1443.
51. Fleck LM (2012) Pharmacogenomics and personalized medicine: Wicked problems, ragged edges and ethical precipices. N Biotechnol 29: 757-768.
52. Shadmi E, Freund T (2013) Targeting patients for multimorbid care management interventions: The case for equity in high-risk patient identification. Int J Equity Health 12: 70.
53. ASSEHS Study Team (2016) White Paper on Deployment of Stratification Methods.
54. Clegg A YJ, Bower P, Cundill B, et al. (2018) Personalised care planning to improve quality of life for older people with Frailty. NIHR, Programme Grants for Applied Research.
55. Hippisley-Cox J, Coupland CA, Mehta N, et al. (2021) Risk prediction of covid-19 related death and hospital admission in adults after covid-19 vaccination: National prospective cohort study. BMJ 374: n2244.