Duke Vaccinates: A Project to Increase Adult Immunization Rates for Four Major Vaccine Types Involving the Entire Primary Care Team

Introduction: Current adult vaccination rates fall well below the Healthy People 2020 goals established by the Centers for Disease Control and Prevention for pneumococcal, influenza, pertussis, and herpes zoster vaccines. The objective of this study was to improve immunization rates in primary care through an enhanced, multi-component “audit & feedback” (A&F) intervention. Methods: The intervention took place over a one-year period (September 2016-August 2017) and involved 24 primary care clinics within one health system in North Carolina. It consisted of pre-specified targets and an online A&F platform, combined with multidisciplinary educational resources and non-physician clinic champions. Six control clinics in the same health system were also identified. Clinic and provider performance for immunization for pneumococcal, influenza, pertussis, and herpes zoster vaccines were tabulated on a monthly basis. Rates were benchmarked against a clinic-defined target, other participating clinics’ immunization rates, and national targets. Results were displayed on an electronic dashboard within the online platform, and printable reports were given to each clinic to post in work areas and discuss at practice meetings on a monthly basis. Educational efforts primarily focused on non-physician clinical personnel (nurses, medical assistants) and consisted of online learning modules and 5-minute videos. Frontline nurses or medical assistants served as site champions to receive and distribute monthly immunization reports and attend quarterly project calls to discuss barriers and solutions with champions from the other clinics. Comparisons between intervention and control clinic groups were made using the t-test or Wilcoxon rank-sum test for continuous variables depending on normality, or Pearson’s chi square test for categorical variables. To assess the effectiveness of the intervention, the likelihood of immunization during the intervention period was compared between intervention and control clinics using logistic regression models. The generalized estimating equations method was used to account for clustering of outcomes within clinics. Analyses included adjusting for baseline vaccination rates and differences in case mix between intervention and control clinics. Results: A total of 209,533 and 64,133 patients were seen in the intervention and control clinics, respectively. Intervention patients were younger (mean age 51 years vs. control 53 years, p < 0.0001) and had higher proportions of patients who were female (60% vs. control 55%, p < 0.0001), and privately insured (65% vs. control 58%, p < 0.0001), but a lower proportion of Caucasian patients (65% vs. control 72%, p < 0.0001). All vaccine types showed increases in immunization rates during the intervention period in both intervention and control practices. After accounting for covariates, patients seen in the intervention clinics were more likely to receive an influenza vaccine (adjusted OR 1.27, 95% CI 1.02-1.58, p = 0.03) and complete the pneumococcal vaccine series (adjusted OR 2.40, 95% CI 1.10-5.24, p = 0.03). No differences between intervention and control clinics were seen for Tdap vaccine (adjusted OR 1.05, 95% CI 0.87-1.26, p = 0.61), or zoster vaccine (adjusted OR 1.16, 95% CI 0.90-1.49, p = 0.25). Discussion: Implementation of an evidence-based, multi-component intervention incorporating audit and feedback, clinic-defined targets, supportive educational resources, and other efforts aimed at engaging non-provider clinic staff (including as champions) appeared to increase the likelihood of influenza and complete pneumococcal vaccination for adults in primary care, but not for pertussis and zoster. Check for updates Citation: Cho AH, Wang TY, Anderson J, et al. (2020) Duke Vaccinates: A Project to Increase Adult Immunization Rates for Four Major Vaccine Types Involving the Entire Primary Care Team. Arch Fam Med Gen Pract 5(1):149-160 cho et al. Arch Fam Med gen Pract 2020, 5(1):149-160 Open Access | Page 150 | of an audit & feedback platform in a target-based model, coupled with supportive educational resources and other efforts aimed at frontline healthcare personnel other than providers. Our objectives were to answer these two questions for each vaccine type (influenza, pneumococcal, tetanus/diphtheria/pertussis (Tdap), and zoster): (1) What is the prevs. post-intervention change in change in immunization rates among eligible patients? (2) Do intervention clinics participating in the QI initiative have greater improvement in appropriate immunization rates compared with control (non-participating) clinics?

of an audit & feedback platform in a target-based model, coupled with supportive educational resources and other efforts aimed at frontline healthcare personnel other than providers.
Our objectives were to answer these two questions for each vaccine type (influenza, pneumococcal, tetanus/diphtheria/pertussis (Tdap), and zoster): (1) What is the pre-vs. post-intervention change in change in immunization rates among eligible patients? (2) Do intervention clinics participating in the QI initiative have greater improvement in appropriate immunization rates compared with control (non-participating) clinics?

Study population
The Duke Vaccinates Adult Immunization Project targeted 24 primary care internal medicine/family medicine clinics; all were part of Duke Primary Care (DPC) in the Duke University Health System. Six other Duke-affiliated primary care internal medicine/family medicine practices were selected as a concurrent control group. The pre-intervention baseline period included all patients over age 18 years with at least one outpatient visit at these clinics between April 1, 2015, and March 31, 2016. The intervention study period included all patients with at least one outpatient visit at these clinics between September 1, 2016, and August 31, 2017.

Vaccine measure selection and patient eligibility
The CDC's Advisory Committee on Immunization Practices (ACIP) recommendations for vaccination against influenza, pneumococcal pneumonia, pertussis, and herpes zoster were used to identify groups who should be vaccinated, based on age and co-morbid conditions [12][13][14][15][16][17]. Five principal measures were defined, based on then-current recommendations; these are summarized in Table 1. Additional demographic and problem list variables were abstracted from the medical record to ascertain each patient's eligibility for each vaccine. The criteria for each vaccination measure are described in Appendix 1.

Introduction
Adult immunization rates in the United States are well below the Healthy People 2020 goals established by the Centers for Disease Control and Prevention (CDC) [1,2]. In 2013, the CDC reported pneumococcal vaccination coverage (either 23-valent pneumococcal polysaccharide vaccine (PPSV23) or 13-valent pneumococcal conjugate vaccine (PCV13)) among adults aged 19-64 years at high risk was 21.2% overall. Among adults aged ≥ 65 years, coverage was 59.7%. The Healthy People 2020 target for both high-risk adults and adults aged 65 years and older is 90% [1].
The rise in pertussis incidence over the past decade has led to revised recommendations for all adults aged ≥ 19 years to receive a single dose tetanus and diphtheria with acellular pertussis (Tdap) vaccine, particularly adults who have or anticipate having close contact with an infant less than 12 months of age [3]. In this group, Tdap coverage was 14.2% in 2012. In 2013, 24.2% of adults aged ≥ 60 years reported receiving herpes zoster vaccination (to prevent shingles), up from 20.1% in 2012. The Healthy People 2020 target is 30% for adults aged 60 years and older [1].
The consequences of these relatively low rates include increased risk for vaccine-preventable illnesses. Older adults may be particularly vulnerable due to age-related changes in immune system function and comorbid conditions [4]. In addition, there is evidence of disparities in immunization rates among racial and ethnic minorities [5,6].
Prior studies of interventions to increase immunization rates have found a number of clinician-directed interventions to be effective, including audit and feedback, clinical decision support and other reminders, financial incentives, and education [7].
Audit and feedback (A&F) specifically involves the collection of data regarding clinical performance, and then the presentation of these data in the form of feedback to the relevant clinicians, with or without recommendations for practice. A systematic review of A&F-based interventions to improve vaccination coverage found that in 12 of 15 studies, A&F, alone or in combination with other interventions, was associated with improvements in immunization rates [8].
In an enhanced version of A&F, a "target-based model" that also posted individual provider performance in the clinic on special posters was shown to significantly increase influenza immunization rates [9]. In another example, A&F involving registered nurses in an outpatient surgical department demonstrated that monthly feedback, as opposed to annual feedback, resulted in larger improvements in pneumococcal vaccination [10]. Shared responsibility between physicians and non-physician healthcare personnel has also been found to be effective [11].
Here we report outcomes of an evidence-based quality improvement (QI) initiative to increase adult immunization rates across a primary care network for four important vaccine-preventable diseases (influenza, pneumococcal pneumonia, pertussis, and herpes zoster), which incorporated key elements from the successful examples cited above: the use dence-based approaches for having "strong" conversations with patients about vaccinations. All of these educational materials were housed in an online learning library. CDC vaccine information sheets and post-vaccination care instructions for patients were also made available. Appendix 4 lists the resources created.
Clinics were encouraged to organically integrate these materials into ongoing staff and provider education, e.g., making use of regularly scheduled clinic staff meetings and other routine communications and training opportunities -including orientation of new nursing staff -but no pre-defined educational "program" per se was implemented across participating sites.

Non-physician clinic champions:
In the interest of promoting shared responsibility across care team roles, each clinic was encouraged to select site champions who were not physicians, other providers, or managers -but ideally frontline medical assistants and nurses responsible for administering vaccinations. These champions were asked to receive and distribute monthly reports and other communications, and represent their clinics at a project kick-off meeting and closing celebration, as well as on quarterly conference calls in which clinic leadership was also encouraged to participate.
Overall, the intervention was designed to integrate feedback on clinical performance in a way that achieved intermittent but regular contact with clinic staff and providers, with minimal disruption to clinic workflow. In addition, value-added tools developed primarily for frontline staff were provided that could be aligned with established clinic activities, but clinics had autonomy in determining how these tools would be used.

Data collection
The data used for this quality improvement project were extracted from the electronic health record (EHR) for the pre-intervention baseline and intervention study periods so that they reflected the outcomes of clinical care. Patients were considered vaccinated if the vaccine was administered in clinic or elsewhere within the Duke Health system, or if it was documented in the record as having been received else-on a monthly basis, utilizing the internet-based MedConcert healthcare performance improvement and reporting platform (Premier; Homestead, PA).
Clinic performance was benchmarked against the other participating clinics and national targets (Healthy People 2020). Overall performance, vaccine-specific immunization rates for particular patient groups, and provider-level immunization rates were made accessible to providers and clinic staff via the same MedConcert platform, organized as an electronic dashboard as shown in Appendix 2. From this dashboard, lists of patients could be generated who were seen but not recorded as having been vaccinated.
Structured, printable reports for all five vaccination measures were also electronically delivered to each clinic monthly (Appendix 3). To complete the feedback loop, clinics were asked to share these reports with clinic staff and providers, as well as post them in the clinic in areas visible to staff and providers (as well as patients, at clinic discretion).

Clinic-defined targets:
Each clinic was also asked to come up with their own, clinic-defined targets for each vaccine measure. The monthly reports charted clinic performance over time against these targets as well as benchmarked against clinics.
Supportive educational resources: Education efforts primarily focused on frontline nursing staff, mostly medical assistants (who comprised the bulk of the clinic workforce directly supporting providers), but also licensed practice nurses (LPNs) and registered nurses (RNs). Guided by the nursing leadership of DPC, specific resources were created addressing current vaccine recommendations and processes of care related to their administration, including videos and online training modules. All clinics were also provided a printed comprehensive immunization training manual, tip sheets, and pocket cards for nursing staff.
Brief "mini" (i.e., five minutes in length) grand rounds-like videos aimed at providers covering each of the four vaccine types were recorded, and conducted as live sessions at two different clinic sites. Additional videos made available by the project sponsor taught providers and nursing staff evi- igible patients seen during the pre-intervention baseline and intervention study periods in both intervention and control clinics. Rates were then compared between the baseline and intervention periods using the P. Thompson test to accommodate a partially overlapped population (i.e., patients seen in the pre-intervention period may also have been seen during the intervention period at the same clinic) [18].
Immunization rates are also presented for pre-specified individual subgroups stratified by age, sex, race, insurance status, history of chronic lung disease, diabetes, and cancer history.
To assess the effectiveness of the intervention, the likelihood of immunization during the intervention period was compared between intervention and control clinics using logistic regression models. The generalized estimating equations method was used to account for clustering of outcomes within clinics (i.e., two patients treated within the same clinic would have more similar outcomes than two patients treated where. Each intervention clinic was asked to complete both a pre-and a post-intervention survey assessing perceived knowledge about vaccines and barriers to immunization. Survey responses were collected at the clinic level rather than the individual staff or provider level. Post-intervention surveys additionally assessed utility of educational materials.

Statistical analysis
Patient characteristics are described for all patients seen in the intervention clinics and control clinics over the intervention period. Comparisons between intervention and control clinic groups were made using t-test or Wilcoxon ranksum test for continuous variables depending on normality, or Pearson's chi square test for categorical variables.
For each vaccine, immunization rates are expressed as the numerator (number of patients who received the vaccine) over the denominator (number of patients eligible for the vaccine). These immunization rates were calculated for all el-

Pneumococcal
Rates of complete vaccination against pneumococcal pneumonia in older adults over age 65 years (i.e., documentation of receipt of both PCV13 and PPSV23 vaccines) increased in absolute terms by 28.1% (Table 3), from a baseline of 29.7%. Increases were seen regardless of age, sex, race, insurance status, and comorbidities, with the smallest change among patients without health insurance (Table 4). In control clinics, pneumococcal immunization rates were 7.7% at baseline and 21.7% during the intervention. Adjusting for baseline vaccination rates, older patients seen in the intervention clinics were more likely to receive both pneumococcal vaccines (OR 1.86, 95% CI 1.06-3.25, p = 0.03). After adjusting for differences in patient case mix between intervention and control clinics, we again observed a more than two-fold higher likelihood of pneumococcal vaccination in patients seen in the intervention clinics (adjusted OR 2.40, 95% CI 1.10-5.24, p = 0.03).
Among younger patients ages 19 through 64 who are either immunocompromised, or immunocompetent with high-risk comorbid conditions, pneumococcal vaccination in the intervention clinics increased by 9.5%, from a baseline of 29.3%. Increases were again seen in all subgroups, with the lowest increase among patients with cancer (Table 4). In control clinics, immunization rates for these younger, higher-risk patients were 20.2% and 25.0% during the baseline and intervention periods, respectively. Only adjusting for baseline vaccination rates, patients seen in the intervention clinics were not significantly more likely to receive both pneumococcal vaccines during the intervention period (OR 1.31, 95% CI 0.97-1.77, p = 0.08). However, after adjusting for differences in patient case mix between intervention and control clinics, we did observe a higher likelihood of complete pneumococcal vaccination in patients treated in the intervention clinics (adjusted OR 1.72, 95% CI 1.10-2.68, p = 0.02).
A detailed breakdown of pneumococcal immunization rates, by vaccine and subgroup, is in Appendix 5.

Tdap
Tdap immunization rates in the intervention clinics increased overall by 11.4%, from a baseline rate of 48.1%. In-at different clinics) [19].
The likelihood that, during the intervention period, an eligible patient treated in an intervention clinic would be more or less likely to receive the vaccine compared with an eligible patient treated in a control clinic is expressed as an odds ratio (OR) with 95% confidence interval (CI). One model was performed only adjusting for baseline immunization rates in the pre-intervention period. A second model additionally adjusted for differences in case mix between the intervention and control clinics, for patients seen during the intervention period -as described by age, sex, race, insurance status, history of chronic lung disease, diabetes, cancer history, alcohol use, liver disease, HIV status, and smoking status.

Results
A total of 209,533 and 64,133 patients were seen in the intervention and control clinics, respectively, during the study period. As shown in Table 2, patients seen in the intervention clinics were younger, more likely to be female and of nonwhite race, and more likely to have private health insurance. The prevalence of comorbidities was similar between groups.
Intervention clinics involved 147 physicians, 46 nurse practitioners and physician assistants, and over 500 clinic staff. All intervention clinics received monthly immunization reports. Provider-level immunization data were accessed for 130 providers in the MedConcert platform, for a median of two times (25 th , 75 th percentiles: 1, 3; maximum 10) over the course of the intervention period. 85.8% of platform access occurred during the first six months of the intervention.
Baseline and intervention immunization rates in the intervention clinics are shown in Table 3. All vaccine types saw significant increases in immunization rates during the intervention period.

Influenza
Influenza vaccination rates in the intervention clinics increased by 7.1% from a baseline of 52.4%. Increases were seen regardless of age, sex, race, insurance status, and presence or absence of comorbid conditions (Table 4). In control clinics, influenza vaccination rates were 39.7% prior to and 46.9% during the intervention. Patients seen in the intervention clinics were more likely to receive an influenza vaccine during the intervention period after adjusting for baseline vaccination rates (odds ratio (OR) 1.27, 95% confidence interval (CI) 1.02-1.58, p = 0.03). Following adjustment for differ-

Use of educational resources
Clinic staff and provider usage of the online training mod-ules is shown in Figure 1. In total, there were 2985 logins

Zoster
Overall zoster vaccination rates in the intervention clin-ics increased by 7.4%, from a baseline of 28.6%. Increases were seen across subgroups; the smallest change was among uninsured adults (from 6.3% to 10.0%) (  Figure 4 summarizes self-reported changes in immunization-related practices. The two largest changes reported involved expanded roles for clinic staff -with nearly 80 percent of practices reporting post-intervention that staff routinely initiate vaccination conversations with patients in clinic, and printed materials were more likely to be rated "extremely" or "very useful" than online educational resources such as modules and videos (Figure 2).
Intervention clinics also reported an increase in staff awareness of the benefit of immunizations (Figure 3). Post-intervention, all clinics reported being "very" or "somewhat familiar" with why influenza, pneumococcal, and Tdap vaccines   "Very useful" combines 'extremely' + 'very useful' survey response categories. "Videos" refers to both videos and online modules.
cifically to obtain vaccinations that were due, or address insurance coverage or cost issues related to vaccinations (data not shown).
Perceived barriers to immunization are summarized in Figure 5. After the intervention, more sites reported low health literacy, misconceptions about vaccine risk, fear of needles, inconvenience of getting vaccines as potential barriers to immunization. Uncertainty regarding cost of vaccination was less commonly reported post-intervention.
nearly 90 percent reporting that staff remind providers when patients are due for one or more immunizations.
In addition, more clinics reported reviewing the immunizations tab in the EHR, consulting sources like the state immunization registry to find vaccinations administered outside of the health system, and documenting outside vaccinations in general as well as reasons for not giving vaccinations (e.g., concurrent illness, patient declines, etc.). However, there was little change post-intervention in the number of practices reporting they routinely remind patients to schedule visits spe-  A similar limitation applies to Tdap -when administered for preventive purposes -although tetanus vaccination for acute injury would be covered, and Tdap is significantly less expensive (retailing for ~$50).
Such cost-sharing can pose significant financial hardship, contributing to the lower immunization rates observed, and is further exacerbated by the need for patients to present elsewhere to receive these vaccinations, and they and/or pharmacies to self-report vaccination in order to be documented in the EHR.
Regarding the utility of educational resources developed for the intervention, the finding that printed materials were rated more useful than more comprehensive online and video-based ones may reflect the needs of experienced medical assistants, where practical aids such as tip sheets -which addressed specific questions such as insurance coverage of certain vaccine types -were felt to be most helpful in day-to-day practice. In contrast, online and video-based materials may actually be more difficult to use in the midst of a busy clinic, but may be more important for newer medical assistants.
The relatively low rates of MedConcert usage observed were probably in part due to reports summarizing the Med-Concert data at a clinic level being furnished to clinic siteswhich was done because the platform itself did not enable creation of easily printable summaries. Provider-specific views were only available through the platform, and with completion of online learning modules by newer staff, likely represented its main use as the project went on.
In addition, there was some overlap with the aforementioned existing EHR tools, which also allowed providers to identify patients lacking certain vaccinations.

Discussion
In summary, an evidence-based, multi-component intervention was implemented over 12 months, incorporating audit and feedback (A&F), clinic-defined targets, supportive educational resources and other efforts specifically aimed at engaging non-provider clinic staff. The likelihood of influenza and complete pneumococcal vaccination for all adult patients was found to be significantly greater in intervention clinics than control clinics, after adjusting for baseline rates and case mix differences. The likelihood of Tdap and zoster vaccination, however, was not found to be statistically different between the two sets of clinics.
Absolute increases in primary care adult vaccination rates were observed across all four vaccine types in both intervention and control clinics. This finding could be attributed in part to the use of the same EHR by intervention and control clinics, with embedded tools for reminding providers and clinic staff when vaccinations were due. For pneumococcal vaccination in particular, another possible factor was that the recommendation to start PCV13 vaccination in older adults had only recently been issued (in 2014), increasing the overall level of awareness among all providers.
The absence of a statistically significant difference in likelihood of zoster vaccination, until recently only recommended for older adults (≥ 60 years old), may point to an important contextual limitation related to Medicare coverage. (Medicare is the federal healthcare insurance program for Americans aged 65 and older, who are permanently disabled, or who are on dialysis for end-stage renal disease.) Medicare Part B insurance, which covers outpatient services, does not routinely cover zoster vaccination in primary care. As a result, Medicare beneficiaries generally have to obtain this on their own at an outside pharmacy.   Many QI interventions of the type described here conventionally focus on providers, in this case usually being the ones asking patients about and ordering vaccines. One of the distinctive elements of our intervention was a deliberate strategy of engaging frontline nursing staff (medical assistants, LPN/RNs) -who are not only responsible for administering vaccines, but through the emergence of team-based clinic workflows, are increasingly empowered to "tee up" health maintenance activities like vaccination on behalf of providers. This, in conjunction with tactics like standing orders, can reduce the likelihood of repeated "missed opportunities" [21,22].
This strategy appears to have succeeded, and was reflected in the dramatic increase, post-intervention, in clinics reporting that non-providers were initiating vaccine conversations with patients and/or reminding providers when the next patient to be seen was due for vaccination.
Thus, as team-based care becomes a fixture of the primary care landscape, our intervention demonstrates that QI efforts that engage these frontline staff, including as champions for their clinics -in contrast to the traditional focus on provider behavior -can be broadly effective in raising immunization rates.
One potentially significant limitation was that all 24 participating practices were part of the same health system primary care network. The six control practices, however, were also affiliated with the same health system and on the same EHR, with access to the same EHR tools. Comparisons of the intervention and control cohorts also found a similar prevalence of comorbid conditions, but did find some statistically significant demographic differences, as well as differences in baseline vaccination rates, which we attempted to address through risk adjustment.
Another operational limitation was that it took more than three weeks from the end of a calendar month to extract, clean, process, and upload that month's data for visualization in the MedConcert platform -and nearly two months before these data could be packaged into printable, structured reports.
This lag was also the reason that the pre-intervention baseline and intervention study periods were not identical, which though not ideal had the benefit of mitigating the risk that preparations for the intervention (e.g., engaging practices, etc.) might alter pre-intervention data. Nevertheless, complete data for two full 12-month periods were compared, both including a contiguous flu season.
We also did not examine the incidence of vaccine-preventable disease in the populations studied, which would require a significantly longer timeframe to examine, although that could be a topic of future study. Furthermore, this intervention was based on then-current recommendations regarding the use of both PPSV23 and PCV13 pneumococcal vaccinations. These were revised in 2019 to recommend shared decision-making regarding PCV13 in particular, owing to the reduction in overall prevalence of pneumococcal disease observed after routine administration of PCV13 to children began in 2010 [20].
In addition to answering the study's main questions, another objective was to develop an intervention that could be replicated in other settings. To this end, our intervention components were intended to be generally supportive of whatever steps individual clinics might take to improve immunization rates -but not to prescribe what those steps should be, or how things like the educational resources should be used.
Two of the more novel approaches that we witnessed included screening key segments of the training videos during staff meetings (rather than asking staff to watch these individually); and recording a staff-created video, set to music, which touted the benefits of immunizations in an entertaining way and was shared with patients. The quarterly conference calls allowed sharing of practices like these, in addition to airing questions and discussion of common issues.
We also tried to refrain from introducing components that would be completely impossible to replicate in the absence of study-specific resources, or in day-to-day practice. For instance, bringing clinic champions together twice in person and three times on conference calls in a 12-month period does not seem unreasonable. Online learning management systems are commonplace, and the technology to record vid-