Acceptability and Reliability of the Paper and Online Versions of the MEPA-III Screener

Abstract

Background

The relative validity of a brief online dietary tool to assess accordance to a Mediterranean diet pattern, the Mediterranean Eating Patterns of Americans (MEPA) III screener has been demonstrated [1]. Further research was necessary to assess usability and the reliability (stability) of this 21-item screener.

Objective/Methods

To assess the 1) Acceptability and feasibility of online and paper versions of the MEPA III screener in patients with Parkinson's disease (PD), and 2) Test retest reliability of the screener. For the first objective, participants with PD completed the online and paper screeners five to seven days apart followed by an exit questionnaire. The latter included a 65-point system usability scoring (SUS) method [2], where higher scores reflect greater acceptability. Feasibility was defined by the time needed to complete the tool; less than 15 minutes was desirable. For assessment of test-retest reliability (stability), health professionals and patients with PD completed the online version of MEPA III twice, five to seven days apart. Statistics were performed using SPSS, version 23. Differences in MEPA III scores, SUS values, and completion times between the different administration modes were assessed with Wilcoxon signed rank tests or t-tests based on normality. General linear models were used to examine the impact of administration order. Repeated online administration of screeners was assessed via Wilcoxon signed rank and intra-class correlation (ICC) tests.

Results

For the first objective, 32 patients with PD (19 men: 13 women, 64.5± 7.3 years) completed both online and paper MEPA III screeners. Median (IQR) scores for paper MEPA III were 10.5 (IQR 9.0, 13.0) and online, 10.0 (IQR 9.0, 12.0). The order in which paper and online screeners were administered did not influence MEPA III scores (p = 0.054), nor were there differences between online or paper MEPA III scores (p = 0.89). In terms of acceptability, paper SUS was 49.7 ± 7.8 and online SUS was 49.1 ± 8.1 (p = 0.26). Feasibility did not differ between paper version (8.6 ± 3.8 minutes) and online version (9.9 ± 4.0 minutes). For assessment of screener stability, 8 patients with PD and 24 health professionals completed the online MEPA III screeners twice roughly a week apart. There was little to no difference in MEPA III scores observed between the first administration 12.0 (10.0, 14.0) and the second, 12.5 (10.0, 14.0) (p = 0.93). The ICC was moderately high (0.84, p < 0.001).

Conclusions

Based on preliminary testing, the MEPA III scores were similar as was rating of the acceptability of the paper and online screeners by patients with PD. The online screener exhibited good stability. Further evaluation is needed in a larger, more diverse group of participants.

Keywords

Mediterranean diet, Parkinson's disease, Nutrition, Diet assessment tool

Introduction

The slow loss of dopaminergic neurons in Parkinson's disease (PD) impacts motor and non-motor functions, resulting in symptoms such as cognitive decline, decreased range of mobility, and dyskinesia. These symptoms can lead to impaired independence and eating behaviors, ultimately affecting nutrition status [3].

There is limited research assessing nutrition status in patients with PD, though there are a few reports on accordance to the Mediterranean diet (MedDiet) and risk of PD diagnosis [4-6]. The MedDiet refers to an eating pattern high in fruits, vegetables, beans, whole grains, olive oil, nuts, fish, and wine. With the possible increased energy needs, unintentional weight loss, and cognitive decline in PD [7-9], this diet may provide multifactorial benefits to this patient group.

Capturing accurate consumption of food groups is essential in appropriately assessing adherence to the MedDiet. Thus, the Mediterranean Eating Pattern for Americans III (MEPA III) screener was developed to assess such a pattern among U.S. adults and, in particular, among adults with PD, but in a minimal amount of time. The MEPA III was developed from the original MEPA [10] and further refined and validated among patients with PD. Evaluation of how acceptable and feasible a dietary screener is in a PD population is essential prior to implementation.

Thus, the purpose of this study is to assess the 1) Comparability of scores as well as the acceptability and feasibility of an online versus paper version of the MEPA III in patients with PD, and 2) Test-retest reliability of an online version of MEPA III amongst patients with PD and health professionals. Demonstration of acceptability, feasibility, and comparability of online or paper administration is needed to support the use of MEPA III in future research and clinical settings.

Methods

Patients with PD were recruited from a tertiary medical center movement disorders clinic between January 1, 2018 and March 1, 2019. To test two different administration modes of MEPA III screener, participants were systematically assigned (alternately) to complete the paper or online version of the MEPA III first, then the other version after a five to seven-day interval. An exit questionnaire that includes system usability (SUS) questions was completed immediately after the screener each time. Thus, the paper version of MEPA III included a paper version of the exit questionnaire, while the online MEPA III included an online exit questionnaire. This portion of the study was designed to assess acceptability via SUS, feasibility as completion time, as well as comparability between both administration modes of the MEPA III.

To test the reliability (test-retest) of the online MEPA III screener. A convenience sample of patients with PD and health professionals (i.e. physicians and registered dietitians) at RUMC was sent a link to complete the screener online twice, five to seven days apart. The time interval was selected to reduce the likelihood that respondents would be primed by prior responses, yet their overall dietary choices had not changed. A priori, researchers considered a difference of less than two points between both online MEPA III administrations to be deemed reliable.

Subject Recruitment

Eligibility for first part included patients who: 1) Had a diagnosis of PD, 2) Were 40 years-old or older, 3) Read and understood English, 4) Had access to an email account and internet. Eligibility for the reliability assessment included patients with PD and health professionals who read and understood English and had access to an email account and internet. All patients and healthcare professionals provided written consent before inclusion to the study. This study was approved by the Institutional Review Board. Clinical rating scores, such as Hoehn and Yahr or HY (Stage 1-5, with higher stages indicating more progressive disease), and other demographics including education level, marital status, body mass index (BMI), and co-morbidities, were obtained from online medical records via Epic (Hyperspace 2017, Verona, Wisconsin).

MEPA III Screener

The MEPA III is the third rendition of an online dietary assessment tool, MEPA, the first version was assessed for relative validity in a cardiology clinic at RUMC [10], with 70 women at risk of cardiovascular disease. The current MEPA III is a 21-item screener, including questions pertaining to the intake of oils, butter, avocados, peanuts, nuts, red meat, fish, poultry, milk, cheese, beans, whole grains, berries, other fruit, green leafy vegetables, starchy vegetables, other vegetables, pastries, alcohol, fast food, pre-packaged foods, unsweetened beverages, and sugar-sweetened beverages. One point is assigned to each food consumption category that meets the criterion target servings, thus giving the MEPA III a score range of 0-21 points. The highest possible score on the screener is 21. Participants were able to complete the MEPA III online through REDCap (2019, Vanderbilt University) or provided the paper version to complete in a systematically assigned order. Relative validity of the screener was demonstrated by concordance of responses to a food frequency questionnaire and agreement between component scores derived from screener and FFQ [10]. With both the online and paper versions of the screener, participants were asked to note the time they started the screener and when they completed the screener. The time needed to complete the tool was used as a measure of feasibility.

Exit questionnaires

The MEPA III Exit Questionnaire was modified to quantify acceptability of the screener [2,11]. It contained 11 SUS questions designed to summarize participants' feedback regarding ease of use of the MEPA III. Eight of these were on a 4-point Likert scale. There were also two questions on a summative rating scale, 1-10, to assess perceived length of the screener (too short to too long) and overall satisfaction with the screener (poor to excellent). The highest possible score on the exit questionnaire is 65, which indicates the highest level of acceptability.

Statistical Analysis

Data analyses were performed using SPSS, version 23 (IBM SPSS, Chicago, IL, USA). Continuous data were examined for normality to determine if non-parametric measures were needed. Descriptive statistics were run for all demographics. Continuous variables were expressed as means ± standard deviations or median and interquartile range if not normally distributed.

General linear models were performed to assess whether order of mode of administration (paper versus online MEPA III) influenced MEPA III scores, and whether scores from the online screener differed from those completed on paper. Similar models were run for SUS scores and completion time. A Wilcoxon signed rank test or paired t test was performed to test for differences between online and paper MEPA III scores; similarly, such tests were run for acceptability( SUS numerical scores) and feasibility (completion times).

Wilcoxon signed rank test were performed to compare differences between of MEPA III scores of participants to determine reliability (stability) of the screener. An intra-class correlation was also calculated to reflect the relationship between scores observed following repeated administration of the same online MEPA III tool in the same participants five to seven days apart. A correlation coefficient of at least 0.8 was important in determining reliability of the screener. Feasibility and acceptability (SUS) was also assessed.

Results

A total of 236 patients were screened as eligible for the first part of the study. Of those screened as eligible, 49 patients provided informed consent and were enrolled in the study. In total, 14 study participants completed only part of the study and 3 were lost to follow up before completing any part of the study (Table 1). The final study sample (n = 32) included 13 women and 19 men, aged 64.5 ± 7.3 years (mean ± SD). Median duration of PD diagnosis was 5.0 (IQR 2.3, 7.0) years. Majority 90%, of participants were at HY Stage 2, with a median UPDRS score of 19.5 (IQR 15.0, 29.3). On average, the participants were slightly overweight (mean ± SD, 25.9 ± 4.9 kg/m²). Less than a fifth of participants had a history of diabetes, and 37.5% had a history of cardiovascular disease. Of the 28 participants who had education level listed in the medical record, nearly all have some college or higher (96.4%). Majority (78.1%) of participants were married.

As shown in Table 2, the difference in MEPA III scores when comparing paper versus online versions of the screener overall was minimal (p = 0.89). No significant difference was found when observing order of modes of administration of MEPA III (p = 0.054), though there was slightly more variability between the two MEPA III scores when the paper version was taken first (1.0 point difference when paper was taken first vs. 0.0 point difference when online was taken first). For example, when the paper version of the screener was delivered first, the paper MEPA III score was 11.5 (9.0, 14.0); when the online version was delivered five to seven days after, a score of 10.5 (9.0, 13.0) was observed. In comparison, when the online version of the screener was given first, the online MEPA III score was 9.5 (8.0, 11.0), followed by the paper MEPA III score of 9.5 (8.0, 11.3).

Overall, average feasibility, as measured by screener completion time, met the criteria of being less than 15 min (mean ± SD, 9.3 ± 3.4 minutes). There was no difference in completion times for the paper version of MEPA III and the online version.

Acceptability based on SUS scores (the highest possible score of 65) was comparable between both administration modes (49.7 ± 7.8 paper vs. 49.1 ± 8.1 online). Both SUS scores recorded moderate acceptability for both online and paper versions (Table 2). In Figure 1, responses to select SUS questions are displayed for those commenting on the paper version of MEPA III (leftmost), while Likert responses for the online version are provided on the rightmost column. Exactly 85% of participants said they would complete the paper version of the screener again if their doctor asked them to, 97% would do so on the online version. Thirty-one percent of participants responded with ‘agree' or ‘strongly agree' that recording answers for how often they ate each food was troublesome for the paper version, while 62% responded the same way for the online version.

In total, 38 individuals provided consent and were enrolled in the reliability sub-study. Four participants were lost to follow-up, while two only completed a portion of the study.

The final sample (n = 32) included 10 males and 22 females, 24 of which were healthcare professionals (19 of which were registered dietitian nutritionists, RDNs). Participants had a median age of 33.3 (28.5, 64.3) years (Table 3).

In assessing the reliability of the online MEPA III screener, the median score was 12.0 (IQR 10.0, 14.0) for the first MEPA III, and 12.5 (IQR 10.0, 14.0) for the second MEPA III taken within five to seven days for all participants (p = 0.86). The median time between the two MEPA III administrations was 7.0 (5.0, 9.8) days for all 32 participants. The intra-class correlation was 0.84 (95% CI (0.67, 0.92), p < 0.001), indicating good stability or consistency amongst all participants' MEPA III scores. Patients with PD exhibited a slightly more stable performance between repeat screeners when compared to both RDNs and other health professionals. Because the sample numbers for the healthcare professionals and patients were small and not balanced, no further statistical analyses were performed. A priori, the present researchers considered difference of two or more points between the two online MEPA III scores to reflect poor stability or reliability. Although the intra-class correlation indicates good stability or consistency overall, nine participants (both HP and patient with PD) had differences of two points or greater between the two online administrations of the screener.

Feasibility, measured as completion time, was similar between first and second administration. Most participants took less time to complete the MEPA III the second time 5.5 (IQR 4.0, 10.3) minutes vs. 5.0 (IQR 3.0, 7.0) minutes, respectively), though this was not significant. Acceptability of the MEPA III screener, measured by the SUS score from the exit questionnaire with the highest score possible being 65, deemed the assessment tool moderately acceptable with a median score of 50.0 (IQR 48.0, 54.0) by all participants.

Discussion

Previous analysis of this screener found a good concordance (r = 0.50, p < 0.001) between coded responses to a 156-item FFQ and the MEPA III screener, indicating that a shorter dietary assessment tool, the MEPA III screener, can accurately measure overall accordance to the MedDiet. Weave, et al. reported agreement between food groups based on the MEPA III scores and those from FFQ MEPA III components. To further explore the acceptability of the MEPA III screener among patients with PD, the exit questionnaire was updated to better quantify usability. The scoring system provided a way to compare the sum of individual's responses on the same quantifiable scale since each question has a numerical score. Each question was then summed to create an overall score of acceptability. This type of usability scoring was used previously by Albar, et al. and Franco, et al., in both adolescent and adult populations [12,13]. Albar, et al. used SUS to assess usability in Myfood24, an online 24 hour recall created in the U.K. Adolescents completed the beta and live version of Myfood24 and completed a usability questionnaire where a score of 68 out of 100 or above was considered "above average" rating of the tool. The mean SUS score was 66 out of 100 for the beta version and 74 out of 100 for the live version of Myfood24, showing an increase in acceptability in the adolescent sample. Franco, et al. also used SUS questionnaires to evaluate usability of online dietary assessment tools in adults. A median score of 75 out of 100 for usability of the online FFQ, EatWellQ8, indicating that acceptability of the tool was "good" according to researchers. In this study average scores of 49.7 and 49.1 for paper and online versions were observed, respectively, and in stability study component, a median score of 50 was obtained. Comparing scores proportionally, the SUS of 50 out of a possible 65 showed a 76.9% rate of acceptability which is comparable to previous studies [12,13] using a similar rating scale for participants. The MEPA III screener performed well and at a similar level of acceptability as prior online dietary assessment tools. Thus, in this sample of adults with earlier stages of PD (most participants were HY Stage 2), both the paper and online MEPA III screeners were deemed acceptable through the measure of SUS.

In these two different study samples, feasibility was also demonstrated. Completion times for the screener that exceed 15 minutes would suggest that the tool was too complex or confusing for participants. There was also a question on the exit questionnaire addressing the participant's perception of the time required to fill out the screener which allowed the recording of personal opinion in addition to the recorded time. This question showed that majority of participants perceived that the MEPA III screener was "just right" in the time required to complete the tool. Few reports in the literature regarding other diet screeners included completion time as a measure of patient usability. The MEDAS screener [14] did not report completion time, though Schröder, et al. described their tool as being "short" in containing 14 questions of food frequency consumption. The limited reporting on completion time is likely due to the variability in the definitions of "acceptability" and "feasibility." Both terms are used interchangeably in current literature, but typically used to describe different modes of feedback or participant responses rather than recorded time. Thus, for this study, current researchers defined both terms to collect participant opinion and an objective measure of time.

MEPA III scores of participants in both sub-studies reflect low accordance to the MedDiet overall. Median scores of 11 points out of 21 in the first study, and 12 points out of 21 in second. These scores are comparable to the MEPA III scores recorded in previous studies of this screener in a similar group of participants [1], with average scores of 10.7 ± 2.7, with an overall range of 6 to 16. Since sample demographics were similar between the current study and Weaver, et al., we might conclude that based on this screener, food behaviors of this sample appear more in line with a Western diet pattern than that of the MedDiet. This could be an opportunity for nutrition education for incorporating healthy fats, leafy greens, and whole grains, while also limiting processed foods, meats, and sweetened beverages [7]. Based on current literature, this education may benefit heart health, maintenance of lean mass, gut health, and possibly preservation of cognition within an aging population, all relevant to patients with PD [8,9].

This study is one of the few in which acceptability, feasibility, and reliability of a MedDiet screener has been examined in this sample. This dietary assessment tool, and those structured similarly, could provide a brief means for clinicians to judge accordance to this diet pattern and be used to create personalized strategies to promote changes in dietary choices.

References

1. Weaver KE, Goldman JG, Ribbens N, et al. (2020) Use of an online Screener: the Mediterranean Diet Pattern for Americans-III in Older Patients with Parkinson's disease. J Nutrition Geriatrics Gerontology 39: 30-43.
2. Brooke J (1996) SUS: A 'Quick and Dirty' Usability Scale. In: PW Jordan, B Thomas, BA Weerdmeester, AL McClelland, Usability evaluation in industry. Taylor and Francis, London, 194: 189-194.
3. Barichella M, Cereda E, Pezzoli G (2009) Major nutritional issues in the management of Parkinson's disease. Movement Disorders: Official Journal of the Movement Disorder Society 24: 1881-1892.
4. Alcalay RN, Gu Y, Mejia-Santana H, et al. (2012) The association between Mediterranean diet adherence and Parkinson's disease. Mov Disord 27: 771-774.
5. Cassani E, Barichella M, Ferri V, et al. (2017) Dietary habits in Parkinson's disease: Adherence to Mediterranean diet. Parkinsonism Relat Disord 42: 40-46.
6. Maraki MI, Yannakoulia M, Stamelou M, et al. (2019) Mediterranean diet adherence is related to reduced probability of prodromal Parkinson's disease. Mov Disord 34: 48-57.
7. Radd-Vagenas S, Kouris-Blazos A, Singh MF, et al. (2017) Evolution of Mediterranean diets and cuisine: Concepts and definitions. Asia Pac J Clin Nutr 26: 749-763.
8. Tangney CC, Kwasny MJ, Li H, et al. (2011) Adherence to a Mediterranean-type dietary pattern and cognitive decline in a community population. Am J Clin Nutr 93: 601-607.
9. Tosti V, Bertozzi B, Fontana L (2018) Health benefits of the Mediterranean diet: Metabolic and molecular mechanisms. J Gerontol A Biol Sci Med Sci 73: 318-326.
10. Cerwinske LA, Rasmussen HE, Lipson S, et al. (2017) Evaluation of a dietary screener: The Mediterranean eating pattern for Americans tool. Journal of Human Nutrition and Dietetics 30: 596-603.
11. Kristal AR, Kolar AS, Fisher JL, et al. (2014) Evaluation of web-based, self-administered, graphical food frequency questionnaire. J Acad Nutr Diet 114: 613-621.
12. Albar SA, Carter MC, Alwan NA, et al. (2015) Formative evaluation of the usability and acceptability of myfood24 among adolescents: A UK online dietary assessments tool. BMC Nutrition 1.
13. Franco R, Alawadhi B, Fallaize R, et al. (2017) A web-based graphical food frequency assessment system: Design, development and usability metrics. JMIR Hum Factors 4: e13.
14. Schröder H, Fitó M, Estruch R, et al. (2011) A short screener is valid for assessing Mediterranean diet adherence among older Spanish men and women. J Nutr 141: 1140-1145.