DIABMEMORY is a telehealth system supporting patients suffering from diabetes mellitus in self- management of their disease. Patients are equipped with a mobile software application running on a Near-FieldCommunication (NFC)-enabled mobile phone to collect essential health parameters, such as glucose levels, blood pressure, well-being, body weight and physical activity on a regular basis. Data are transmitted to a central database, where physicians and diabetes experts can review incoming data and provide patients with a textual feedback (delivered as short text message) on a weekly basis. Physicians are furthermore provided with two default templates. Since 2010, 386 patients have transmitted more than 200,000 datasets. In this work we present a content analysis of 3,519 textual feedbacks written over the course of 9 months. The work is organized as follows: First, we introduce an annotation scheme following based on Taylor's model for information use. A random sample of 188 feedbacks was selected and annotated by two annotators with experience in the field of telehealth. Each feedback was annotated with one or more categories: Motivational, confirmational, personal or factual. Annotators' agreement was evaluated using Cohen's kappa. Second, we trained 6 classifiers (Naive Bayes) based on this annotation to automatically categorize the whole corpus. The classifiers were evaluated using 8-fold cross-validation and were applied to the whole corpus. Furthermore we performed a keyword-analysis extracting top-terms in the categories blood pressure (21 terms), food & nutrition (20 terms), weight (16 terms), medication (7 terms), physical activity (21 terms), well-being (3 terms) and glucose levels (26 terms). Results: Most feedbacks are of motivational nature (60.41 %). Out of 3519 feedbacks, only 19.26% contained confirmational content. In terms of content, the majority of the feedbacks refer to glucose levels (42.95%) and blood pressure 32.98%. While diet has a significant impact on diabetes, only 10.36% of the feedbacks refer to diet instructions. Conclusion: physicians actively use feedbacks, though it is not used to engage in an active dialogue with the patient. It still has to be investigated, why
the number of references to diet and nutrition is low compared to advice on glucose and blood pressure. 1
DIABMEMORY is a telehealth system supporting patients suffering from diabetes mellitus in selfmanagement of their disease. Patients are equipped with a mobile software application running on a Near-Field-Communication (NFC)-enabled mobile phone1 to collect essential health parameters, such as glucose levels, blood pressure, well-being, body weight and physical activity on a regular basis. Data are transmitted to a central database, where physicians and diabetes experts can review incoming data and provide patients with a textual feedback (delivered as short text message) on a weekly basis. Physicians are furthermore provided with two default templates.
Since 2010, 386 patients have transmitted more than 200,000 datasets. In this work we present a content analysis of 3,519 textual feedbacks written over the course of 9 months and discuss how these results can be used to improve the system in near future. 2
A random sample of 188 feedbacks was selected for manual annotation by two annotators with experience in the field of telehealth. Feedbacks were categorized following Taylor’s model for information use2. Each feedback was annotated with one or more of following categories: motivational (encouraging patients to pursue the current course of action), instrumental (providing patients with clear instructions), confirmational (actively retrieving more information), personal (personal engagement; chit-chat) or factual (precise numbers on a person’s health status).
Annotators’ agreement was evaluated using Cohen’s kappa.
We then trained 6 classifiers (Naive Bayes) based on this annotation to automatically categorize the whole corpus. The classifiers were evaluated using 8-fold crossvalidation. Furthermore we performed a keyword-analysis to gain a deeper insight into the content extracting top-terms in the categories blood pressure (21 terms), food & nutrition (20 terms), weight (16 terms), medication (7 terms), physical activity (21 terms), well-being (3 terms) and glucose levels (26 terms).
Out of 3,519 feedbacks written by 55 physicians, 481 (13.68%) referred to one of the two standard templates. Physicians used 29 words on average (minimum 1 word, maximum 89 words; standard deviation 18.79; maximum length of a message was limited to 500 characters). 3,331 feedbacks were automatically categorized. (Average F1-Measure for classifiers: 0.76; Cohen’s kappa for annotation: 0.89.) Table 1 illustrates the distribution of categories. Results of the content analysis are summarized in Table 2. The low number of confirmational feedbacks suggests that the physician-patient dialogue should be improved on a technical level. Furthermore, an analysis of the content found in category confirmational revealed that a large number of questions are related to diet. It can be argued, that proper documentation of nutrition should be incorporated into the system. Finally, the large number of references to activity and weight suggest, that better ways to track physical activity are required.
Since DIABMEMORY provides a module for medication management, the low number of references to medication might not come as a surprise. However, it also shows that feedback messages are not used to optimize dosage of insulin or oral antidiabetic drugs.
Physicians actively use the feedback mechanism provided in DIABMEMORY, mainly delivering motivational content. Default templates are rarely used. Physicians focus on glucose levels and blood pressure mainly, while feedback on well-being plays only a minor role.
It has yet to be investigated if these feedbacks have any health impact. Moreover, more research needs to be done to see, what type of feedback is most effective. Another interesting question could be to investigate, whether feedbacks could be used to predict compliance outcome (e.g. predicting drop-outs of the telehealth program).
The classifiers presented in this work could be used to support physicians in their daily work by high-lightening patients that need more attention. Patients that were provided with instructive and confirmational feedbacks could be prioritized, while patients receiving only motivational feedback could be lower ranked. Furthermore, these results can be used to provide physicians with more personalized default templates using methods of natural language generation.