Deconstructing Persuasive Strategies in Mental Health Apps Based on User Reviews using Natural Language Processing

Abstract

Text Mining is concerned with extracting interesting and significant patterns or knowledge from unstructured text data. In this paper, we applied the text mining approach using natural language processing (NLP) techniques, especially topic modelling (with automated topic labelling), in deconstructing the persuasive strategies implemented or employed by 100 mental health apps based on user reviews. We focus on the persuasive strategies in the primary task support category of the Persuasive Systems Design (PSD) framework. We used the Latent Dirichlet Allocation (LDA) topic modelling algorithm, in conjunction with semantic attributes, to achieve our goal. Our experimental results revealed that self-monitoring is the most employed persuasive strategy. Finally, we compare our findings with that obtained using manual coding method and found significant similarities.

Keywords

• Text mining
• Persuasive strategy
• Mobile apps
• Mental health
• User reviews
• Natural language processing
• Topic modelling

-

Smartphone ownership continues to increase rapidly in both emerging and advanced economies [ 1, 2 ]. Evidence shows that there are 3.06 billion smartphone users globally in 2018 and this figure is expected to surpass 4 billion by 2025 [ 3 ]. As a result, mobile applications (or apps) are proliferating and download rate is projected to keep increasing steadily. For instance, 2.8 million and 2.2 million apps are available for download on Google Play and App Store respectively [ 4 ] and the number of downloads on Google Play was 21.3 billion [ 5 ] as at 2019. Among these are mobile health (mHealth) apps that leverage embedded sensors in smartphones and connected wearables, as well as technology-assisted self-reporting, to deliver health interventions to patients [ 6 ]. For instance, mental health apps can use global positioning system (GPS) and accelerometer data to compute physical and spatial activity measures for detecting depressive symptoms and anxiety [ 7, 8 ]. In addition, integration with wearable sensors for real-time monitoring of heart rate, blood pressure, temperature, skin conductance, etc. can help identify stress [ 9, 10 ]. This shows that mHealth apps have a considerable potential to support early detection and treatment of medical conditions, thereby promoting healthy lifestyles and behaviours in patients, including improved mental health. In addition, mHealth apps designers utilize persuasive techniques that influence patients to imbibe these health behaviours in their daily lives.

Over the years, previous research has conducted systematic reviews to identify persuasive strategies employed in mHealth apps [ 11–13 ]. The widely used approach is to have expert reviewers download the apps and manually code them using the Persuasive Systems Design (PSD) framework [ 14 ] or behaviour change theories [ 15 ]. While this approach may seem to be effective, it can be very costly or impracticable for wider studies that involve hundreds or thousands of apps. To address this issue, we applied the text mining approach using natural language processing (NLP) techniques and topic modelling (with automated topic labelling) on app reviews (i.e., user reviews or comments about apps) which is a more efficient and practicable, but less costly, way of detecting the persuasive strategies implemented in large number of mobile apps. Although research in this area is still emerging, it has the potential to yield successful and reliable outcomes since text mining has been widely applied on social media posts and online/app reviews to generate insights for addressing challenges (or supporting decision making) in many domains [ 16–18 ], including health [ 19, 20 ]. Furthermore, user reviews of mobile technologies capture information about functionality [ 21 ] which is useful in detecting the persuasive features present in apps.

Our methodology combines several well-known computational techniques as follows: (1) Extract user reviews for mental health apps on both Google Play and App Store using Heedzy tool1 [ 22 ], and preprocess the data using NLP techniques to prepare it for analysis; (2) Apply the Latent Dirichlet Allocation (LDA) topic modelling algorithm on user reviews to detect main topics or themes; (3) Label each topic with persuasive strategies (in the primary task support category of the PSD framework) using semantic attributes depicting each strategy. Next, we compare our findings with that obtained using manual coding method and found significant similarities. Finally, we discuss the implication of our findings. 2

Related Work

Existing research has used the Persuasive Systems Design (PSD) framework or Behaviour Change Techniques (BCTs) to deconstruct the persuasive strategies employed in mobile apps. For example, Alqahtani et al. [ 13 ] reviewed mental health apps to identify persuasive strategies employed. Similarly, Chang et al. [ 23 ] and Oyebode et al. [ 12 ] reviewed mental health apps and mHealth apps respectively using the PSD framework, while Langrial et al. [ 24 ] reviewed apps for personal wellbeing using the same framework. However, all the researchers coded the apps manually which is less efficient, time-consuming, and prone to coding mistakes.

Al-Ramahi et al. [ 25 ] adopted a text mining approach to identify existing persuasive strategies and propose new ones. They applied the topic modelling technique (using LDA algorithm) on online user reviews to extract main topics or themes de

User reviews on both stores were extracted in May 2018 using the paid version of the tool scribing the reviews. However, they manually labelled the topics with appropriate persuasive strategies based on the words associated with each topic. This manual labelling of topics is also less efficient and automated options should be considered.

To address these gaps, our work applies the text mining approach using NLP techniques and topic modelling (with automated topic labelling) in deconstructing the persuasive strategies (in the primary task support category of the PSD framework) implemented or employed by mental health apps, thereby eliminating manual coding of apps and manual labelling of topics. The automatic labelling was achieved using semantic attributes that represent various persuasive strategies. 3

Methodology

The main goal of this paper is to identify persuasive strategies employed by mental health apps based on user reviews. To achieve this, we used well-known computational techniques. As shown in Figure 1, the steps taken to achieve our goal are summarized below. 1. We extracted user reviews for mental health apps on both Google Play and App

Store using Heedzy tool. 2. We preprocessed the data using NLP techniques to prepare it for analysis. 3. We classified each review as either positive, negative, or neutral using user ratingbased criteria. 4. We vectorized the user reviews (i.e., converting them into numeric form) using the

Term Frequency Inverse Document Frequency (TF-IDF) weighting technique. 5. We applied the Latent Dirichlet Allocation (LDA) algorithm on the vectorized user reviews to detect main topics or themes. 6. We labelled each topic with persuasive strategies using semantic attributes depicting each strategy.

Data Collection (user reviews)

Data Vectorization (TF-IDF)

Topic Modelling Latent Dirichlet Allocation (LDA) algorithm

Data Preprocessing

Sentiment Classification

Topic Labelling

Persuasive Strategies

Semantic Attributes

Data Collection

To identify eligible apps, we first performed a search on Google Play (for Android apps) and App Store (for iOS apps) using various keywords including anxiety, stress, depression, emotion, mental health, and mood. We got a total of 183 and 254 apps from Google Play and Apple Store respectively as search results. Secondly, we excluded apps whose description do not relate to mental health, apps with less than five user reviews, and non-English apps. Afterwards, we collected 101,715 user reviews for 105 eligible apps using the Heedzy tool. If an app is published on both Google Play and App Store, we merged the reviews on both platforms. 3.2

Data Preprocessing

Next, we applied the following preprocessing steps using NLP techniques to clean the data and prepare it for analysis: 1. Convert words to lowercase 2. Reduce repeated characters (e.g. “goooood” becomes “good”) 3. Remove numbers 4. Expand contractions (e.g. “there's” becomes “there is”, etc.) 5. Replace slangs with English words using online slang dictionaries [ 26, 27 ] which contain 5,434 entries combined 6. Remove punctuation, special characters, and extra whitespaces 7. Remove stopwords (e.g. the, an, will, shall, let, may, can, it, with, of, this, and, as, etc.) 8. Lemmatize words using the WordNet Lemmatizer (which is part of the nltk module of Python and uses WordNet [ 28 ] behind the scene) so that words can be converted to their root form (e.g., “better” becomes “good”, while “statistics” becomes “statistic”) 9. Remove duplicates After data preprocessing, the total reviews reduced to 88,125 across 104 apps. 3.3

Sentiment Classification

Next, we classified each review as either positive, negative, or neutral sentiment polarity based on user ratings. On Google Play and App Store, users assign star ratings to apps on a scale of 1 to 5 (where 1 star represents “very dissatisfied” and 5 stars represents “very satisfied”). In our dataset, a total of 68,247 reviews have user ratings (representing 77.4% of total reviews). Table 1 shows user ratings and the corresponding sentiment polarity, as well as the total reviews for each rating. This approach has been used by previous research, such as [ 21, 29 ], to determine the sentiment polarity of app reviews. Since feature request is one of the most common issue type in negative reviews [ 29, 30 ], we excluded them from our analysis. We retained only positive reviews (n=57705) since they mostly reflect user opinions or experience about features and strategies already implemented in the apps. The positive reviews are distributed across 100 apps, as shown in Table 2. To identify main topics or themes describing the reviews, we applied Latent Dirichlet Allocation (LDA) which is a widely used and efficient topic modelling algorithm [ 31, 32 ]. We implemented the algorithm in Python programming language. Prior to applying the LDA, we vectorized the documents (i.e., reviews) using the popular Term Frequency Inverse Document Frequency (TF-IDF) weighting scheme since it considers both frequency and relevance when assigning weight to words [ 33 ]. The idea is to pass the vectorized reviews for each app to the LDA algorithm to generate topics.

The LDA algorithm returns top topics in the reviews, along with top words for each topic (where K and N are set to 50 and 10 respectively). We set the value of to 50, based on previous research which shows that perplexity reduces with an increase in number of topics but later converges to a fixed value (approximately 50) [ 25 ].

In summary, for each of the 100 apps, we retrieved top 50 topics and top 10 words that describe each topic. The words corresponding to each topic will be used to determine appropriate labels (i.e., persuasive strategies), as described in the next section. 3.5

Topic Labelling, Semantic Attributes, and Persuasive Strategies

The final step is to infer persuasive strategies from the words associated with the topics, and then assign the persuasive strategies as topic labels. Instead of achieving this manually, we employed the approach proposed by [ 34 ] which involves assessing the existence of persuasive strategies using semantic attributes. We generated the semantic attributes for each persuasive strategy in the primary task support category of the PSD framework using the WordNet lexical database [ 28 ]. The attributes were verified and validated by two persuasive technology experts for appropriateness. Table 3 shows the persuasive strategies and the corresponding semantic attributes.

Semantic Attributes personalize, relevant, personalization, personal, individual, profile, personality, personalise, individualize, personify, relevance, suitable, suit, fit, adjust, change, adjustable, edit, editable, amend, modify, flexible, control, customization, customize, customise, customizable, customisable, modifiable, changeable, adapt, adaptable, refine, alter, tailor, tailored track, statistics, statistic, measure, progress, goal, history, view, display, activity, analysis, record, monitor, graph, chart, log, logging, journal, diary, duration, speed, pace, time, insight, insightful, recording, journaling simple, simplify, automatically, automatic, quickly, immediately, instantly, quick, instant, immediate, simplicity, straightforward step-by-step, guide, stepwise, gradual, step, instruction, procedure, procedural, process, journey, stage, plan, guided, guidance simulate, virtual, imitate, visualization, sound, audio, video, image, observe, simulator, effect, game, animation, environment, voice, practice, train, practise, learn, drill, rehearsal, intro, introduction, rehearse We developed a Python program to match the words corresponding to each topic with the semantic attributes, and then label the topic with the appropriate persuasive strategy (or strategies) based on the matching attribute(s). 4

Results

Based on our experimental results, the self-monitoring persuasive strategy emerged as the most employed overall (n=92/100), followed by personalization and tailoring (n=83/100) and simulation and rehearsal (n=81/100), as shown in Figure 2. However, reduction (n=77/100) followed by tunneling (n=53/100) are the least employed strategies.

Table 4 presents sample topics (out of the 5000 extracted topics) for some of the apps including the top 10 words associated with each topic, the matching semantic attributes, topic label (i.e., persuasive strategy (PS)), and sample user reviews. “It gives good reminders for a type A personality who needs to learn to slow down”2 [R37] “The animation which let me know when we breathe and when we hold works magically” [R42] “Great little application to learn deep breathing without having to count…” [R50] “It has everything I need to keep an anxiety diary…” [R290] “Awesome application beautiful designs and it is a great way to keep on track of the moods!” [R253] “An amazing application I just loved the way it gives me my emotional statistics…” [R391] 3 Minute Mindfulness Cognitive Diary CBT Self-Help Daylio

User reviews are included verbatim throughout the paper, including spelling and grammatical mistakes.

Comparing our results with that of manual approach

Next, we compared our findings with the results obtained by Alqahtani et al. [ 13 ] after manually coding the mental health apps using PSD framework and the Behaviour Change Techniques (BCTs). To aid comparison, we refer to our method as “automated approach”, and Alqhatani et al.’s method as “manual approach”. According to both results, self-monitoring emerged as the most employed strategy in both automated and manual approaches. Similarly, personalization (called “personalization and tailoring” in the automated approach) emerged as the second most employed persuasive strategy in both approaches. However, while reduction and tunneling are the least employed strategies in the automated approach, rehearsal and reduction are the least employed in the manual approach.

In summary, both approaches agreed that self-monitoring and personalization are the top 2 (most employed) persuasive strategies in mental health apps. On the other hand, both approaches slightly agreed on the least employed strategies which may be due to human coding errors in the manual approach. 5

Discussion

In this paper, we applied the text mining approach in identifying persuasive strategies employed by mental health apps based on user reviews. The results of our experiment revealed that self-monitoring is the most employed strategy, closely followed by personalization and tailoring. This aligns with the results of a prior research that applied the manual method of deconstructing persuasive strategies in mental health apps (see Section 4.1). We discuss the implications of our findings in subsequent sub-sections. 5.1

Self-monitoring and Personalization as key strategies for mental health interventions

Self-monitoring is an essential requirement for digital health interventions, especially mHealth apps for emotional and mental wellbeing. For example, research shows that apps for self-monitoring of mood (i.e., mood tracking) increase emotional selfawareness which causes a reduction in depressive and anxiety symptoms [ 35–37 ]. In addition, Hetrick et al. included the mood monitoring feature, as well as personalized interventions, in their app design for young adults to aid self-treatment of depression [ 38 ]. Thus, by employing both self-monitoring and personalization, developers can deliver effective persuasive technological interventions that promote better health in patients with mental health issues. 5.2

Text Mining as a useful approach for persuasive technology research

Text mining can be useful for research involving persuasive technologies. Most persuasive technology researchers still rely on empirical studies that require manual efforts, and this technique usually limits the scope of research due to various factors such as cost, study population size and diversity challenges, generalizability issues due to limited data, etc. In addition, manual approaches are prone to errors. Text mining algorithms usually leverage large textual datasets (such as social media data, user reviews on app stores, open data, organizational data, etc.) for analysis. As a result, their outputs can uncover valuable and useful insights that may be impossible to achieve through other methods. Text mining has been applied in healthcare [ 39–41 ] and other domains [ 42, 43 ] over the years, and can be exploited by PT researchers to generate data-driven insights that inform the design, development, and evaluation of persuasive systems. We demonstrated the applicability of text mining on mental health app reviews from persuasiveness standpoint, but it can be extended to other areas of interest. 5.3

User Reviews as a way of evaluating persuasive apps

Two of the limitations hindering the advancement of the field of persuasive technology are the lack of study on the effectiveness of persuasive apps over a long-term and the limited number of participants that are usually involved in the app evaluation [ 44, 45 ]. Most existing persuasive apps in the literature did not evaluate the effectiveness of their apps, and even those that did, the evaluation is often in a controlled environment and for a short-term. This is mainly due to the cost of conducting a long-term study in terms of time and other resources. Also, most evaluation results are not generalizable due to limited number of participants and their similarity. We argue that our findings, which show the possibility of applying text mining on user reviews to uncover insights about the apps, suggest that user reviews could be a reliable and costeffective alternative for evaluating the effectiveness of persuasive apps both in short and long-term. The findings are more generalizable considering the diversity of reviewers. Hence, public user reviews of apps and text mining are promising techniques for evaluating the effectiveness of persuasive applications. 5.4

Tunneling and Reduction as potential strategies designers of mental health apps should employ

Our findings revealed that reduction followed by tunneling are the least employed strategies. This is surprising since mental health patients are often advised to avoid stressful conditions, as well as complex tasks that may raise their stress levels and worsen their health. Also, lack of guidance in mental health apps can impair concentration and make users to be easily frustrated [ 46 ]. Therefore, designers of mental health apps should consider their users by reducing tasks into simpler steps (reduction) and providing necessary guidance (tunneling) until they complete those tasks. 6

Limitations

First, user reviews or comments about each app reflect opinion about features the users have interacted with or noticed. However, it is possible there are other features not highlighted in the reviews since they may not be noticeable or personally relevant for a user. As a result, certain persuasive strategies could be more obvious or prevalent than others. Second, the one-word semantic attributes used in our analysis may not be exhaustive; hence, including phrasal attributes may help to boost coverage. 7

Conclusion and Future Work

In this paper, we applied the text mining approach using natural language processing (NLP) techniques and topic modelling (with automated topic labelling) in deconstructing the persuasive strategies (in the primary task support category of the PSD framework) implemented or employed by mental health apps based on user reviews. We used the Latent Dirichlet Allocation (LDA) algorithm and Semantic Attributes to achieve our goal. Our experimental results revealed that self-monitoring is the most employed persuasive strategy overall, followed by personalization and tailoring, and simulation and rehearsal. However, reduction followed by tunneling emerged as the least employed strategies.

In our future work, we plan to conduct a large-scale study on mental health apps, as well as apps in other health domains, based on user reviews using the text mining approach to investigate the persuasiveness and effectiveness of these apps. We will extend the semantic attributes to cover all the 28 persuasive strategies of the PSD framework for our study. Finally, we will explore negative reviews to investigate the effect of how persuasive strategies are implemented on overall user experience, and then recommend design solutions to address identified gaps.

References

1. Poushter , J.: Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies . ( 2016 ). http://www.pewglobal.org/ 2016 /02/22/smartphone-ownership -andinternet-usage-continues-to-climb-in-emerging-economies/

2. Taylor , K., Silver , L. : Smartphone Ownership Is Growing Rapidly Around the World, but Not Always Equally . ( 2019 ). https://www.pewresearch.org/global/2019/02/05/smartphoneownership-is -growing-rapidly-around-the-world-but-not-always-equally/

3. GSM Association: The Mobile Economy 2019 . ( 2019 ). https://www.gsmaintelligence.com/research/?file=b9a6e6202ee1d5f787cfebb95d3639c5& download

4. Blair , I. : Mobile App Download and Usage Statistics. ( 2019 ). https://buildfire.com/appstatistics/

5. Clement , J.: Mobile app usage - Statistics & Facts. ( 2019 ). https://www.statista.com/topics/1002/mobile-app-usage/

6. Areán , P.A. , Ly , K.H. , Andersson , G. : Mobile technology for mental health assessment . Dialogues Clin. Neurosci . 18 , 163 - 169 ( 2016 )

7. Tucker , I. , Smith , L. -A.: Topology and mental distress: self-care in the life spaces of home . J. Health Psychol . 19 , 176 - 83 ( 2014 ). https://doi.org/10.1177/1359105313500260

8. Mason , M.J. , Korpela , K. : Activity spaces and urban adolescent substance use and emotional health . J. Adolesc . 32 , 925 - 939 ( 2009 ). https://doi.org/10.1016/j.adolescence. 2008 . 08 .004

9. Sano , A. , Taylor , S., McHill , A.W. , Phillips , A.J.K. , Barger , L.K. , Klerman , E. , Picard , R.: Identifying objective physiological markers and modifiable behaviors for self-reported stress and mental health status using wearable sensors and mobile phones: Observational study . J. Med. Internet Res . 20 , e210 ( 2018 ). https://doi.org/10.2196/jmir.9410

10. Yu , B. , Hu , J. , Funk , M. , Feijs , L.: DeLight: biofeedback through ambient light for stress intervention and relaxation assistance . Pers. Ubiquitous Comput . 22 , 787 - 805 ( 2018 ). https://doi.org/10.1007/s00779-018-1141-6

11. Fadhil , A. , Wang , Y. : Health Behaviour Change Techniques in Diabetes Management Applications: A Systematic Review . ( 2019 ). http://arxiv.org/abs/ 1904 .09884

12. Oyebode , O. , Ndulue , C. , Alhasani , M. , Orji , R.: Persuasive Mobile Apps for Health and Wellness : A Comparative Systematic Review . In: International Conference on Persuasive Technology. pp. 1 - 12 ( 2020 )

13. Alqahtani , F. , Khalifah , G. Al, Oyebode , O. , Orji , R.: Apps for Mental Health : An Evaluation of Behavior Change Strategies and Recommendations for Future Development . Front. Artif. Intell. 2 , 1 - 11 ( 2019 ). https://doi.org/10.3389/frai. 2019 .00030

14. Oinas-Kukkonen , H. , Harjumaa , M. : Persuasive Systems Design: Key Issues, Process Model, and System Features . Commun. Assoc. Inf. Syst . 24 , 96 ( 2009 )

15. Orji , R. , Orji , F. , Oyibo , K. , Ajah , I.A. : Personalizing health theories in persuasive game interventions to gamer types: An African perspective . In: Proceedings of the Second African Conference for Human Computer Interaction: Thriving Communities . pp. 45 - 56 ( 2018 )

16. Du , X. , Kowalski , M. , Varde , A.S. , de Melo, G. , Taylor , R.W.: Public opinion matters: Mining Social Media Text for Environmental Management . ACM SIGWEB Newsl . 1 - 15 ( 2020 ). https://doi.org/10.1145/3352683.3352688

17. Pejić Bach , M. , Krstić , Ž., Seljan , S. , Turulja , L. : Text Mining for Big Data Analysis in Financial Sector: A Literature Review . Sustainability . 11 , 1277 ( 2019 ). https://doi.org/10.3390/su11051277

18. Kuhzady , S. , Ghasemi , V. : Factors influencing customers' satisfaction and dissatisfaction with hotels: A text-mining approach . Tour. Anal . 24 , 69 - 79 ( 2019 ). https://doi.org/10.3727/108354219X15458295631972

19. Bollegala , D. , Maskell , S. , Sloane , R. , Hajne , J. , Pirmohamed , M. : Causality Patterns for Detecting Adverse Drug Reactions From Social Media: Text Mining Approach . JMIR public Heal. Surveill . 4 , e51 ( 2018 ). https://doi.org/10.2196/publichealth.8214

20. Toulis , A. , Golab , L. : Social media mining to understand public mental health . In: Data Management and Analytics for Medicine and Healthcare . pp. 55 - 70 . Springer Verlag ( 2017 )

21. Gebauer , J. , Tang , Y. , Baimai , C. : User requirements of mobile technology: Results from a content analysis of user reviews . Inf . Syst. E-bus. Manag . 6 , 361 - 384 ( 2008 ). https://doi.org/10.1007/s10257-007-0074-9

22. Heedzy: Download app reviews from iTunes App Store & Google Play, https://heedzy.com/

23. Kaasinen , E. , Mattila , E. , Lammi , H. , Kivinen , T. , Välkkynen , P. : Technology Acceptance Model for Mobile Services as a Design Framework . In: Human-Computer Interaction and Innovation in Handheld, Mobile and Wearable Technologies . pp. 80 - 107 . IGI Global ( 2011 )

24. Langrial , S. , Lehto , T. , Oinas-Kukkonen , H. , Harjumaa , M. , Karppinen , P. : Native Mobile Applications For Personal Well- Being: A Persuasive Systems Design Evaluation . In: PACIS 2012 Proceedings ( 2012 )

25. Al-Ramahi , M.A. , Liu , J. , El-Gayar , O.F. : Discovering design principles for health behavioral change support systems: A text mining approach . ACM Trans. Manag. Inf. Syst. 8 , ( 2017 ). https://doi.org/10.1145/3055534

26. Slang Words Dictionary, https://raw.githubusercontent.com/sifei/Dictionary-forSentiment-Analysis/master/slang/acrynom.csv

27. Slang Lookup Table, https://raw.githubusercontent.com/felipebravom/StaticTwitterSent/master/extra/SentiStren gth/SlangLookupTable.txt

28. Fellbaum , C. : WordNet and wordnets . In: Encyclopedia of Language and Linguistics . pp. 665 - 670 . Elsevier Science ( 2005 )

29. McIlroy , S. , Ali , N. , Khalid , H., E. Hassan , A. : Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews . Empir. Softw. Eng . 21 , 1067 - 1106 ( 2016 ). https://doi.org/10.1007/s10664-015-9375-7

30. Khalid , H. , Shihab , E. , Nagappan , M. , Hassan , A.E. : What do mobile app users complain about ? IEEE Softw. 32 , 70 - 77 ( 2015 ). https://doi.org/10.1109/MS. 2014 .50

31. Blei , D.M. , Ng , A.Y. , Jordan , M.I. : Latent Dirichlet allocation . J. Mach. Learn. Res . 3 , 993 - 1022 ( 2003 ). https://doi.org/10.1016/b978-0 -12-411519-4 . 00006 - 9

32. Hoffman , M.D. , Blei , D.M. , Bach , F. : Online Learning for latent Dirichlet allocation (Supplementary Material) . Nature . 1- 9 ( 2010 ). https://doi.org/10.1.1.187.1883

33. Juan Ramos: Using TF-IDF to determine word relevance in document queries . Proc. first Instr. Conf. Mach. Learn . 242 , 133 - 142 ( 2003 )

34. Yoganathan , D. , Kajanan , S. : Designing fitness apps using persuasive technology: A text mining approach . Pacific Asia Conf. Inf. Syst. PACIS 2015 - Proc. ( 2015 )

35. Bakker , D. , Rickard , N.: Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism . J. Affect . Disord. 227 , 432 - 442 ( 2018 ). https://doi.org/10.1016/j.jad. 2017 . 11 .016

36. Morris , M.E. , Kathawala , Q. , Leen , T.K. , Gorenstein , E.E. , Guilak , F. , Labhard , M. , Deleeuw , W.: Mobile therapy: Case study evaluations of a cell phone application for emotional self-awareness . J. Med. Internet Res . 12 , 1 - 21 ( 2010 ). https://doi.org/10.2196/jmir.1371

37. Kauer , S.D. , Reid , S.C. , Crooke , A.H.D. , Khor , A. , Hearps , S.J.C. , Jorm , A.F. , Sanci , L. , Patton , G.: Self-monitoring using mobile phones in the early stages of adolescent depression: Randomized controlled trial . J. Med. Internet Res . 14 , 1 - 17 ( 2012 ). https://doi.org/10.2196/jmir.1858

38. Hetrick , S.E. , Robinson , J. , Burge , E. , Blandon , R. , Mobilio , B. , Rice , S.M. , Simmons , M.B. , Alvarez-Jimenez , M. , Goodrich , S. , Davey , C.G. : Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young eople with major depression, suicidal ideation, and self-harm . J. Med. Internet Res . 20 , 1 - 14 ( 2018 ). https://doi.org/10.2196/mental.9041

39. Boit , J. , El-Gayar , O. : Topical Mining of Malaria Using Social Media. A Text Mining Approach . In: Proceedings of the 53rd Hawaii International Conference on System Sciences. Hawaii International Conference on System Sciences ( 2020 )

40. Simmons , M. , Singhal , A. , Lu , Z. : Text mining for precision medicine: Bringing structure to ehrs and biomedical literature to understand genes and health . In: Advances in Experimental Medicine and Biology . pp. 139 - 166 . Springer New York LLC ( 2016 )

41. Payton , F.C. , Yarger , L.K. , Pinter , A.T. : Text Mining Mental Health Reports for Issues Impacting Today's College Students: Qualitative Study . JMIR Ment. Heal . 5 , e10032 ( 2018 ). https://doi.org/10.2196/10032

42. Deng , T. , Lee , Y.J. , Xie , K. : Management Responses to Online Hotel Reviews: Text Mining to Lift Reputation and Revenue . SSRN Electron. J. ( 2019 ). https://doi.org/10.2139/ssrn.3416224

43. Sezgen , E. , Mason , K.J. , Mayer , R.: Voice of airline passenger: A text mining approach to understand customer satisfaction . J. Air Transp. Manag . 77 , 65 - 74 ( 2019 ). https://doi.org/10.1016/j.jairtraman. 2019 . 04 .001

44. Orji , R. , Moffatt , K. : Persuasive technology for health and wellness: State-of-the-art and emerging trends . Health Informatics J . 24 , 66 - 91 ( 2018 ). https://doi.org/10.1177/1460458216650979

45. Orji , R. , Mandryk , R.L. , Vassileva , J.: Improving the efficacy of games for change using personalization models . ACM Trans. Comput. Interact . 24 , ( 2017 ). https://doi.org/10.1145/3119929

46. Alqahtani , F. , Orji , R.: Usability Issues in Mental Health Applications. Adjun. Publ. 27th Conf. User Model. Adapt. Pers. - UMAP'19 Adjun . 343 - 348 ( 2019 ). https://doi.org/10.1145/3314183.3323676