Keeping users engaged with mHealth applications is important but dificult to achieve. We describe the development of a smartphone-based application designed to promote health and wellness in church communities, along with mechanisms explicitly designed to maintain engagement. We evaluated religiously tailored techno-spiritual engagement mechanisms, including a prayer posting wall, pastor announcements, an embodied conversational agent for dialogue-based scriptural reflections and health coaching, and tailored push notifications. We conducted a four-week pilot study with 25 participants from two churches, measuring high levels of participant acceptance and satisfaction with all features of the application. Engagement with the app was higher for users considered to be more religious and correlated with the number of notifications received. Our findings demonstrate that our tailored mechanisms can increase engagement with an mHealth app.
[12, 13]. This section explores research specifically on the Based on these participatory design sessions, we deuse and efectiveness of these strategies for promoting veloped an app with multiple features, including an ECA engagement with mHealth systems. named Clara (Figure 1a). When the app is launched for
Studies have shown that mobile applications for pro- the first time, Clara introduces herself to the user and moting healthy behaviors that implement features for provides an overview of the features and intervention social support can have positive efects on health out- content within the app. On subsequent launches, the comes [10, 11]. For example, a formative evaluation of app shows a menu of options for quick access to its core a mobile cardiovascular health app designed in partner- features (Figure 1b). One of the options initiates a conship with church-going African Americans found that a versation with Clara. In the first session of the day, Clara forum-style social networking feature was supportive, en- greets the user and engages in rapport building with sojoyable, encouraging, and motivating to maintain healthy cial chat on topics ranging from the weather to movies behaviors, such as diet and physical activity [14]. and sharing important events that have occurred since
Push notifications have also been used in mHealth apps the last time they spoke such as their church pastor postto increase engagement. Research has shown that users ing an announcement. Finally, Clara presents a list of who received push notifications have better health out- topics that users can choose to talk about, such as exercomes or higher engagement with mHealth apps overall cise, nutrition, or scriptural meditation. [2] than those not receiving any notifications in vari- The mHealth functions of the app promote nutrition, ous mHealth contexts, such as mental disorder treatment including fruit and vegetable consumption and healthy [12] or weight loss maintenance [13]. Based on previ- hydration, and physical activity. These are primarily ous research, components such as notification content provided by coaching from the ECA following the stagescan be important factors to influence responsiveness to of-change model as a theoretical framework [20], with notifications [ 15]. Bidargaddi et al. [16] showed that no- goal setting and behavior logging for users in later stages tifications with tailored health suggestions improved the of change. Several of the topics related to healthy living overall likelihood of users interacting with an mHealth are designed to provide users with the tools they need app while tailored insights from self-monitored data led to feel motivated to improve their health, to set goals, to greater engagement amongst frequent app users and and to keep track of their progress. Each topic has an showed the importance of tailoring content to increase introduction where Clara provides information on the engagement. topic at hand, for example, the current recommendations for exercise or fruit consumption, as well as getting to know the user’s current engagement in these health3. Application Design related behaviors. Clara encourages the user to engage in conversation about health behavior change by asking them what they think about exercise or healthy eating, their reasons for change and why they are interested in change, how they would like Clara to help them, and the benefits of the health behavior for them personally. The In order to successfully implement the mHealth app in our stakeholders’ communities, we conducted nine focus groups to better understand the participants’ church communities and to gain insights into how to efectively tailor the app [17, 18, 19]. conversations are tailored according to the user’s stage of change, which indicates their intentions to change their health behavior [20].
information and engage with them through regular updates (Figure 1c). Whenever a new pastor announcement is received, the user receives a push notification if they are not currently using the app, and sees a numbered indicator in the main app screen showing the number of new and unread announcements.
implemented the Bible Story of the Day and Scriptural Meditation features. When users select the option to talk to Clara, they are provided with an option to either hear a Bible story or engage in scriptural meditation led by Clara. For the Bible Story of the Day, all users are presented with the same Bible story for a particular day to encourage discussing the story with others in their church. Clara provides a summary of the story and allows the user to engage by providing various responses to the story. For the Scriptural Meditation, users have the option to choose the Bible passage they would like to read and are guided by Clara through the process of reflection and meditation.
The ECA speaks using a synthesized voice and her dialogue is driven by a hierarchical task network-based dialogue manager. Her non-verbal behavior is automatically generated using BEAT [21]. We implemented a religiously tailored social chat module and representations of agent-led faith-based activities, as well as designing the agent dialogue to include both a community-situated greeting (e.g., “Hello Sister Kathy!”) and farewell (e.g., “Blessings to you. Take care!”). At the end of each conversation, Clara shares a scripture verse in the same way that church members often end church-based activities.
to as the Prayer Center that enables users to connect
with their fellow congregants. The first feature is the
Prayer Wall, where users can post textual messages that
include requests for prayers or praise (e.g., expressions
of gratitude or joy) to share with other members of their
church (Figure 1d). The messages are displayed as cards, 4.5. Push Notifications
similar to posts on a forum, and can be updated, replied The final engagement mechanism we designed is the
to, and added to one’s own private list of prayers. The ability to send push notifications to users’ mobile devices.
second feature is a private Prayer List to remind users of We tailored the push notifications to encourage users to
the prayers they intend to say for others, allowing use engage with the health features or the other religiously
of the app as a tool to support private prayer practices. tailored engagement mechanisms. We used Google’s
Users can add their own prayers to the list manually and Firebase Cloud Messaging to send three diferent types of
remove them or add prayer requests and praise reports push notifications to both iOS and Android devices with
from the Prayer Wall. a server-based notification service and decision rules.
Reminder Notifications: To remind users to regularly
4.3. Pastor Announcements engage with Clara, we send check-in reminders to talk
to her if the user has not used the app or received other
To further engage app users with their church commu- notifications in the past 24 hours. We religiously tailored
nity, we developed a mechanism for the church pastors the push notifications by randomly assigning certain
to reach out to members in a one-way broadcast com- notifications to include a Bible verse. To further diversify
munication. These Pastor Announcements give church the content of these notifications, we randomly delivered
leaders the means to provide members with news and diferent variations of messages by framing some in first
person with an image of Clara and others in third person Technology literacy was assessed at the beginning of the
without an image. Figure 2 shows some examples of the study with the following questions: (
Social Notifications: We also designed notifications day?” and (
Health Notifications: Notifications related to the Satisfaction with each engagement mechanism and the health behavior interventions were sent when users had overall app was assessed using a four-item, seven-point set a physical activity or nutrition goal. Reminder noti- composite measure (Anchors 1=“not at all satisfied”, Anifcations were sent when they should check-in with the chor 7=“very satisfied”) per mechanism. The number of ECA regarding their physical activity goal, while notifi- logins, sessions with the ECA, interactions with diferent cations for nutrition goals included relevant tips for fruit features, and push notifications were all derived from and vegetable intake and hydration. The tips also address data logged by the system for each individual user. user-reported barriers to goal attainment. Seventeen participant post-evaluation interviews were audio recorded and transcribed by a professional transcription service. We conducted a deductive analysis 5. Evaluation Study guided by sensitizing concepts which focused on participant satisfaction with each engagement mechanism, authenticity of the religious components, and participant usage of the system. We used elements of grounded theory method, including open coding, selective coding, and memoing [23].
We evaluated the efectiveness of our religiously tailored engagement strategies in keeping users engaged with our smartphone-based mHealth app in a single-group, quasi-experimental field study. The duration of the study was four weeks (28 days), with continuous logging of system usage and weekly self-report assessments of engagement and satisfaction. The study was approved by our institution’s IRB.
Due to COVID-19 restrictions, participants were enrolled into the study via video conference sessions during which they provided consent, were guided on installation and use of the app, and filled out baseline questionnaires.
They were asked to use the app every day during the four-week study. After each of the first three weeks, electronic surveys were emailed to participants to measure their satisfaction with the app.
After four weeks, participants completed a final exit questionnaire and were invited to participate in an optional qualitative interview with a member of the research team. Participants were paid a fixed amount once, at the end of the study; they were not paid based on how much they used the app.
cruited for the study (14 from Church A and 11 from Church B). Participants self-identified primarily as Black (19 Black, 2 Black & Hispanic, 2 Black & Native American, 1 Hispanic/Latinx, 1 White). Eighteen of the participants identified as women, 5 identified as men, 1 as genderqueer, and 1 as non-binary/third gender. Participants were aged 19 to 75 (Mean=55, Median=62, SD=18). All participants but one had at least some post-secondary education.
5.3.1. Usage Statistics and Satisfaction During the four-week study period, participants (N=25) 5.1. Measures & Data Collection used the app a total of 560 times with each participant logging in to the app an average of 22 times (Min=4, Max=54, We assessed user engagement and satisfaction using SD=13.3). They logged in to the app, on average, half of self-reported measures and system-logged usage metrics. the 28 study days. Participants posted 82 prayer requests, added 43 prayer requests to private prayer lists, and wrote Clara resonated with them while younger participants 45 replies to prayer requests. Participants in Church A did not necessarily view Clara to be very relatable. each received 1 pastor announcement, whereas Church Bible stories told by Clara were viewed as calming, B received 2 pastor announcements. Each participant en- relaxing, and reflective. P7 stated, “ Clara gave me room gaged with the ECA 8 times on average, comprising 36.6% to think.” They were seen as accurate and increased of their sessions with the app. During their sessions with participants’ confidence in Clara more generally. Satisthe ECA, they chose the topic of Scriptural Meditation 84 faction with the Bible Story of the Day was positively times, Bible Story of the Day 50 times, Physical Activity related with satisfaction of Clara (rho=0.479, p<0.05). In 26 times, and Nutrition 25 times. Furthermore, partici- addition, perception by the participants of Clara being pants engaged with the health behavior tracking features religious, spiritual, and of faith was positively correlated (e.g., tracking of physical activity, hydration, etc.) a total with their satisfaction of the Scriptural Meditation feature of 801 times, with each participant, on average, using (rho=0.774, p<0.001). However, participants expressed the feature 32 times. Overall, participants were satisfied wanting to see more scriptures related to diferent topics with the app features, as well as the overall app, scoring or categories so that they could easily find personally taisignificantly higher than a neutral rating for all features lored scriptures for what they were feeling in any given (Table 1). moment. Some participants felt the stories were limited
The ECA was the most-used part of the app (Table 1). and child-like and would like some tailoring based on During the post-study interview, participants reflected level-of-knowledge. on their perceptions of Clara and her role. The second-most-used part of the app was the Prayer Center (Table 1). In general, the Prayer Center increased P7: It may seem like a script because opportunities for the religious practice of supporting the it...she was programmed to do this, but it community in prayer, because participants were able to still felt good, because it just felt genuine, interact with the prayer wall anytime and anywhere. It because she asked so many questions to also served as a memory aid because participants were kinda gauge you, kinda like, where are able to use the prayer list to keep track of their prayer you at right now, how can I help to get concerns. P2 said, “I don’t have to remember, what I want you in that direction type of thing. That’s to pray for is right there.” Participants stated that initially the feeling that I liked a lot. The feeling they were interested in seeing who from the church was like, god man, I appreciate you. using the Prayer Center and wanted to connect with other members through the feature. All participants reported One participant viewed Clara as a member of the church wanting more of a critical mass and wanting a notification who took part in the culturally-situated practices: “Clara system that would remind them to pray. P3 said, “It’s nice felt like home. Even if we are laughing and rolling on the to be connected to the church, especially right now. I just lfoor. We still end in prayer. ” P11 explored how details of wish more members had been there.” the interaction, e.g., Clara ending the conversation with For health features such as setting health goals with scripture, or Clara discussing Bible stories, mirrored how Clara or tracking health behaviors, we observed that physical church events were run in her church. Partici- 22 out of the 25 participants used at least one health pants said that Clara was able to gain their trust by not feature. Among the 22 participants who used the health only signaling values but being factually accurate, espe- features, users who rated high in the private religious cially around health topics. Furthermore, participants practices sub-scale of the BMMRS had higher usage of who were retired or near retirement age expressed how Social: Pastor announcement Reminder: Check-in with the ECA Health: Physical activity goal check-in Social: Prayer added to prayer list Social: Prayer request reply Table 2 For participants who received health tip notifications, Top 5 notification types leading to the most app use. Click- they appreciated them overall. However, participants through rate is the ratio of number of push notifications found it dificult to refer back to the tips as they disapclicked/tapped over number of push notifications received. peared after being viewed, and some were not necessarPush Notification Type Click-through Rate ily sent during a time participants could actively act on them. Participants also suggested other types of health 31.8% tips such as congratulatory messages that would allow 31.3% them to know what goal they had reached (P3), and ac27.3% countability notifications to help get them to their goals
(P2). 5.3.3. Engagement: Usage Over Time To explore which factors led to the most longitudinal use health features within the app (mean=50.7) compared to of the app, we developed a multilevel model, to account those who rated low (mean=23.3), based on a two-sample for individual diferences. The intraclass correlation coeft-test (p<0.05). Private religious practices ratings were ifcient (ICC) was calculated to be 15.4% for daily number categorized into high and low based on the median score. of logins, indicating that the majority of the variance existed within a user. The best fitting model was se5.3.2. Push Notifications lected based on Akaike information criterion (AIC) and Bayesian information criterion (BIC) values and model interpretability. The number of logins on any given day was modeled as a linear pattern of change, with study day, daily number of notifications received, and participants’ religiosity based on the private religious practices sub-scale from the BMMRS [22] as predictors. For the selected model, intercept and slope for study day were modeled as random efects to capture individual diferences at baseline and over time. Table 3 shows the fixed efects parameter estimates of the selected model for predicting daily number of logins. The final equation is:
tifications were sent and 17.5% (N=111) of those were clicked on by participants. Each participant received a mean of 25.4 notifications in total (Min=15, Max=48, SD=8.5) and, on average, clicked on 17.2% of those notifications (Mean=4.4, Min=0, Max=23, SD=6.3). Push notifications that promoted engagement with Pastor Announcement, the Prayer Center, and talking to the ECA, led to the most app use (Table 2). Notifications alerting users of new pastor announcements had the highest click-through rate (=31.8%).
Participants were satisfied with the notifications they received, scoring significantly above neutral on a satisfaction rating scale (Mean=5.3, seven-point composite scale with 4=neutral, parametric test, p<0.001). Participants were also asked about their perceived pleasantness of notifications (single-item scale, Anchor 1=“Unpleasant”, Anchor 7=“Pleasant”). The ratings were significantly diferent from neutral (=4), with a median score of 5 (onesample Wilcoxon signed rank test, p<0.01), indicating they felt receiving notifications was a mostly pleasant experience.
During the post-study interviews, participants found the reminder notifications helpful.
to-time since there were times I would forget it, but I would get a notification from Clara, and I’m like, you know what, I’m sorry shorty, forgot, hold on!
Daily # Of Logins = 0.696 − 0.039 * SD
+ 0.164 * N + 0.117 * R where SD is the study day, N is the daily number of notiifcations, and R is the private religious practices score.
Participants’ religiosity ratings were centered at the lowest score (coded as 0-7). Study day was centered at the first day (day 0). The intercept can be interpreted as the estimated number of logins on the first day, for a participant who received no notification on that day, and rated the lowest on the private religious practices sub- We did not find that the use of the app led to signifscale. Based on the model, the number of notifications icant pre-post improvements in health attitudes or bereceived on a given day has a positive efect on the out- haviors, likely due to the short duration of the study and come, as with 1 unit increase, the outcome increases by a small convenience sample. We plan to evaluate the 0.164 (p<0.001, Table 3). Study day has a negative efect efectiveness of our app in improving health behaviors on the outcome, as usage decreases slightly over time. through future studies and used this pilot study to mainly Religiosity has a positive efect on daily number of logins, evaluate the engagement mechanisms. Also, due to the as with 1 unit increase on the private religious practices participants of our pilot study being financially compensub-scale, the outcome increases by 0.117 (p<0.05, Table sated, we can expect some level of response bias where 3). engagement and satisfaction with our app can partly be
Using Pearson’s r correlation tests, we observed that associated with being compensated rather than the enparticipants with lower technology literacy had higher gagement mechanisms. In addition, we did not explore private religious practices scores (r=-0.477, p<0.05). Sim- the duration of ECA sessions or scrolling through the ilarly, those who were older had higher private reli- Prayer Wall as a measure of engagement. Measuring gious practices scores (r=0.748, p<0.001). These findings session duration in light of app suspension, exits, and showed that participants who were older or used fewer interrupts is very error-prone, and we felt that these data smartphone-based messaging technologies, were more were too noisy to warrant analysis. Finally, this pilot active in their private religious practices and logged into study lacked a control condition, and a series of studies the app more often when notifications were sent to them. to systematically evaluate each engagement mechanism relative to a control is ultimately needed. 5.3.4. Health Behavior Change Outcomes
Maintaining user retention with mobile interventions is essential for afecting longitudinal outcomes in health, education, and other application domains. Utilizing welltailored engagement mechanisms that meet the interests and needs of a priority population may be useful in maintaining engagement with the app over time and more efective at providing help and interventions to users. We explored a range of religiously tailored engagement strategies that can be used to motivate church community members to interact with a smartphone-based mHealth app.
We described and evaluated six religiously tailored engagement mechanisms in this pilot study: an ECA, Prayer Center, Pastor Announcements, Bible Story of the Day, Scriptural Meditation, and push notifications. We demonstrated that push notifications were efective at driving the use of the app, with the number of notifications per day and the user’s private religious practices score being significant factors in predicting the number of user logins per day throughout the study. We found that user satisfaction with all elements of the app was high with several participants finding Clara to be relatable and having personal resonance with the religiosity, spirituality, and role she exhibits. Our study demonstrated that religiously tailored engagement mechanisms that fulfill techno-spiritual functions can help religious users engage with an mHealth application.
tional Science Foundation under Grant No.1831755. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. [4] G. J. Norman, M. F. Zabinski, M. A. Adams, D. E. [14] L. C. Brewer, A. Kumbamu, C. Smith, S. JenkRosenberg, A. L. Yaroch, A. A. Atienza, A review ins, C. Jones, S. N. Hayes, L. Burke, L. A. Cooper, of ehealth interventions for physical activity and C. A. Patten, A cardiovascular health and welldietary behavior change, American journal of pre- ness mobile health intervention among churchventive medicine 33 (2007) 336–345. going african americans: Formative evaluation [5] S. Michie, L. Yardley, R. West, K. Patrick, F. Greaves, of the faith! app, JMIR Formative Research 4 Developing and evaluating digital interventions to (2020) e21450. URL: https://formative.jmir.org/2020/ promote behavior change in health and health care: 11/e21450. doi:10.2196/21450.
Recommendations resulting from an international [15] A. Mehrotra, M. Musolesi, R. Hendley, V. Pejovic, workshop, J Med Internet Res 19 (2017) e232. Designing content-driven intelligent notification [6] T. Bickmore, D. Schulman, L. Yin, Main- mechanisms for mobile applications, in: Proceedtaining engagement in long-term interventions ings of the 2015 ACM International Joint Conferwith relational agents, Applied Artificial In- ence on Pervasive and Ubiquitous Computing, Ubitelligence 24 (2010) 648–666. URL: https://doi. Comp ’15, Association for Computing Machinery, org/10.1080/08839514.2010.492259. doi:10.1080/ New York, NY, USA, 2015, p. 813–824. URL: https: 08839514.2010.492259, pMID: 21318052. //doi.org/10.1145/2750858.2807544. doi:10.1145/ [7] R. P. Hawkins, M. Kreuter, K. Resnicow, M. Fishbein, 2750858.2807544.
A. Dijkstra, Understanding tailoring in communi- [16] N. Bidargaddi, T. Pituch, H. Maaieh, C. Short, cating about health, Health education research 23 V. Strecher, Predicting which type of push noti(2008) 454–466. ifcation content motivates users to engage in a self[8] G. Bell, No more sms from jesus: Ubicomp, reli- monitoring app, Preventive medicine reports 11 gion and techno-spiritual practices, in: P. Dourish, (2018) 267–273.
A. Friday (Eds.), UbiComp 2006: Ubiquitous Com- [17] T. K. O’Leary, E. Stowell, E. Kimani, D. Parmar, puting, Springer Berlin Heidelberg, Berlin, Heidel- S. Olafsson, J. Hofman, A. G. Parker, M. K. Paascheberg, 2006, pp. 141–158. Orlow, T. Bickmore, Community-based cultural [9] J. Cassell, Embodied conversational agents: rep- tailoring of virtual agents, in: Proceedings of the resentation and intelligence in user interfaces, AI 20th ACM International Conference on Intelligent magazine 22 (2001) 67–67. Virtual Agents, IVA ’20, Association for Computing [10] J. M. Petersen, E. Kemps, L. K. Lewis, I. Prichard, Machinery, New York, NY, USA, 2020. URL: https: Psychological mechanisms underlying the relation- //doi.org/10.1145/3383652.3423875. doi:10.1145/ ship between commercial physical activity app 3383652.3423875. use and physical activity engagement, Psychology [18] E. Stowell, T. K. O’Leary, E. Kimani, M. K. Paascheof Sport and Exercise 51 (2020) 101719. URL: Orlow, T. Bickmore, A. G. Parker, Investigating https://www.sciencedirect.com/science/article/pii/ opportunities for crowdsourcing in church-based S1469029219308568. doi:https://doi.org/10. health interventions: A participatory design study, 1016/j.psychsport.2020.101719. in: Proceedings of the 2020 CHI Conference on [11] F. D. Barber, Social support and physical activity Human Factors in Computing Systems, Association engagement by cancer survivors., Clinical Journal for Computing Machinery, New York, NY, USA, of Oncology Nursing 16 (2012) E84 – E98. URL: 2020, p. 1–12. URL: https://doi.org/10.1145/3313831. http://libproxy.clemson.edu/login?url=https: 3376833. //search.ebscohost.com/login.aspx?direct=true& [19] T. K. O’Leary, E. Stowell, J. A. Hofman, M. Paaschedb=a2h&AN=76169513. Orlow, T. Bickmore, A. G. Parker, Examining the [12] F. Gravenhorst, A. Muaremi, J. Bardram, intersections of race, religion & community techA. Grünerbl, O. Mayora, G. Wurzer, M. Frost, nologies: A photovoice study, in: Proceedings of the V. Osmani, B. Arnrich, P. Lukowicz, et al., Mobile 2021 CHI Conference on Human Factors in Comphones as medical devices in mental disorder puting Systems, Association for Computing Matreatment: an overview, Personal and Ubiquitous chinery, New York, NY, USA, 2021, p. 1–19. URL: Computing 19 (2015) 335–353. https://doi.org/10.1145/3411764.3445418. [13] A. Hernández-Reyes, F. Cámara-Martos, G. M. Re- [20] J. O. Prochaska, W. F. Velicer, The transtheoretical cio, R. Molina-Luque, M. Romero-Saldaña, R. M. model of health behavior change, American Journal Rojas, Push notifications from a mobile app to im- of Health Promotion 12 (1997) 38–48. URL: https: prove the body composition of overweight or obese //doi.org/10.4278/0890-1171-12.1.38. doi:10.4278/ women: randomized controlled trial, JMIR mHealth 0890-1171-12.1.38, pMID: 10170434. and uHealth 8 (2020) e13747. URL: https://mhealth. [21] J. Cassell, H. H. Vilhjálmsson, T. Bickmore, Beat: jmir.org/2020/2/e13747. doi:10.2196/13747. the behavior expression animation toolkit, in: H. Prendinger, M. Ishizuka (Eds.), Life-Like Characters: Tools, Afective Functions, and Applications, Springer Berlin Heidelberg, Berlin, Heidelberg, 2004, pp. 163–185. URL: https://doi. org/10.1007/978-3-662-08373-4_8. doi:10.1007/ 978-3-662-08373-4_8. [22] K. S. Masters, Brief Multidimensional Measure of
Religiousness/Spirituality (BMMRS), Springer New York, New York, NY, 2013, pp. 267–269. URL: https: //doi.org/10.1007/978-1-4419-1005-9_1577. doi:10.
1007/978-1-4419-1005-9_1577. [23] J. M. Corbin, A. Strauss, Grounded theory research:
Procedures, canons, and evaluative criteria, Qualitative sociology 13 (1990) 3–21.