Mobile apps as health interventions have proven effective in various settings such as the management of weight, sleep deprivation and depression. Yet, thorough descriptions and analyses of the development and implementation processes of such apps are still quite rare. Moreover, app process evaluations covering more than one health behaviour are even more scarce. This study describes a process evaluation for the implementation of a digital intervention developed for supporting micro-entrepreneurs' well-being through recovering from work and job strain in a study trial setting. The health intervention was implemented as a mobile app, which coached the users in seven selected health related domains: 1) Exercising (physical activity), 2) Stress management, 3) Time management (efficient working hours), 4) Recovery from work, 5) Sleep, 6) Healthy nutrition (dietary behaviour) and 7) Sedentary behaviour (excessive sitting). A total of 1225 eligible micro-entrepreneurs were included in the intervention study out of which 613 participants were randomized to the intervention group and 612 to the control group. A process evaluation regarding the intervention group was carried out after the trial. This paper focuses on discussing the technological intervention's fidelity and dose, the latter including both doses delivered and received. The dose delivered was extensive, whereas the dose received was less extensive. Fidelity was met with various levels, from none to some depending on user activity, as those users who used the app gained support for reaching their goals.
We feel it is of utmost importance to support
microentrepreneurs’ work ability and recovery
from work through means that are tailored for the
needs of this group. Microentrepreneurs
experience high stress due to their specific work
conditions, high financial responsibilities,
economic problems, high demands and workload
[
Researchers have reported promising results
from information technology -based health
interventions in areas such as management of
stress, sleep, physical activity and dietary habits
[
There exists a myriad of mobile health
(mHealth) apps and an impressive number of
studies on the effectiveness and outcomes of these
mHealth apps [
In the Promo@Work Entrepreneurs project,
mobile app known as Recover! was developed.
This was used in Randomized Controlled Trial
(RCT) intervention to help microentrepreneurs to
recover from work. See the protocol for details
[
We conducted a process evaluation [
This process evaluation study uses the concepts of dose delivered, dose received and fidelity: the dose delivered refers to expected use, the dose received to actual use and the fidelity to the extent the latter follows the designed principles of the mobile intervention. These three concepts highlight how the dose received can vary from the dose delivered, and how this process is mediated by fidelity.
In this study, we address the following research questions:
Dose:
RQ1a: What type of counselling content was delivered to the users through the intervention and how was it carried out in the software system? [Dose delivered]
RQ1b: How much of the intervention the participants received, that is, which mobile app features they used and to what extent? [Dose received] Fidelity
RQ2: Did the participants use the features of the mobile app in the way it was planned and expected?
The results show typical trajectories of the intervention and frequent deviations from ideal trajectory. They contribute to explaining the use of the mobile app as a mediating factor for the outcome. Findings will help to identify risk factors in mobile delivered interventions. This study will also contribute to the body of knowledge on software features that appeal to users and on transforming theoretical behavioural knowledge into software design and architecture.
By addressing these research questions, the feasibility of the intervention is investigated, the adherence to the intervention protocol is evaluated, and barriers and facilitators for future implementation are explored.
Self-Determination Theory (SDT) [
SDT explains motivation for behaviour change
and suggests that there are three basic
psychological needs that an individual should
meet in order to feel autonomously motivated:
autonomy, competence, and relatedness [
TTM describes six different stages in terms of a person’s readiness for changing their behaviour (precontemplation, contemplation, preparation, action, maintenance and termination). Each of these stages demands different kind of actions from both the individuals changing their behaviour and their potential “tutor”, which in this case is the mobile app. [26.] TTM was used in this study as a loose conceptual framework in behind of modelling users’ readiness for change.
PSD is a framework for analysis of persuasion
context and for persuasive system feature
selection [
Behaviour Change Techniques (BCT)
taxonomy [
The counselling content was planned with the theoretical background in mind. In addition to this, research-based evidence on themes that are related to influencing work ability and recovery from work, and the understanding of the target group gained through previous practical work and workshops with micro-entrepreneurs were utilized in creating the content.
Thus, evidence-based themes related to
microentrepreneurs’ work ability and recovery from
work were identified. Relevant domains are sleep,
physical activity/sedentary behaviour, dietary
habits [
Drawing from the evidence-based background the app was designed to provide: 1) elements which make relevant personal reflection on motivation for healthy behaviour (SDT) 2) goal setting which is in relation to the user readiness to change; goals should include features that support the users’ possibilities to enhance their skills (TTM) 3) a wide range of evidence based BCT’s that enable the user to address all aspects of behaviour change. 4) relevant and supportive feedback on behaviour, enhancing intrinsic forms of motivation. 5) persuasive features e.g., selfmonitoring tools to support users in achieving their goals (PSD)
The fact that the intervention was delivered through a mobile app set limitations for the ways in which the theoretical principles could realize in the app. First, while SDT emphasizes an individual’s self-determination in choosing actions and setting goals, the options open for the user needed to be limited in order to be able to develop the app and further to improve the user experience. A too complex structure might have resulted in an unclear user interface and lowered user experience. Therefore, we limited the number of options in all the parts of the app while maintaining the user’s ability to move freely in the app.
Second, relating to the earlier, while TTM
includes six stages of change, we evaluated this to
be too broad and too complicated as a structure for
this app. Users who are in what TTM calls
precontemplation stage might not use the app,
since that stage does not comprise of concrete
actions towards behaviour change [
Thirdly, while SDT emphasizes the need for relatedness as one fundamental aspects of motivation we had to bargain on this. The app cannot in essence share the human experience of the user. Relatedness was incorporated in the app especially through empathetic and appreciative tone in giving instructions and feedback, but it was given less emphasis compared to needs of autonomy and competence.
The RCT examined the effects of counselling delivered through mobile app on work ability and recovery from work among microentrepreneurs.
The randomized controlled trial intervention
was targeted at micro-entrepreneurs.
Microenterprises in the European Union are enterprises
that employ fewer than 10 people with a financial
turnover less than EUR 10 million [
When compared to other people,
microentrepreneurs do not take as many days off from
work [
After the randomization process of dividing enrolled participants into intervention and control groups, and after removing duplicate enrolments, the intervention group consisted of 613 individual participant accounts. Thus, the participants (n=1225) were randomized into intervention group (n=613) or into control group (n=612).
This study concentrates on the intervention group only. The intervention group used the mobile app for eight weeks after baseline survey. The app collected user navigation related log data during the eight-week intervention.
372 users downloaded and logged into the app at least once, thus started using it during the eightweek intervention period. 3.1.
The consortium developed an information and communication technology -based intervention, which consisted of 1) mobile software app and 2) counselling content.
The intervention included the counselling content and the software, through which it was delivered. Furthermore, the counselling content was merged with the substance content, which was based on the needs of the target population and evidence from research on factors enhancing work ability and recovery, theoretical framework for the counselling and counselling methods, and tailoring the substance content and counselling methods.
The system was implemented by a traditional
frame for native apps; this approach was chosen
to support the use with or without network
connection [
As SDT had been used when developing the app, navigation within the app was not restricted.
The counselling content of the app was provided for the users in three ways: 1) introductory content, 2) main content, and 3) supportive tools for self-monitoring and selfreflection.
At the beginning, the system was introduced for the users and a description of the app, and its goal setting features was presented. This was followed by a set of (self-)reflective questions related to the seven health problem domains. The same predefined questions were also used for determining the health problems of the users. This enabled the system to suggest a relevant health module for the user. Suggesting the most relevant modules was considered to enhance their autonomy in line with SDT framework.
Any of these background questions could be skipped. However, skipping the background questions would affect the algorithm that defined which health module would be suggested. In an ideal situation, the user would answer all questions carefully, which in turn would trigger an ideal suggestion for one of the health modules. The user could also choose to ignore the suggestion and start from a module of one’s own liking. Later, the user would be prompted with another suggestion, this time dependent on the answers of the whole user base.
Based on the identified themes related to micro-entrepreneurs’ work ability and recovery from work, eight themes were covered in the app. Each of the following seven themes formed an independent module in the app: 1) Exercising (physical activity), 2) Stress management, 3) Time management (efficient working hours), 4) Recovery from work, 5) Sleep, 6) Healthy nutrition (dietary behaviour) and 7) Sedentary behaviour (excessive sitting). Additionally, eighth theme (alcohol consumption) was integrated into the modules of the other domains. Each module contained an introductory section with two videos (audio with captions): 1) a rehearsal, and 2) an informative description of the health problem domain. Additionally, introductory sections of modules contained a small set of questions or advice meant to evoke self-reflection with the users about their current health situations. Each module also contained several interactive tasks, which were composed to align with TTM, thus divided into three categories in relation to goal setting: Think and observe, Act and do, and Maintenance. Within the first two categories, the tasks could be completed swiftly or span over a course of time, whereas the maintenance category contained only tasks that could be done during a longer period.
Each health problem domain module contained at least one supportive tool for selfreflection and self-monitoring. The tools were introduced (and suggested) for the user within the tasks in the modules. In addition to self-reflection and self-monitoring, some of the tools were intended to support the user in their target behaviour. Please see table 1 for the tools.
According to Moore et al. [
Based on the log data collected by the app, we
will provide descriptive quantitative information
on fidelity and dose of the intervention [
Log data about the use of the app was collected into MySQL database of the back-end server. Everything the users did within the app was logged with a timestamp, thus enabling a thorough data collection about the use. Each user could be individually separated from the rest via a pseudonymized identification code. The MySQL files were imported into comma separated excel files and prepared for analysis in the following way: the intervention period was 56 days, counting the login day as the first full day of use. 4.2.
Dose. A timestamped log data file consisting of the navigation and actions of each user was used for analysing the dose. Before the beginning of the actual analysis work, the requirements for ‘dose delivered’ and ‘dose received’ were discussed and decided together with the researchers. The analysis took place in Microsoft Excel with search functions, filtering, pivot tables and formulas. When necessary, the data file was divided into separate files containing the navigation and actions of single individuals.
Fidelity. The analysis for the fidelity was done similarly in Excel, with more of the work done manually, although with the help of search functions and filters. The requirements for ‘fidelity’ were similarly discussed and decided together with the researchers before the actual analysis work. A table with the different parts of fidelity marked was created and filled individually for each of the 372 users.
Dose delivered. There were total of 51 different tasks and seven self-monitoring tools (see Table 1). Additionally, 11 different videos were available within the health problem domains.
When developing the app, we assumed that the users would proceed within the app in a certain manner - following the logic of sequential process we had planned. Each element of the app was planned to support each other and together they formed an effective entity (package) for the users to proceed within to achieve their goals.
When choosing a health problem domain for the first time, module specific introductory content was delivered to the users. The user could skip the videos, as well as the reflective questions, and would not be prompted about them later. The videos remained accessible through the main menu. Feedback was provided for the users upon completion of tasks, thus accessing the feedback required no extra effort from the users. Any action or task within the app could be stopped by pressing the ‘home’ button.
As regards fidelity, we decided to measure it by using the operationalization of the logic model. More specifically, we used the short-term outcomes as measurement for fidelity (see fig 1). Fig. 1. Operationalization of the logic model and short-term outcomes.
The logic model portrays the operationalization of short-term outcomes regarding fidelity. The background theories and evidence based BCTs are combined in the structure of the app, although partially as compromises. For example, TTM is combined with SDT, as the users can choose the goals, which suit them best, but they will be choosing them from a pre-defined selection. Selfmonitoring as a PSD principle could be seen to support either the TTM based goals within the app, or the personal goals set by the users.
Dose received. The contents that were available in the app were the same for each individual user. The dose received was therefore solely depending on the actions and choices of the user.
Out of 372 participants, who downloaded the app and logged in at least once, 328 proceeded to at least some health problem module at some point (see Table 2). 239 participants proceeded into the stages-of-change goal setting in at least one module, but only 191 of them chose a task. Eventually, 147 finished at least one task (see Table 2 for tasks). 257 users visited (at least) a second health problem domain module during their use of the app. Additionally, 140 users also chose (at least) a second goal and task, with 92 users completing at least two or more tasks either immediately or retrospectively (continuing to use the app after a break for example) during the intervention period.
159 participants begun at least one long-term task, but only 92 of them completed the task, thus got feedback tailored according to behaviour change techniques and logic model. Only 35 users completed a long-term task from the ‘maintenance’ category.
A total of 1078 modules were chosen between the participants, with an average of 3,29 modules per user and median of 3 per user. A total of 1993 tasks were chosen between the participants, with an average of 10,43 tasks per user and median of 5 per user. ‘Sleep’ was the most popular health module both by the number of users accessing the module and by the number of users completing a task in the module. The participants completed a total of 603 tasks with an average of 4,09 tasks per user and median of 2 per user.
Fig 2. App use and users regarding content
During the intervention group period of use,
there were 4385 complete user sessions (logins)
between the participants. The extremes were
between one login and 156 logins, thus least
active user(s) logged in only once, whereas the
most active user logged in more or less three times
a day on average. The average amount of logins
was 11,8 and the median was 5 logins per user.
However, it should be noted that there were 46
users that logged in only once and similarly there
were another 49 users that logged in only twice.
In both cases, the amount of use could not be
considered as real use, but rather testing the app,
after which the app was not adopted for use for
whatever reason. One of the persuasive features
based on the PSD model [
In one hand, users might be less prone to advance in manner intended by developers, rather advancing through app depending on their personal interests. It would demand commitment from the users to advance in a manner intended by developers. Apparently, this was not the case, as only 147 users finished at least one task, thus the idea regarding commitment was not realized for majority of the users.
However, there were users who used selfmonitoring tools instead of completing tasks. Of the participants who did not complete any task, 39 had accessed the tools (presumably for testing purposes) and another 50 had tried out the tools a few times, although there was no actual sustained use of the tools.
Nevertheless, 37 users, who never completed a task, practically used actively only the tools of the app, thus being active users in a way that was not fully anticipated by the developers in this case. It should be mentioned, however, that originally the tools were meant to be accessed only through tasks; later (before the intervention period started) it was changed that tools could be accessed at any given moment from the main menu - even before choosing a goal or a task. Thus, even if not fully anticipated, and not in line with the planned sequential logic model, those who used only tools were still receiving the dose. This was because in sequentially advancing guidance, the users would face several different BCTs.
In any case, the dose received seemed to have differed substantially from the dose delivered. When considering that only 147 users completed at least one task and additionally another 37 users were active users of tools, the proportion of active users was not that substantial in the end. Thus, 184 users received the dose, which is practically half (49,5%) of the users of the intervention group, whereas another half (188, 50,5%) did not receive the dose.
Fidelity. When the users completed a task, they were given feedback. Those completing a short task were given condensed and brief feedback, whereas those completing longer tasks received more comprehensive (task and content specific) tailored feedback. On one hand, for those users who only used tools and never received feedback from completed tasks (or did not access counselling content), it could be argued that the short-term outcomes were not fully realized, and thus for those users, the fidelity was not met. On the other hand, the logic model that we had in mind failed to cover the type of use associated with modern apps properly, because it could be possible to use apps in ways not foreseen even by developers.
As regards apps, users can set personal goals in addition to goal setting provided by systems. Often it might not be known whether the users have set themselves personal goals, for which they, for example, need to use self-monitoring tools. Additionally, it might be very difficult to determine on the behalf of the user whether s/he has successfully met the personal goal. In this case, the log data clearly shows that there were several active users, who used only selfmonitoring tools, and so it could be speculated that they were using the self-monitoring tools to meet their personal goals; they had not set goals or completed tasks within the TTM based goal setting provided by the system.
Therefore, use of self-monitoring tools could be included into the operationalization of the logic model of fidelity in this process evaluation. Furthermore, self-monitoring tools are specifically linked to goal setting in the PSD model, while also being one of the primary task principles. Thus, the short-term outcomes of the logic model could be seen to cover aspects of TTM, SDT and PSD - all the underlying theories involved in the intervention.
Based on the logic model, 147 users proceeded accordingly, thus filling the minimum requirements for fidelity: choosing a module, setting a goal, and completing a task while receiving feedback. In their case, fidelity was fulfilled. However, not all of them proceeded strictly sequentially, but instead they may have tried out tools or other modules before or after choosing a goal. Nevertheless, eventually fidelity was met as chosen tasks were completed. Furthermore, 37 users used self-monitoring tools provided by the app, thus for them fidelity could also be said to be filled, but regarding their personal goals rather than those provided by the system.
Four different, partially overlapping groups could be categorized from the whole user base: 1) those who completed at least one task, 2) those who used tools, but did not complete any tasks, 3) those who completed at least one task and used tools, and 4) those who did not complete any tasks or did not use tools actively.
The first group consisted of 147 users, who completed at least one task. 55 of them received only the brief feedback, thus they completed only short tasks, whereas 92 (63%) completed long term tasks and received more comprehensive tailored feedback. 23 of the formers completed a short task from a module they chose at the beginning of use, whereas 37 either changed the module before a short task completion or navigated within the content before choosing a module and completing a short task. 36 of those, who completed at least one long term task, proceeded within the module according to the logic model, thus completed the task from the module they chose at the beginning of use.
56 of those, who completed at least one long term task either changed the module before a long-term task completion or navigated within the content before choosing a module and completing a long-term task. Thus, 42% of those completing only short-term tasks continued with the module they chose at the beginning, and 39% of those completing long term tasks continued with the module they chose at the beginning. Therefore, it seems that there is hardly any difference between continuing with a module chosen at the beginning or choosing a module later within the first group regarding short- or long-term tasks.
The second group consisted of 37 users, who used tools actively, but did not complete any tasks. From the group, five users did not choose any module at any point of use but used tools actively. 32 users chose a module, but only five of them chose a task within the module, although did not complete any tasks. Thus, the users in the second group were keen on using tools, rather than even choosing a pre-defined goal (14%), which seems to indicate that they were using the self-monitoring tools to reflect on their personal goals.
The third group consisted of 76 users, who used tools actively and completed at least one task. From the group, only 16 did not complete a long-term task, thus 60 (79%) did. Therefore, it seems that those users, who were completing tasks and using tools were more likely as a group to receive comprehensive task and content specific tailored feedback. Additionally, only 25 users (33%) completed either short- or long-term task from the module they chose at the beginning of the use. These numbers could very well indicate that those users decided to choose the modules, tasks, goals, and tools freely from the selection as per SDT. This in turn led to active use of the app features, and in the manner more users were receiving tailored feedback leading to fidelity being met in all aspects. It could also mean that the users in the third group were the most motivated for behaviour change to begin with or the app content and structure suited them better than the rest.
The fourth group was the largest and consisted of 188 users, who did not complete any tasks and did not use tools actively. They may have had briefly tested the self-monitoring tools once or twice, but no significant use could be said to have taken place. Similarly, they may have glanced through the contents of the app, and even chosen a goal, but never completed a task. Nevertheless, 98 (52%) of them did not access the self-monitoring tools at all. Furthermore, 39 (21%) of this group never accessed any module in the app and 149 (79%) did not choose any goal to begin with.
Nearly half, 84 (45%), of the fourth group never selected any goals or accessed any of the self-monitoring tools. Thus, on the whole user base level, fidelity was not met at any level of the short-term outcome logic model for (84) 22,5% of the users regarding the app use.
It is not uncommon for users to stop using apps
for various reasons, since nowadays there is
always another option available to be downloaded
quickly and effortlessly. Furthermore, our
participants seemed to be relative busy in their
lives. The multitude of choice for apps concerns
mHealth apps as well, although the quality of the
products may vary. Health apps may be
abandoned very early (e.g., within two weeks of
download) by many users, often because of poor
usability or the reasons that the apps do not line in
with the preferences and goals of the users [
In this process evaluation study, the implementation process of a mobile app -based health intervention for helping microentrepreneurs to recover from work was evaluated. With content consisting of 51 different tasks, seven self-monitoring tools and 11 different videos within seven health problem domains (including introductory and information content), the dose delivered of the implementation could be said to be comprehensive. The dose received however could not be said to be that extensive, since only half of the users received the dose. Participation rate for the intervention could have been better, since only 60,7 percent (372 out of 613) of enrolled participants downloaded, installed, and logged into the app in the first place.
Fidelity of the intervention was not met at any level for 22,5% of those users who logged into the app at least once during the intervention period. For the rest, who had logged into the app at least once, fidelity was met on some level depending on their activity. It could be said that they gained support for their chosen goals: either ones from the stage-of-change goal setting of the app or personal ones derived from real life settings.
Micro-entrepreneurs could be said to be a
demanding target group for this intervention.
They are stressed out, have specific work
conditions, high financial responsibilities,
economic problems, and work more than most
while facing difficulties balancing work and free
time [
As an additional detail, there were some indications in the log data that some users seemed to be expecting to receive tasks automatically. In other words, to be guided through the intervention, which seems somewhat in conflict with principles of SDT. More than few people kept on opening ‘on-going tasks’ menu in the app even if they had not chosen any tasks in the first place. There were even some that kept on opening the app occasionally, only to check the ‘on-going tasks’ menu from the app without taking any other actions in the app.
The presumable reasons for this kind of
behaviour regarding the app use would be that the
users were expecting that they would be assigned
tasks by the system. Tunneling is providing means
for actions that brings users closer to the target
behaviour [
The limitations concern the technical functionality of the app, as some users faced technical issues during the intervention. It is not known, or cannot be interpreted reliably from the log data to what extent the possible flaws affected the use of the app.
The slight possibility of bias in this process evaluation concerns the analysis of the log data and the quantitative results due to large amount of manual work in the analysis. The manual work in part was due to inconsistent way that some of log data was marked (and thus had to be checked manually). We tried to be very careful with the work and tried to eliminate miscalculations by checking correct marking from source code whenever possible.
The app offers the intervention content in a standardized format, which in practice means that the dose delivered is always the same for all users – at least in the sense that the same content is available for all users although they were expected to choose themselves only some of the contents, i.e., those relevant for them. The dose received is different case, since it is always depended on the users’ actions and choices within the app. However, it should be mentioned that technical difficulties e.g., bugs, might affect these actions and choices somewhat.
Regarding the users, we managed to find four different groups of user profiles. The largest group consisted of those whose use was from scarce to practically non-existent. The reasons for this kind of behaviour may vary, but virtually guiding these people through the intervention by tunneling could very well have increased their engagement considerably. The third largest group consisted of users that took advantage from all aspects of the app and thus seemed to fare the best;79% of them completed long term tasks, thus gaining tailored feedback. Nevertheless, it could be difficult to determine the amount of use, whether measured in time or number of logins, that is sufficient for saturation: that the user is unable to receive any future added value from the intervention.
For this target group, micro-entrepreneurs, it seems that offering content, tools, and tasks from ‘nutrition’ and ‘sedentary behaviour’ was not as useful for them as for example ‘sleep’ and ‘stress’ were. This is most likely because of the occupational settings of micro-entrepreneurs: they can be very stressed out and sleeping poorly. Furthermore, their working conditions might vary substantially from those of common office workers who spend most of their working hours sitting. As regards ‘nutrition’, information about this topic is commonly available via e.g., newspapers and magazines et cetera. Thus, this could lead into a situation where any further counselling on the subject might not be as interesting as others.
Tasks from the ‘maintenance’ category of TTM based goal setting seemed to be substantially less interesting than other type of tasks. Therefore, it could be speculated that people in the ‘maintenance’ stage have already experienced changes in their behaviour and might not need additional support for behaviour change contrary to those that are in the earlier stages. Furthermore, guidance via mobile app may not also be the best platform for those at maintenance stage – at least not for this target group. Still, there were users who completed tasks from this category, but not as much as those who completed tasks from the other categories. This may reflect the type of participants enrolling into the intervention: those in the ‘maintenance’ stage might not be as eager to participate as those that are in more dire need of an intervention regarding their health behaviour.
As for future research, further analysis of use could be beneficial, but concentrating on whether there could reside an ideal way to proceed in a designed app. This in turn could have a substantial impact on the implementation of interventions; in other words, what kind of structure and content should the users be offered in an intervention.
We wish to thank our colleagues from Promo@work. Special thanks to Janne Jaako. This research was funded by the Strategic Research Council at the Academy of Finland (contract 303430, responsible party: Finnish Institute of Occupational Health).