We describe our results from the implementation of machine learning and AI methods in three digital health initiatives serving individuals across the mental health spectrum in Singapore. The first initiative is Project HOPES, which we launched in 2019 for patients with serious mental illnesses. Originating as an observational study on digital phenotypes (collected via smartphones and wrist wearables) of 100 patients with schizophrenia, the tool has now been introduced as a service within a tertiary setting and has expanded to include patients with depression. The strategy dynamically prioritizes patients for early review by care coordinators according to need, which may avoid hospitalizations. Machine learning is used to predict clinical status, i.e., symptoms and functioning, and to predict relapses and other adverse clinical events; the latter can be done at 92% sensitivity and 90% specificity with the available digital biomarkers. The second initiative we describe is (www.mindline.sg), a platform for mental wellness in the general population that we created in 2020. Through a public-facing website, we deliver over 800 resources including wellness education, clinically validated self-assessments and triaging, and interactive resources, including an AI chatbot. Launched at the height of the COVID-19 pandemic, with all its attendant stresses, the platform has been visited by somewhere between 10 to 20% of the national population by the end of 2023. The third initiative we describe is Let's Talk (https://letstalk.mindline.sg), an online peer-support mental health network, which was co-created with youth advocates. The need for this platform was discovered through extensive studies with youth who expressed a desire for human-based support beyond the proliferation of digital solutions. In its first year, the site has been visited by over 80,000 unique users. Trained moderators review content on the site for safety and accuracy, and qualified therapists provide professional support through the free and anonymous Ask-A-Therapist service. To scale this service with a growing user base, we have been trialing the use of generative models to aid our therapists in finding relevant resources according to a user's need and to encourage empathetic writing.
CEUR ceur-ws.org
Mental Health conditions comprise one of the largest burdens of disease worldwide, especially when measured in Years Lived with Disability (YLDs). Stigma is prevalent in many countries and cultures, which discourages help-seeking. At the MOH Ofice for Healthcare Transformation and the Institute of Mental Health in Singapore, we adopt a population health approach starting from each end of the mental health spectrum and “working our way in”. On the one hand, we serve the needs of patients with serious mental illnesses, which are defined by the US National Institute of Mental Health (NIMH) as “mental, behavioral, or emotional disorder[s] resulting Machine Learning for Cognitive and Mental Health Workshop (ML4CMH), AAAI 2024, Vancouver, BC, Canada ∗Corresponding author. nEvelop-O tools across the mental health ecosystem in Singapore.
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
what-is-mental-wellness/ 1.1. Serious Mental Illnesses: Digital other) digital measures were predictive of poor sympPhenotyping and AI in Schizophrenia toms and functioning: irregular sleep habits (including and Depression increased time spent awake in bed and in light stage sleep), decreased steps and GPS mobility, decreased text Patients in Singapore with schizophrenia are usually messages sent, slowed tapping speed, and increased heart treated in the specialist or hospital setting. After dis- rate while asleep, among others [6, 5]. charge to the community, it is not uncommon to see We have also investigated the use of machine learnrelapses. Around 80% sufer at least one relapse within ing to predict adverse clinical events, including relapse ifve years of initial remission [ 1] which often result in of psychosis. For these purposes, we defined a relapse emergency room visits or re-hospitalizations. Relapse is as a rehospitalization due to psychosis symptoms or a extremely disruptive to a patient’s life and rehospitaliza- significant deterioration in clinical status defined using a tion incurs large costs. Relapsing patients often exhibit validated clinician-assessed scale measuring general psypsychotic symptoms such as hallucinations, delusions, or chopathology symptoms. Other clinical events include disordered thinking. They might display changes in sleep emergency room visits, readmissions due to reasons other behaviors, mood, social withdrawal and disorganized be- than psychosis, and unscheduled clinical visits. For these haviors. An emerging strategy to detect such changes predictive models, we explored both unsupervised learninvolves digital phenotyping, defined as the “moment- ing approaches to anomaly detection, including time seby-moment quantification of the individual-level human ries smoothing and forecasting methods (including the phenotype in situ using data from personal digital de- method originally used by Henson et al. [7]) and isolavices” [2]. In 2019, we began the HOPE-S observational tion forests, as well as supervised learning approaches study [3], for which we developed the HOPES digital utilizing generalized linear models, random forests, and phenotyping platform described in detail by Wang et al. gradient boosting trees. The models first establish a pa[4]. As of the end of 2023, the platform has been in con- tient’s baseline (on the multivariate data) and attempt tinuous operation with patients and clinicians for over to detect deviations from that baseline at the individualfour years. Events are currently recorded from the user’s level. Retrospective analyses from our observational trial smartphone including mobility (derived from obfuscated indicate that we can detect adverse clinical events. The GPS coordinates), tapping speed on the keyboard, am- model that performed best varied by situation. For exbient light, screen time, accelerometry, and sociability ample, when using all digital measures (constituting a indices (derived from calls, SMS and WhatsApp calls and very high-dimensional and not very interpretable feature messaging). Events are also captured from a wrist wear- space), the gradient boosting tree performed best. But able device measuring heart rate, heart rate variability, when we restricted the dataset to the one or two most inactivity (through step counts), and sleep (including stag- terpretable features from each sensor (which are studied ing and eficiency). by clinical staf to explain the model), the isolation forest
Over the course of the study, we continuously col- always performed best. The best model to use therefore lected digital phenotyping data from 100 patients with depends on the operating mode, and ultimately will be schizophrenia (each patient was followed up for a six- guided by clinical requirements. month period), with clinical assessments performed ev- Here, we briefly report indicative performance metrics ery six weeks to measure symptoms and functioning. from the gradient boosting tree model. A more thorough The total data collected throughout the trial consists of report of the methodological descriptions and predictive over 220 million events. We found generally high com- results will be reported in upcoming publications. A claspliance in wearing of the wrist device (91% of all possible sification setup is used, as studied by Ben-Zeev et al. [8], data was successfully collected in the week following and sensitivity, specificity, and the harmonic mean score enrolment), which required patients to wear the device (between sensitivity and specificity) are reported. In our at all times, including to sleep, and successful data col- clinical setting, we are willing to accept a reasonable lection from the smartphone (82% of all possible data number of false alarms (i.e., lower the algorithm speciwas collected), which only required patients to not close ifcity), to keep sensitivity high. A false positive alarm the App in the background [5]. We note, however, that may result in the care coordinator giving the participant this high compliance rate was likely aided by a modest a call to check in or sending them an inquiring and supinconvenience fee that was provided to the trial patients. portive text message. We therefore give more emphasis Some patients, to whom the study was ofered, declined to sensitivity, i.e., being able to sense a deterioration in to participate for reasons including privacy or intrusive- patient health, even if mild. This is acceptable for our ness, leaving us with the interesting challenge as to how clinical partners who envision a shift from the traditional we can ameliorate and obviate these concerns in the fu- reactive model of care to a proactive one where early ture. In our initial analyses associating clinical status detection and intervention might provide extra support with digital markers, we found that the following (and and therefore prevent adverse clinical events. We therewhich has the interpretation that sensitivity is times as important as specificity (where we note that can be greater than or less than one depending on whether you value sensitivity or specificity more). This measure is directly analogous to the weighted -score used in ture and display the “relative feature importance weights” classification. In the experiments that follow, we report (the impurity-based feature importance measures imple 2, where sensitivity is considered twice as important as mented by most packages) in fig. 2. In this experiment, specificity. we see that tapping speed is considered the most impor
Boxplots of 2 over ten 85%/15% training/testing tant feature, followed by GPS-based and wrist device dataset splits for two machine learning models are shown measures, including distance travelled, time at home, in fig. 1. The median score (over the test sets) using the sleep eficiency score, and number of steps. This example gradient boosting tree is 2 = 91.0% (91.5% sensitivity, only considers a small subset of the features, however, 89.6% specificity). Note that the displayed sensitivity and and we do see such rankings change depending on the specificity scores are the median scores for these metrics experimental setup and the feature set provided. over the test sets; they do not correspond to the compo- Having successfully completed the above study, we nents used to compute the reported 2 score. This pre- are now piloting the use of this model as a preventative dictive power is high but may be dificult to achieve in a service at the Institute of Mental Health (a large menclinical service (versus a controlled study) where compli- tal healthcare facility) in Singapore. This service is now ance to wearable and smartphone data collection without being extended to serve both patients with schizophreifnancial incentives could be more challenging. The study nia and mood disorders including major depression. For by Cohen et al. [9] indicates that this real-world compli- this new HOPES Clinical Service, patient-facing compoance could drop to 50%. Missing and delayed data uploads nents for the smartphone App have been developed to are common in clinical practice. Experiments on subsets supplement what was purely passive monitoring in the of the features suggest that performance could drop to previous observational trial. Supported on both Android 2 = 85.6% (87.4% sensitivity, 78.7% specificity) with a and iOS, patients may now interact with the App through sparse model containing a subset of measures that are rel- Ecological Momentary Assessments (EMAs), wherein they atively easier to collect (steps, heart rate, accelerometer, may answer some questions as to how they are feeling at screen time, taps in Apps, and tapping speed). the time of a prompt, and they may additionally record
To explore those digital measures that appear most what factors or stressors might be contributing to those important for prediction, we fit the gradient boosting feelings. Such data collection is sometimes referred to as tree model on the prototypical measures from each fea- active monitoring. The EMA responses are transmitted Digital phenotyping and EMA signals to the care coordinators to aid in care and supplement the HOPES App. These interventions, based on the EMAs the digital phenotyping data, which together surface pa- as well as the passive data, are timely and individualized, tients on a clinical dashboard. Our strategy is to have for example, sleep exercises are delivered when patients clinicians in a care monitoring center regularly observe have poor sleep for two nights or more, and mindfulthe anomaly detection signals on the dashboard, and with ness exercises are delivered when patients indicate low the help of digital tools that can explore a patient’s data mood on the EMAs, as just two examples. Such interventhe clinical staf may decide to take further interventions, tions are known as Ecological Momentary Interventions including earlier clinical review for a patient. (EMIs), and some of these digital therapeutic exercises
The clinical dashboard is designed to achieve the efec- are inspired by cognitive behavioral therapy (CBT).
tiveness of Intensive Case Management [
Patients with detected needs of lower acuity (i.e., those in the Y state) are supported with automated interventions comprised of digital therapeutics delivered through Machine learning is used to predict a patient’s clinical status based on the multivariate set of digital markers, i.e., potential predictors of a patient’s current symptom severity and functioning. This may potentially avert the need to come in for clinical visits as is currently required under standard care. Machine learning is also used in the longitudinal prediction of relapses and may raise the color coding of severity and prioritization of the patient for attention by the care coordinator. We also note that AI is used in a self-contained way in some of the EMI tools in the form of a mental wellness chatbot, which we will describe in the next section on our digital mental wellness platform, mindline.sg.
The machine learning models utilized for clinical event Many countries and cultures face persistent challenges prediction include traditional time series smoothing, iso- to mental health awareness and promotion including lation forests, generalized linear models, random forests, stigma toward mental disorders, reluctance to seek help, and gradient boosting trees. The methods used for dis- low mental health literacy, a lack of trained mental health covering associations between clinical scales and digi- personnel, and underdeveloped mental healthcare ecosystal markers were multiple linear regression and multi- tems. The COVID-19 pandemic created a surge in mental level/hierarchical linear regression. A detailed descrip- healthcare needs, demanding a new impetus to addresstion of these methods is beyond the scope of this paper ing these shortcomings. The pandemic also accelerated and will appear in upcoming publications. the adoption and acceptance of digital health solutions,
Explainable signals are important for the clinical ser- creating a new opportunity for innovative approaches to vice. The multivariate nature of our digital phenotyping address mental healthcare needs. data aids clinicians in the interpretation of alerts – for In June 2020, we launched a Web App, mindline.sg which we provide digital tools to allow clinicians to keep (www.mindline.sg), serving as a digital mental health retrack of patient data. It is important to note that our ma- source website that has grown to include over 800 curated chine learning-based relapse prediction algorithm is only resources, a clinically-validated self-assessment tool for permitted to increase patient severity (from G to Y or from depression and anxiety, and a fully integrated AI chatbot Y to R), which may prompt attention from a care coordi- developed for mental health applications from Wysa3, nator (see fig. 3). The AI algorithm may never relegate a a leading partner in the field. More recently, we added patient to lower acuity. This is not only for safety, but curricular and structured learning materials with Intelalso aligns with Singapore’s national regulatory guide- lect4, another leading partner in digital mental health. lines for predictive models in clinical service. As further The landing page and a therapeutic exercise with the experience is gained in the detection and management chatbot are shown in fig. 4. The self-assessment tool of patients with digital phenotyping, we may be able to is comprised of the conventional scales of GAD-7 and move to a wider use of AI algorithms and to allow them PHQ-9 and triages users into well, mild, moderate, and to play a more definitive role in the selection of patients crisis levels. The triage status allows us to customize confor clinical review. As new regulatory guidelines emerge tent and recommend appropriate therapeutic exercises. and are navigated, pivots to our design and strategy may Tailored products for youth and working adults are also occur. provided to the public. Business-to-Business (B2B) customizations and engagements serve a range of ecosystem partners, including workplace partners and educational 1.4. What’s next? institutions. Resources on common risk factors (such as So far, the bulk of our experience has been with ifnancial, employment and caregiver stress) are included, schizophrenia. Our service, however, has expanded to addressing a broad spectrum of determinants of mental include patients with depression and takes a transdiag- health. The platform was developed to be anonymous nostic approach, which may justify further research trials and to contain authoritative and localised content for varto develop and refine algorithms for depression in the ious levels of distress, with a focus on wellness and mild local context. Additionally, an upcoming research study needs. Moderate-to-severe needs are primarily served will evaluate our clinical service. Finally, we are currently by detection (through the self-assessment triage and AI expanding our digital phenotyping strategy to mental chatbot) and referral to professional support, which inwellness. In this way, our digital phenotyping and ma- cludes counselling centres and emergency 24/7 services, chine learning tools are moving “inward” toward mild all according to a clinician-designed protocol. severity and well-populations. The platform has shown remarkable uptake. The site has been visited by between 10 and 20% of the (targeted) national population. The variability of this estimate is due to the anonymity feature of the site (we can only detect cookie IDs), which is a key feature enabling barrierfree access. If unique users visit from multiple machines or browsers, we may record them more than once. Indeed,
it is a learning from our implementation that anonymity
does limit our ability to evaluate the platform. Another
learning has been that successfully scaling the platform
required expansive and sustained digital marketing
efforts, as well as strategic ecosystem partnerships through
the B2B products, and investment in partnership with
educational institutions and healthcare providers. About
60% of our user acquisitions come from ads that we post
on social media; the next most frequent acquisitions are
from search, direct entry of the URL, or use of a QR
code from our flyers and posters. Referrals also occur
from B2B partner sites. The most popular resources used
on the platform are sleep aids. “Mood check-ins” and
the self-assessment tool are also popular. The
distribution of moods includes “tired”, “unmotivated”, “anxious”,
“positive”, “frustrated” and “sad” (in order of decreasing
frequency). Scores for GAD-7 and PHQ-9 show a
moderate number in a state of crisis: however, we believe
that this frequency is afected by anonymous users trying
out diferent answers to the questions to trigger
diferent triage levels and seeing what resources are ofered,
mainly out of curiosity. We do not view this kind of usage
negatively, as we feel that it is important for users to be
educated as to what resources are available in times of
need, either for themselves or a friend or relative. We
have published these results in both process and impact
evaluation studies [
The goal of the mindline.sg platform is to empower individuals in the community to take charge of their own mental health and to provide them the tools they need to ofer basic support (“first aid”) to themselves and those around them, all through the ease and convenience of a barrier-free digital solution. This aligns with our strategy to improve population health through digital tools that enable self-empowerment and self-management. Such a strategy transfers some of the care from the system onto the individual and their significant others and moves some care from the clinical setting into the community and the home.
Wysa is deployed to engage, triage, chat with, and direct the users to a range of relaxation, mindfulness, and CBT-inspired exercises. The Wysa chatbot is designed by a team of psychologists actively involved in patient counseling and has been subjected to numerous studies evaluating efectiveness [ 13, 14]5. Beyond the chatbot, however, we have limited data collected on the site (again, due to the anonymity feature), which limits any machine learning and AI eforts. A mobile App is presently being developed to give users an alternative option, which may be able to better leverage AI to improve user experience have been over 2,800 posts and over 370 Ask-a-Therapist and benefit. questions had been answered. As the platform’s user base scales, we anticipate challenges in moderating the 2.3. What’s next? platform and responding to questions in a timely manner while maintaining quality. We have therefore started The present wellness tool is mainly designed to serve trialing the use of large language models (LLMs) to asthose who are well or have mild conditions, with referral sist our staf therapists in searching for relevant content to human-based resources for moderate and crisis cases. from a trusted knowledge base (mindline.sg) based on We are also exploring the incorporation of clinical ad- the user’s need (inferred from the posted question). We junct tools such as validated and localized internet CBT used GPT-3.5 from OpenAI, which we fine-tuned using (iCBT) tools [15], which can be used by mental health- over 300 question-answer pairs from the Ask-a-Therapist trained primary care providers. The mobile App will service. Retrieval-augmented generation (RAG) [17] is enable longitudinal data tracking and clinical manage- used to produce the most suitable resources from the ment, which in turn may provide new opportunities for knowledge base, which is indexed from all resources in AI and machine learning in the service. We have already mindline.sg. While the therapist may copy-and-paste recconducted a feasibility study of the tool in this population ommended resources (and their descriptions) from this [16]. tool, they remain fully responsible for the content in their reply. The LLM assistant has helped our staf therapists 3. Digitally-Enabled Peer and with close to 30 responses so far, where 88% of those responses have been rated as helpful by the therapist.
Professional Mental Wellness In fig. 5, we show information on the Ask-A-Therapist Support for Youth service and screenshots on how our therapists use the LLM assistant.
After extensive workshops, focus group discussions, and co-design sessions with youth (including those with lived experience), we discovered a desire for social support and meaningful human interactions delivered in a safe online environment. In particular, human-based support is now specifically sought amongst the proliferation of purely digital self-management solutions. We therefore co-created with youth advocates an online peer support network called Let’s Talk (https://letstalk.mindline.sg).
The platform soft-launched in October 2022 and has been piloting since. By end of December 2023, the site had received over 80,000 unique visitors (as measured by Google Analytics). This peer-support network’s value proposition, over other platforms such as Reddit, includes close oversight and management by trained moderators and professional therapists. The moderators and therapists maintain a constructive and supportive atmosphere in the forum; other comparable forums have sufered from trolling, toxicity, spam, and scams. It is noted that some “medically themed” forums are overly commercial or are used by practitioners to advertise their own practices.
Let’s Talk also provides an Ask-a-Therapist service where users can pose a question to a panel of qualified professional therapists whom we have engaged; their response is asynchronous but usually occurs within 24 hours. The therapists follow a protocol defined by a Clinical Advisory Panel to deal with pressing needs and crisis conditions.
As of December 2023, from among the over 80,000 visitors to the site, over 6,000 have registered an anonymous user account (which is required to post content but is not required to access the forum and read posts). There
the mindline.sg platform described in the previous section, and it serves as a companion site to Let’s Talk. Where youth mindline enables self-management and selfempowerment through digital-only “self-help” tools, Let’s Talk ofers a purely human-based form of therapy and engagement. We believe the platform may also address determinants of mental health based on sociability and that users ofering support to others has benefits for the helper and the helped. The strategy provides a lowbarrier means of accessing professional support and provides this support (which is at the individual level) at scale.
The LLM assistant exemplifies one of our strategies for generative models in healthcare in which an AI agent acts as an assistant to care providers. As we scale, we envision that such agents can dramatically reduce the time to search for a pool of potentially optimal therapies for a patient, client, or user’s exhibited needs at the appropriate time, saving the clinician or care provider time. A suficiently large pool of resources, as is the case in mindline.sg, can guard against repetition or a narrow set of recommended resources. Future plans for AI and machine learning in Let’s Talk include using LLMs to review therapist responses to check for or encourage a more empathetic tone. We may also use LLMs to train our peer supporter volunteers to a standardized level. Finally, LLMs or other NLP techniques can be used to continually assess content on the site to detect toxicity, spam, and misinformation.
by patients under care or users among our population. They confirm the promise of usefulness of digital and AI tools in providing improved digitally-enabled therapeutics and interventions. These initiatives demonstrate our strategy of starting at extreme ends of the mental health spectrum and working our way in toward the middle, blending interventional tools as we go. Through this strategy, we aim to cover the entire life-course and spectrum of acuity. Along the way, we are incorporating further care and ecosystem partners, including additional tertiary care partners, primary care providers, allied health, community organizations, peer supporters, workplace partners, and educational partners. We hope to communicate our learnings to others with a similar mission. We also hope to learn from others who are on a similar journey.
phenotyping study described in the first section. The mindline.sg and Let’s Talk services both have Terms of Use (available on their websites) that indicate that usage data may be collected and used for research purposes and for service improvement. The authors declare no conflicts of interest.
and further technical details on the machine learning and AI models, including quantitative experimental results. We have added in precise definitions of mental wellness and serious mental illnesses in a new Introduction section. We have made it clear that we are describing results from implementation in the Abstract and Introduction. These changes address all requests by the reviewers.