Helping Therapists with NLP-Annotated Recommendation

Abstract

We propose a recommendation system that suggests treatment strategies to a therapist during the psychotherapy session in real-time. Our system uses a turn-level rating mechanism that predicts the therapeutic outcome by computing a similarity score between the deep embedding of a scoring inventory, and the current sentence that the patient is speaking. The system automatically transcribes a continuous audio stream and separates it into turns of the patient and of the therapist and perform real-time inference of their therapeutic working alliance. The dialogue pairs along with their computed working alliance as ratings are then fed into a deep reinforcement learning recommendation system where the sessions are treated as users and the topics are treated as items. Other than evaluating the empirical advantages of the core components on an existing dataset of psychotherapy sessions, we demonstrate the efectiveness of this system in a web app.

Keywords

• eol>Computational psychiatry
• Recommendation system
• Natural language processing
• Psychotherapy
• Reinforcement learning
• Deep learning

1. Introduction

Mental illness is not only a severe healthcare problem in

the US (1 in 5 estimated by National Institute of Mental Health) but also a major global issue [ 1 ]. However, most countries including the states face severe shortage of mental health practioners, such as psychiatrists and clincal psychologists [ 2 ]. In recent two years, this demand gap was especially amplified by the toll of COVID-19 pandemic on everyone’s mental health [ 3 ]. Current education systems and training programs cannot catch up to this trend because each licensed therapist requires years of continual learning and supervised training. Even when a therapist is ripe for independent practice, many still seek weekly supervision from “supervisors”, who are usually a more senior therapist that have seen many more years of patients and serve as “a crucial triad of learning dificulties that tend to confront beginning therapists in their training” [ 4 ]. These supervisors provide necessary guidance and periodic feedback to junior therapists with Figure 1: Major components of the system. respect to their development of mindedness, psychotherapist identities and treatment roadblocks they face in their own cases. are conducting their own psychotherapy. Like a super

In this work, we present SupervisorBot, a virtual AI visor, SupervisorBot ofers feedback and guidance that companion that provides real-time feedback and recom- are case-dependent. Like a supervisor, SupervisorBot has mends treatment strategy to the therapists while they seen thousands of historical therapy sessions and case studies. The base of our recommendation system relies TBhaeihraapnisLtsinw, iGthuiNllLePrm-AonnCoetcactheidaRnedcoDmjamlleenldBaotiuonne.foIunfJ.o2i0n2t3P.rHoceelepdiinnggs on a rating system that evaluates how good a treatment of the ACM IUI 2023 Workshops. Sydney, Australia, 7 pages strategy is. As the mental state of a patient can be compli* Corresponding author. cated to characterize, we gravitates our approach towards $ [e-mail] (B. Lin); [e-mail] well-defined clinical outcomes. The working alliance is (G. Cecchi); [e-mail] (D. Bounefouf) such a psychological concept that is shown to be highly € h00tt0p0s-:0//0w02w-7w9.7n9e-u5r5o0i9nf(eBr.enLicne).com/ (B. Lin) predictive of the success of psychotherapy in clinical © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License setting [ 5 ]. It describes several important cognitive and CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) emotional components of the relationship between these two agents in conversation, including the agreement on (WAI), a set of self-report measurement questionnaire the goals to be achieved and the tasks to be carried out, that quantifies the therapeutic bond, task agreement, and and the bond, trust and respect to be established over goal agreement [11, 12, 13]. Operationally, our goal is the course of the dialogue [ 6 ]. In [ 7 ], we developed a to derive from these 36 items three alliance scales: the natural language processing (NLP) approach to infer this task scale, the bond scale and the goal scale. They meaquantity in real-time as ratings. Here we propose the sures the three major themes of psychotherapy outcomes: Reinforced Recommendation model for Dialogue topics (1) the collaborative nature of the dialogue participants’ in psychiatric Disorders (R2D2), a the first ever recom- relationship; (2) the afective bond between them, and mendation system of dialogue topics proposed for the (3) their capabilities to agree on treatment-related shortpsychotherapy setting. It transcribes the session in real- term tasks and long-term goals. The score corresponding time, predicts the therapeutic outcome as a turn-level to the three scales comes from a key table which specirating, and recommends treatment strategy that is best ifes the positivity or the sign weight to be applied on the for the current context and state of the psychotherapy. questionnaire answer when summing in the end. It is the first step to solving the global issue of mental Transcription and real-time rating assessment. health by augmenting the treatment and education of Now we are ready for real-time quality annotation. Given clinical practitioners with a recommendation system of the audio stream for a given user, we first transcribe the therapeutic strategy. diarized audio stream with standard automatic speech recognition module [14]. Following the approach proposed in [ 7, 15, 16, 17 ], we embed both the dialogue turns 2. Methods and WAI items with deep sentence or paragraph embeddings (in this case, Doc2Vec [18]), and then compute the cosine similarity between the embedding vectors of the turn and its corresponding inventory vectors. With that, for each turn (either by patient or by therapist), we obtain a 36-dimension working alliance score, which we may save in a bidirectional relational database as in [19].

Topic modeling as recommendation items. First, we define the “items”, “users”, “contents” and “ratings” in our recommendation system. Here, the “items” the system recommends are treatment strategies. In this exFig 1 is an outline of the analytic framework. The continuous audio stream is fed into the system. First, we perform the speaker diarization (e.g. using real-time solutions such as [ 8, 9, 10 ]) which separates audio into dyads of doctor-patient, which are then transcribed into natural language turns for real-time downstream analyses.

Therapeutic quality ratings. After we obtain a relatively well diarization result, we can configure the quality assessment setting by specifying a proper inventory.

In this system, we use the Working Alliance Inventory ample, we represents these strategies as a topic that the state is progressed to the next therapeutic states. As a therapist should initiate or continue for the next turn. first step, we evaluate three popular deep RL algorithms. Given a large text corpus of many psychotherapy ses- Based on the deterministic policy gradient in an actorsions, as in [20] we can first perform topic modeling to critic architecture, the Deep Deterministic Policy Gradiextract the main concepts discussed in the psychotherapy. ents (DDPG) [ 41 ] is a model-free algorithm for continous We use the Embedded Topic Model (ETM) [21] in this action spaces, and one of the first successful algorithms work because it was shown to create the most diverse to learn policies end-to-end. Building upon the Double concepts in psychological corpus [20]. In this study, we Q-Learning [ 42 ], Twin Delayed DDPG (TD3) [ 43 ] is a use annotate each turn with their most likely topic and similar solution is proposed to correct for the overestiidentifies seven unique topics (Topic 0 is about figuring mated value issue, and yields more competitive results out, self-discovery and reminiscence; Topic 1 is about in various game settings. As the online data collection play. Topic 2 is about anger, scare and sadness. Topic 3 is of RL models are usually time consuming, in real world about counts. Topic 6 is about explicit ways to deal with industrial setting, these models are usually trained using stress, such as keep busying and reaching out for help. previously collected data. As a result, there is a growing Topic 7 is about numbers. Topic 8 is about continuation.) popularity of ofline reinforcement learning methods [ 44 ].

Recommendation system setting. Then, we pair Among them, Batch Constrained Q-Learning (BCQ) [ 45 ] these “items” with the “users” and “contents”, which in is the first continuous control deep RL algorithm with our case, would be the patientID, his or her previous competitive results in of policy evaluations by restricting turns, their aggregated formats and other meta data. For the agent’s exploration in action space. instance, we know that within each sessions, there exists Reinforced Recommendation model for Dialogue many pairs of turns, and they would belong to the same topics in psychiatric Disorders (R2D2). Combining “user”. However, one can also assign all turns belong to all the elements, we have our R2D2 model (Figure 2A). one clinical label, or all turns related to a certain topic Here we identify each session as a user, and the states as one “user”. In this example, we choose the session are frames of dialogues which can be labelled their topics ids as users. And lastly, the “ratings” would be patient’s in real time and their ratings with a working alliance inferred alliance scores predictive of the therapeutic out- (WA) inference module. The reinforcement learning core, comes. Creating this database from historical data, we powered by deep RL, predicts the best action represented can train our system. Since we have defined our users, by an embedding for the items (topics). This embedding items, contents and ratings, the recommendation engine can be pre-computed, for instance, using dimension recan be easily crafted with content-based [22, 23, 24] and duction techniques to find clusters of diferent topics in collaborative filtering [ 25, 26, 27, 28]. Since our session a low-dimensional space. We use the Doc2Vec embedturns are sequential and can specify a state or times- ding of the original dialogue turns, averaged by their tamp, it might be suitable for RL [29, 30, 31] and session- topic labels, such that each action (i.e. the topic id) have based methods [32, 33, 34], which can be neuroscience an averaged representation in the sentence embedding or psychiatry-inspired [35, 36, 37, 38] to provide further space. This action representation can be translated into interpretable clinical insights. During the deployment, a topic label with nearest neighbor, and a given dialogue our system registers our session as a new “user” if we response will be selected from the historical dialogue adopt a session-based item, providing punctuated rater data to continue the conversation. The reward can then evaluations as inference anchors [ 39 ]. Next steps include be computed using the working alliance rate. predicting these inference anchors as states (like [20? ]) and training chatbots as reinforcement learning agents given these states. 3. Empirical results apDpreoeapcrheeisn.foRrecienmforecnetmleenatr nleianrgnirnegcoamppmroeanchdeastiaorne Experimental setting. To evaluate the recommendation efectively applied in language and speech tasks (as re- systems, we preprocess the Alex Street psychotherapy viewed in [ 40 ]), among which recommendation is an dataset 1, which consists of transcribed recordings of over important use case. As shown in Figure 2, the reinforce- 950 therapy sessions between multiple anonymized therament learning environment is formulated such that the pists and patients, into a recommendation system format recommendation agent takes an action by recommend- (219,999 recommendation actions) and then split it into ing a strategy (say, a discussion topic). And the therapist 95/5 train-test sets. The dataset consists of four types of will interact with the patient taking that suggestion into psychiatric conditions: anxiety, depression, schizophreaccount. The dialogue interaction, in turn, has a quality nia and suicidal cases. We train R2D2 on each of the text evaluation of some sort (say, the therapeutic working corpus, as well as on all four together. To set up the batch alliance score). This serves as a reward to the recommen- 1https://alexanderstreet.com/products/counseling-anddation agent to update its weights. In the meanwhile, the psychotherapy-transcripts-series training for reinforcement learning, we cut the turns into Table 1 frames of 10 turn pairs and use a batch size of 32. We Pearson’s r of the actual actions taken in the test set with represent the action spaces (the topics to recommend) their predicted actions in three candidate embedding spaces: the averaged 300- Anxi Depr Schi Suic All dimension Doc2Vec embedding for each topic, the aver- R2D2-DDPG-TASK 0.3796 0.3376 0.1556 0.3292 0.0578 aged 36-dimension principal component analysis (PCA) RR22DD22--DDDDPPGG--BGOONALD 00..02746117 00..33863882 00..14553899 -00..00827130 00..12425453 embedding, and the averaged 2-dimension Uniform Man- R2D2-TD3-TASK 0.0707 0.1310 0.0443 0.3188 0.3357 ifold Approximation and Projection (UMAP) embedding. RR22DD22--TTDD33--BGOONALD 00..20091884 00..32326232 00..40590989 00..23004740 00..31710615 Due to the space limit, we only present the results for the R2D2-BCQ-TASK 0.1128 0.4042 0.1401 0.1422 0.0825 ifrst embedding, but leave the others in appendix. We R2D2-BCQ-BOND 0.0778 0.0876 0.0987 0.4152 0.0885 train the R2D2 with three reinforcement learning agents R2D2-BCQ-GOAL 0.0810 0.1231 0.0833 0.0788 0.0780 (DDPG, TD3 and BCQ) each for 50 epochs, where their losses consistently drop and converge in a stable way. of deep reinforcement learning and model architectures. Based on the loss curve, there are no overfitting in all Ethical considerations. Following the ethical guidemodel training processes. lines in [ 46, 47 ] and the operational suggestions in [ 48 ],

Empirical results. To evaluate the performance of the we make sure that all training examples that we evalthree recommendation agents, we compute the Pearson’s uate on are properly anonymized with pre- and postr of the recommended actions with their corresponding processing techniques, and disclaim that these investigaground truth actions the test set (Table 1). Since we are tions are proof of concept and require extensive caution the first system in this application problem, there are no to prevent from the pitfall of over-interpretation. state-of-the-art or baseline so far. Instead, we compare among variants of R2D2. Other than testing on diferent subset of the datasets and reinforcement learning algo- 4. Web-Based System: rithms, we also use three diferent scales of working al- SupervisorBot liance as our ratings: task, bond and goal, which measures diferent aspects of emotional alignments in psychother- “SupervisorBot” is an interactive web-based system (Fig 3 apy. We observe that the best performing model for four [ 39 ]). We first give users the instructions on how to use disorders are: R2D2-DDPG-TASK for depression sessions the system. Then they are lead to input their own invenwith a correlation of 0.3796, R2D2-BCQ-TASK for depres- tory used to evaluate the dialogue quality. In this case, we sion session (0.4042), R2D2-TD3-GOAL for schizophrenia put in a default one, using the working alliance inventory. sessions (0.4599) and R2D2-BCQ-BOND for suicidal ses- They are guided to input the score scale corresponding sions (0.4152). If we consider all four classes together, to each inventory item and click on “Analyze” to finalize. R2D2-TD3-GOAL appears to be the best performing mod- In the speaker diarization part, we compute and visualize els (0.3765). We notice that the DDPG and TD3 bases of the Mel Frequency Cepstral Coeficients (MFCC) in a slidR2D2 yields similar rankings among using three working ing window fashion given the real-time audio input from alliance scales as their ratings, while the BCQ tends not microphone, with the MFCC bands color coded in the to. For instance, in schizophrenia cases, the alignment in page. Finishing these two steps as the preparation, the the goal scale appear to provide a far more advantageous system is now running, and the therapist can sit back and recommendation prediction than the other two implicit go on with the session. The app now moves to the annofeedbacks (task and bond alignments), while in R2D2- tation panel, where the therapist can see that a transcript BCQ, the efect is less pronounced. For specific disorders, is displayed, along with who is speaking. The computed R2D2-DDPG is the recommender winner for anxiety, de- alliance score in the three scales are also dynamically pression and schizophrenia, and R2D2-TD3 is the winner displayed in real-time according to the content of the for suicidal cases (which should be taken with a grain dialogue turn. This is helpful information to assist the of salt considering the small amount of data we have therapist. And in our last panel, we have our recommenon them). When pooling the sessions of four disorders dation guidance. The topics to choose from are ranked together, the recommender winner appears to be R2D2- and top N are displayed. The therapist can use it as a hint TD3, which may suggest that R2D2-TD3, given its twin and initiate his response given a top recommendation. delayed mechanism to correct for value overestimation, The system will transcribe his response and highlight the are better suited for heterogeneous rather than homoge- topic he most likely ended up choosing in the last round, neous cases. It was a surprise that R2D2-BCQ doesn’t and save that information as part of historical data. The demonstrate in our dataset, an advantage to constrain the system refreshes its parameters at the end of each session possible extrapolation errors by the non-ofline methods. to fit new data.

This evaluation provides a proof of concept. Future work will focus on systematically comparing a larger spectrum

5. Conclusions and Future Directions

In this work, we provide a practical example of how a

real-time recommendation system can help therapists better treat their patients in psychotherapy sessions with informative clinical annotations and recommendations of treatment strategies with deep reinforcement learning.

Although in this example, the strategies are the topics for the therapist to initiate or continue, the same approach can be extended to more complex and nuanced treatment suggestions. For instance, in the ABC approach of cognitive behavioral therapy (CBT), our system can suggest a belief (B) to guide the patients to better understand the causality between the activating event (A) and its consequence (C).

Before we conclude, another interesting perspective to view this line of research is hidden in Figure 1: while the recommendation agent is driven by reinforcement learning, the therapist (and even patient) have their agency which updates under the reinforcement learning principles. For instance, the patient can directly ofer feedback to the therapists. And given the feedback, the therapist may adjust his or her internal model to weigh on the quality of the suggestions by the recommendation agent.

Next steps include modeling these theory of minds and confidence levels in this multi-participant human computer interaction setting. providing punctuated rater evaluations as inference anchors. Next steps include predicting these inference anchors as states (like [ 20, 49 ]) and training chatbots as reinforcement learning agents given these states (like [36, 37, 38]).

INTERSPEECH, 2020. Neural collaborative filtering, in: Proceedings of [10] B. Lin, X. Zhang, Speaker diarization as a fully the 26th international conference on world wide online learning problem in minivox, arXiv preprint web, 2017, pp. 173–182.

arXiv:2006.04376 (2020). [27] Y. Koren, S. Rendle, R. Bell, Advances in collabo[11] A. O. Horvath, An exploratory study of the work- rative filtering, Recommender systems handbook ing alliance: Its measurement and relationship to (2022) 91–142.

therapy outcome (1981). [28] X. Su, T. M. Khoshgoftaar, A survey of collabo[12] T. J. Tracey, A. M. Kokotovic, Factor structure of rative filtering techniques, Advances in artificial the working alliance inventory., Psychological As- intelligence 2009 (2009). sessment: A journal of consulting and clinical psy- [29] G. Zheng, F. Zhang, Z. Zheng, Y. Xiang, N. J. Yuan, chology 1 (1989) 207. X. Xie, Z. Li, Drn: A deep reinforcement learning [13] D. J. Martin, J. P. Garske, M. K. Davis, Relation of framework for news recommendation, in: Proceedthe therapeutic alliance with outcome and other ings of the 2018 world wide web conference, 2018, variables: a meta-analytic review., Journal of con- pp. 167–176.

sulting and clinical psychology 68 (2000) 438. [30] X. Wang, Y. Wang, D. Hsu, Y. Wang, Exploration [14] J. Adorf, Web speech api, KTH Royal Institute of in interactive personalized music recommendation:

Technology (2013) 1–11. a reinforcement learning approach, ACM TOMM [15] B. Lin, G. Cecchi, D. Bounefouf, Working alliance 11 (2014) 1–22.

transformer for psychotherapy dialogue classifica- [31] L. Zou, L. Xia, P. Du, Z. Zhang, T. Bai, W. Liu, J.-Y. tion, arXiv preprint arXiv:2210.15603 (2022). Nie, D. Yin, Pseudo dyna-q: A reinforcement learn[16] B. Lin, Personality efect on psychotherapy out- ing framework for interactive recommendation, in: come: A predictive natural language processing Proceedings of the 13th International Conference framework, arXiv preprint (2022). on Web Search and Data Mining, 2020, pp. 816–824. [17] B. Lin, Voice2Alliance: automatic speaker diariza- [32] J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, J. Ma, Neution and quality assurance of conversational align- ral attentive session-based recommendation, in: ment, in: INTERSPEECH, 2022. Proceedings of the 2017 ACM on Conference on [18] Q. Le, T. Mikolov, Distributed representations of Information and Knowledge Management, 2017, pp. sentences and documents, in: International confer- 1419–1428. ence on machine learning, PMLR, 2014, pp. 1188– [33] S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, T. Tan, 1196. Session-based recommendation with graph neural [19] B. Lin, Knowledge management system with nlp- networks, in: Proceedings of AAAI, volume 33, assisted annotations: A brief survey and outlook, 2019, pp. 346–353.

in: CIKM Workshops, 2022. [34] M. Ludewig, D. Jannach, Evaluation of session[20] B. Lin, D. Bounefouf, G. Cecchi, R. Tejwani, Neural based recommendation algorithms, User Modeling topic modeling of psychotherapy sessions, arXiv and User-Adapted Interaction 28 (2018) 331–390. preprint arXiv:2204.10189 (2022). [35] B. Lin, D. Bounefouf, G. Cecchi, Split Q Learn[21] R. Wang, X. Hu, D. Zhou, Y. He, Y. Xiong, C. Ye, ing: Reinforcement Learning with Two-Stream ReH. Xu, Neural topic modeling with bidirectional wards, in: Proceedings of the Twenty-Eighth Interadversarial training, in: Proceedings of the 58th An- national Joint Conference on Artificial Intelligence, nual Meeting of the Association for Computational IJCAI-19, International Joint Conferences on ArLinguistics, 2020, pp. 340–350. tificial Intelligence Organization, 2019, pp. 6448– [22] M. J. Pazzani, D. Billsus, Content-based recommen- 6449. URL: https://doi.org/10.24963/ijcai.2019/913. dation systems, in: The adaptive web, Springer, doi:10.24963/ijcai.2019/913. 2007, pp. 325–341. [36] B. Lin, G. Cecchi, D. Bounefouf, J. Reinen, I. Rish, A [23] C. Basu, H. Hirsh, W. Cohen, et al., Recommen- story of two streams: Reinforcement learning moddation as classification: Using social and content- els from human behavior and neuropsychiatry, in: based information in recommendation, in: Aaai/i- Proceedings of the 19th International Conference aai, 1998, pp. 714–720. on Autonomous Agents and MultiAgent Systems, [24] C. C. Aggarwal, et al., Recommender systems, vol- 2020, pp. 744–752.

ume 1, Springer, 2016. [37] B. Lin, G. Cecchi, D. Bounefouf, J. Reinen, I. Rish, [25] B. Sarwar, G. Karypis, J. Konstan, J. Riedl, Item- Models of human behavioral agents in bandits, conbased collaborative filtering recommendation algo- textual bandits and rl, in: International Workrithms, in: Proceedings of international conference shop on Human Brain and Artificial Intelligence, on World Wide Web, 2001, pp. 285–295. Springer, 2021, pp. 14–33. [26] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, T.-S. Chua, [38] B. Lin, G. Cecchi, D. Bounefouf, J. Reinen, I. Rish,

References

[1] V. Patel , S. Saxena , C. Lund , G. Thornicroft , F. Baingana , P. Bolton , D. Chisholm , P. Y. Collins, J. L. Cooper , J. Eaton , et al., The lancet commission on global mental health and sustainable development , The lancet 392 ( 2018 ) 1553 - 1598 .

[2] A. Satiani , J. Niedermier , B. Satiani , D. P. Svendsen , Projected workforce of psychiatrists in the united states: a population analysis , Psychiatric Services 69 ( 2018 ) 710 - 713 .

[3] X. Wang , S. Hegde , C. Son , B. Keller, A. Smith , F. Sasangohar , et al., Investigating mental health of us college students during the covid-19 pandemic: cross-sectional survey study , Journal of medical Internet research 22 ( 2020 ) e22817 .

[4] C. E. Watkins Jr , Being and becoming a psychotherapy supervisor: The crucial triad of learning dififculties , American Journal of Psychotherapy 67 ( 2013 ) 134 - 150 .

[5] B. E. Wampold , How important are the common factors in psychotherapy? an update , World Psych 14 ( 2015 ) 270 - 277 .

[6] E. S. Bordin , The generalizability of the psychoanalytic concept of the working alliance ., Psychotherapy: Theory, research & practice ( 1979 ) 252 .

[7] B. Lin , G. Cecchi , D. Bounefouf , Deep annotation of therapeutic working alliance in psychotherapy , arXiv preprint arXiv:2204.05522 ( 2022 ).

[8] B. Lin , X. Zhang, Speaker diarization as a fully online bandit learning problem in minivox , in: ACML, 2021 , pp. 1660 - 1674 .

[9] B. Lin , X. Zhang, Voiceid on the fly: A speaker recognition system that learns from scratch, in: Unified models of human behavioral agents in bandits, contextual bandits and rl , arXiv preprint arXiv: 2005 . 04544 ( 2020 ).

[39] B. Lin , G. Cecchi , D. Bounefouf , Supervisorbot: Nlp-annotated real-time recommendations of psychotherapy treatment strategies with deep reinforcement learning , arXiv preprint arXiv:2208.13077 ( 2022 ).

[40] B. Lin , Reinforcement learning and bandits for speech and language processing: Tutorial, review and outlook , arXiv preprint arXiv:2210.13623 ( 2022 ).

[41] T. P. Lillicrap , J. J. Hunt , A. Pritzel , N. Heess , T. Erez , Y. Tassa , D. Silver , D. Wierstra , Continuous control with deep reinforcement learning , arXiv preprint arXiv:1509.02971 ( 2015 ).

[42] H. Hasselt , Double q-learning , Advances in neural information processing systems 23 ( 2010 ).

[43] S. Fujimoto , H. Hoof , D. Meger , Addressing function approximation error in actor-critic methods , in: International conference on machine learning, PMLR , 2018 , pp. 1587 - 1596 .

[44] S. Levine , A. Kumar , G. Tucker, J. Fu , Ofline reinforcement learning: Tutorial, review, and perspectives on open problems , arXiv preprint arXiv: 2005 . 01643 ( 2020 ).

[45] S. Fujimoto , D. Meger , D. Precup , Of-policy deep reinforcement learning without exploration , in: International conference on machine learning, PMLR , 2019 , pp. 2052 - 2062 .

[46] T. Matthews , K. O'Leary , A. Turner , M. Sleeper , J. P. Woelfer , M. Shelton , C. Manthorne , E. F. Churchill , S. Consolvo , Stories from survivors: Privacy & security practices when coping with intimate partner abuse , in: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , 2017 , pp. 2189 - 2201 .

[47] S. Graham , C. Depp , E. E. Lee , C. Nebeker , X. Tu , H. -C. Kim , D. V. Jeste , Artificial intelligence for mental health and mental illnesses: an overview , Current psychiatry reports 21 ( 2019 ) 1 - 18 .

[48] B. Lin , Computational inference in cognitive science: Operational, societal and ethical considerations , arXiv preprint arXiv:2210.13526 ( 2022 ).

[49] B. Lin , D. Bounefouf , G. Cecchi, Predicting human decision making in psychological tasks with recurrent neural networks , PloS one ( 2022 ).