The research project RiMotivAzione aims at helping post-stroke patients who are following an arm and hand rehabilitation path. In this paper we present the RiMotivAzione corpus, the first collection of dialogues between physiotherapists and patients recorded in an Italian hospital and annotated following the RIAS annotation protocol. We describe the dataset, the methodologies applied and our first investigations on relevant features of the dialogue process. The corpus was the basis for the design of a conversational interface integrated with a wearable device for rehabilitation, to be used by the patient during the exercises that he or she may perform independently.1
In recent years, computational linguistics and
medical research have started to collaborate in
order to analyze the communication in the
healthcare domain, in particular between clinicians and
patients. From a medical perspective, linguistic
analysis and dialogue modeling can be used to
better understand and potentially enhance
communication in different healthcare settings
Natural Language Processing (NLP)
technologies are also used to develop new communicative
tools, e.g. virtual assistants, to alleviate the
burden on medical personnel or shift to a home-based
patient-centered model of care. Through mHealth
(mobile health), for example, people can receive
assistance at home, and monitoring devices can
check the well-being of a person (Sezgin et al.,
1Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
2018). A recent review of scientific literature
about Artificial Intelligence and IoT in healthcare
can b
The research project RiMotivAzione aims at helping the patients who suffered from a stroke and are following an arm and hand rehabilitation path. The goal is to motivate the patients to follow the assigned exercises through the use of a new wearable device with motion sensors developed by the Istituto Italiano di Tecnologia (IIT), integrated with a visual App and a conversational interface. This last component guides the user through the therapeutic path proposing the exercises, giving advice and asking for feedback.
The implementation of voice technologies in the
healthcare domain allows for patients with motor
impairments to interact with devices through
spoken language
As we described in the previous section, the study
of communication and conversation in the
medical domain is growing in the last years, as well
as the introduction of conversational agents in the
healthcare sector. A review of current applications
and evaluation measures of conversational agents
used for health-related purposes can be found, for
example, in
The analysis of speech acts and conversational
interaction can play a relevant role in dialogue
modeling for healthcare thanks to the
classification of utterances, the analysis of dialogue turns
and threads, the discovery of recurrent patterns.
Speech acts have been investigated in linguistics
and computational linguistics for long.
Specifically, the task of automatic speech act
recognition has been addressed leveraging both
supervised and unsupervised approach
In the RiMotivAzione project, we deal with
physiotherapy sessions in a hospital. The task is
to collect and analyze para-linguistic and
linguistic data, according to the aforementioned goal of
the research project. In this specific setting, i.e.
conversational analysis of physician-patient
discourse, the most widely used method is the Roter
Interaction Analysis System (RIAS). RIAS was
developed as a tagset for coding medical dialogue
since 1991 by Debra Roter et al.
The RiMotivAzione corpus includes two complete cycles of physiotherapy sessions with two patients in post-stroke rehabilitation (namely, P1 and P2) and three physiotherapists (T1, T2, T3). The interviews were video recorded in IRCCS Fondazione Ospedale "San Camillo" in Venice. Each session lasted about 1 hour. The physiotherapy cycle for patient P1 included 14 sessions, while P2 took 16 sessions. Therefore the total duration of recordings is about 30 hours.
The patients were carefully selected by the doctors, since they must present some features. Above all, they had to agree to be part of the experimentation and they needed to talk in Italian. In an environment where dialect is still strong, their ability to speak Italian was not to be treated lightly. Moreover, the patients did not have to present any issues related to aphasia. These requirements restrained the viable options to two candidates.
Both speakers were encouraged to talk freely about any topic that may have emerged. Their only constraint was the use of Italian; when people slipped into dialectal terminology (in this case, Venetian), it was explicitly marked with the <dialect> tag in the corpus. The audio tracks were transcribed and annotated following Savy’s (2005) guidelines for orthographic transcription for spoken Italian, where applicable. As a pre-processing, we used two Automatic Speech Recognition (ASR) systems, i.e. Google Speechto-Text and Nuance Transcription Engine. Automatic transcriptions were corrected manually and anonymized. Video and audio tracks have been separately saved for future projects.
Overlapping between the two speakers and pauses were not marked, as it was not relevant to our study. Similarly, any intervention in the dialogue from a third party was not transcribed since our interest was solely in the doctor and patient’s linguistic behaviours. Each dialogue turn of the corpus was annotated by two different annotators following the RIAS guidelines. All the annotators have a background in linguistics and a specific education about linguistic corpora. As a single dialogue turn may contain more than one sentence and more than one speech act, the tags assigned to each turn may be more than one.
RIAS tagset includes 29 categories divided in four macro-categories called Medical Interview Functions (MIF) that cover the majority of the exchanges between a doctor and a patient: Data Gathering, Information Exchange, Emotional Expression and Responsiveness, Partnership Building and Activation. Table 1 contains the list of categories occurring at least 200 times in the corpus, together with examples.
To the best of the authors’ knowledge, the RIAS system has never been used to annotate sessions of physiotherapy until now. This means that not all of the tags applied completely to the situation, or that some tags may be under-represented compared to other studies: for instance, the tag Concerns was applied to few sentences, since patients in physiotherapy sessions may inherently express less concern than oncological patients.
All the categories
Another issue concerns the use of irony. Specifically, Patient 2 heavily employed irony while talking to the therapist, even when the dialogue concerned his health and well-being. Irony is hard to interpret, resulting in the difficulty to assign correctly a tag to those sentences. Tag Jokes was used in this case, and where inappropriate, a discussion between the annotators oriented the choice.
As the annotation task was difficult and it was inherently affected by subjectivity, we measured the resulting inter-annotator agreement and we put in place strategies to solve the disagreement, in order to annotate all the dialogue turns. The agreement calculated at this stage, according to the Cohen’s score, was promising (k = 0.63). In case of disagreement (about 25% of the data), the process was followed by reconciliation or a final decision by a super annotator, where the two annotators could not overcome the disagreement.
The RiMotivAzione corpus has been built and archived according to GDPR norms. It is not publicly available but it can be requested to the authors for research purposes. 4
The RiMotivAzione corpus contains about 98778 tokens. The total number of dialogue turns is 7670: 3377 dialogue turns in P1 sessions, 4293 in P2 sessions.
In Table 2 and Table 3 we reported the number of types, tokens, the ratio between types and tokens (the Lexical Richness Index) and the number of questions for the two patients.
It is worth noticing that Lexical Richness Index
ranges from 0 to 1 and it is closer to 0 in the
doctors’ speech, meaning that medical personnel
employ a poorer vocabulary while talking to a patient.
This is due to the fact that a therapist needs to stick
to a protocol and cannot digress over a certain
limit. On the other hand, the patient talks
quantitatively less: he pronounces fewer words, and
most of the time those words are simple answers
to the questions posed by the clinician. The patient
talks less but he can wander more across
conversation topics: he may disclose some personal detail
about his life or just chit chat. This behavior is
actually encouraged by the therapist, since it makes
the therapy session less dull and more spontaneous
for both the participants
As the main goal of the study is to replicate
the clinician’s communicative style onto a
conversational interface, the major interest is on how
the therapists talk, rather than the patients.
Patients’ manner of speaking is taken into
consideration when imagining all the orders or phrases
that the user could say to the voice assistant to
express his needs. Table 4 and Table 5 list the most
frequent Verbs and Adjectives pronounced by the
physiotherapists. Apart from "Okay", which is the
most frequent word for both therapists (1231 and
1019 occurrences), both therapists often use
adjectives of positive value: bravissimo, bravo, ottimo,
buono. Other frequent words are mainly verbs
expressed at the first plural person, such as we do,
we’ll try, or equivalent expressions (let’s relax).
The use of the "we" is a communication element
that aims at putting on the same level the
clinician and the patient; the goal is to make the
patient feel more comfortable and therefore
enhancing the probability of therapy adherence. At the
same time, adjectives such as "good" and "very
good" praise the patient’s efforts, underlining the
progress he is making. The psychological
component is of paramount importance during
physiotherapy, especially for patients that suffered a
stroke
The quantitative analysis operated over the annotated corpus confirms the qualitative remarks made so far. In Figure 1 we present the distribution of dialogue tags, both for patients and therapists, i.e. the distribution of utterance type according to RIAS categories. We plotted on a logarithmic scale the frequencies of the tags.
Sentences annotated as Social talk were
abundant, while those marked as Concerns were
copious just for a patient, because he was
frustrated about his health situation and the
difficulties to manage the physiotherapy. During the
sessions with Patient 1, the physiotherapist was able
to engage a conversation about a hobby of his
(motorcycles); even though this discussion topic
is not relevant to the therapy, the fact that they
were talking about something interesting for the
patient contributed to the improvement of his
medical condition
All of these conversational elements are put in place willingly by the clinician and, even more, it is the style patients are used to. In the voice assistant design we try to mirror these strategies, providing praises when appropriate and asking questions to constantly monitor the user’s well-being. The data extracted from the transcription and the annotation represents the most frequent linguistic behaviors emerged during the conversations. These patterns were used to build the conversational style and infrastructure of the dialogue system. 5
We created a corpus of conversations between patients and clinicians, in Italian, and we annotated the dialogue turns according to the Roter Interaction Analysis System (RIAS). This corpus was the first step in the design of a conversational interface integrated with a smart wearable device, to guide and assist the patients through the exercises assigned by the physiotherapist.
The first step in the future work will be to deepen the linguistic analysis conducted on the corpus, especially regarding the tagged dialogue acts. A stronger qualitative investigation over the data will be carried out. The second step will be to enrich the dataset: unfortunately, only two patients were deemed appropriate for the experimentation, while a corpus should contain dialogues from more speakers.
The RiMotivAzione corpus can be requested to the authors for research purposes.
The system prototype will be tested in San Camillo Hospital by a set of stroke patients, following the clinical trial procedures. Thanks to the results of the test, we will produce experimental data to investigate if and how a voice assistant integrated with a wearable device can increase the effectiveness of the therapy. 6
RiMotivAzione is a two-year Research and Innovation project supported by POR FESR 20142020 Regione Piemonte. The partners are Koiné Sistemi, CELI, IRCCS Fondazione Ospedale San Camillo, Synesthesia, Istituto Italiano di Tecnologia (IIT) and Morecognition. We are thankful to our colleagues and project partners, in particular Paolo Ariano and Nicoló Celadon.
P. Basil
D. Beltrami, G. Gagliardi, Rossini Favretti, E. R.,
Ghidoni, F. Tambu
C.L. Chang, B.K. Park, and S.S. Kim. 2013. Conversational analysis of medical discourse in rehabilitation: A study in korea. The journal of spinal cord medicine, 36(1):24–30.
I. Chaoua, D. R. Recupero, S. Consoli, A. Harma, and
R. H
C.M. Delany, I. Edwards, G.M. Jensen, and E. Skinner. 2010. Closing the gap between ethics knowledge and practice through active engagement: an applied model of physical therapy ethics. Physical Therapy, 90(7):1068–1078.
Andreas Stolcke, Klaus Ries, and Elizabeth Shriberg Noah Coccaro. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3).