=Paper=
{{Paper
|id=Vol-1956/GHItaly17_paper_06
|storemode=property
|title=Euphoni: A System to Support Speech Therapy
|pdfUrl=https://ceur-ws.org/Vol-1956/GHItaly17_paper_06.pdf
|volume=Vol-1956
|authors=Ombretta Gaggi,Nicola Favaro
|dblpUrl=https://dblp.org/rec/conf/chitaly/GaggiF17
}}
==Euphoni: A System to Support Speech Therapy==
https://ceur-ws.org/Vol-1956/GHItaly17_paper_06.pdf
Euphoni: a system to support speech therapy
Nicola Favaro, Ombretta Gaggi Ilaria Gatto, Cristian Leorin, Marco Simoni
Department of Mathematics, University of Padua Department of Neuroscience, University of Padua
nicola.favaro.4@studenti.unipd.it, ilaria.gatto@studenti.unipd.it,
gaggi@math.unipd.it cristian.leorin@unipd.it,
marco.simoni.1@studenti.unipd.it
ABSTRACT • difficulty in remembering how to reproduce a target voice
Many rehabilitation therapies ask patients to perform many technique without a clinician assistance,
exercises every day. This requirement strongly stresses the
patient who often abandons the therapy too early. This is par- • difficulty in judging the results and the accuracy of their
ticulary true in speech therapy which usually has an estimated efforts and
drop out before reaching the treatment goal of 65%, which is a
very high number. In this paper we propose Euphoni, a system • concerns about appearing unnatural when using the target
for distance care of voice problems. The system includes a technique conversation.
mobile application which allows to establish a voice therapy The main difficulty is related to remembering to practice and
protocol which is able to help the patient when performing use the target technique throughout the day. Patients report
exercises at home. Moreover, we use a gamification approach that they are busy, tired, upset and many of them declared that
to encourage the patient to hold on. they are not able to perform the exercises in absence of the
therapist.
ACM Classification Keywords
H.5.2. User Interfaces: Prototyping; J.3. Life and Medical Patients report that the presence of the voice therapist and
Sciences: Medical information systems the quality of their relationship with him (or her) is crucial to
avoid drop out. Unfortunately, daily meeting with the therapist
Author Keywords is not a possible solution for all patients, so other answers to
Serious Games; Mobile Applications; Speech Therapy this problem must be investigated.
Since smartphones are always on the user’s pocket, the use
INTRODUCTION
of a mobile application can help to continuously assist the
Drop-out-from-therapy as well as poor patients adherence are
patient. More in details, a mobile application can be used in
well known phenomena which affect many medical treatments,
many different ways [18]:
lowering the effectiveness of many therapies. These problems
are particularly relevant in rehabilitation treatments, especially • as a support to remember to practice the exercises using alert
when patients are supposed to perform many exercises every messages, calendars, etc., e. g., Liang et al demonstrated
day. This is the case of clinical voice therapy, where therapy that text-based reminders have strong efficacy in improving
protocols typically require daily activities and to learn a target adherence to health behavior goals [13],
voice-production technique.
• as a feedback system for practice completion in order to
In voice therapy, patients perceive the treatment exercises as track healthy behaviors (e. g. the number of exercises done
very challenging [17]. Voice treatment has usually an esti- during the week) and
mated drop out before reaching the treatment goal of 65%,
which is a very high number, compared to other programs like, • as a tutorial to help patients producing the target technique
for example, physical exercise and smoking cessation [15]. independently providing them videos, recordings, or por-
Patients report a variety of barriers to treatment adherence be- tions of therapy practice between therapy sessions.
tween voice therapy sessions. Leer and Connor [19] described Mobile health platforms have the potential to extend voice
the reasons for poor patient adherence as: treatment from therapy session to patient’s environment, to
• poor motivation, increase the practise of the treatment, to motivate patients with
feedback and gamification elements, and to support patients.
To be effective, direct voice therapy protocols require daily
Permission to make digital or hard copies of all or part of this work for personal practice and generalization of a target voice production tech-
or academic purposes is granted without fee provided that copies are not made or nique between treatment session [19]. Technology has the
distributed for profit or commercial advantage and that copies bear this notice and the
GHITALY17:
full 1st first
citation on the Workshop
page. Toon
copyGames-Human Interaction,
otherwise, or republish, Aprilon 18th,
to post servers2017,
or to
potential to optimize patients’ adherence to health behavior
Cagliari, Italy.
redistribute to lists, requires prior specific permission and/or a fee. programs, assisting them anytime and anywhere, but speech
Copyright © 2017 for the individual papers by the papers' authors. Copying
Copyright © by the paper’s authors.
permitted for 1private
st and academic purposes.Interaction,
This volume is published and
and language therapists often report that they can’t rely to
GHITALY17: Workshop on Games-Human September 18th, 2017,
copyrighted
Cagliari, by its editors.
Italy. affordable and good applications.
In this paper, we describe Euphoni, a system for distance care Despite many good results in different clinical treatments, the
of voice problems. The system includes a mobile application possibility to use of technology in voice treatment is almost an
which allows to establish a voice therapy protocol which is unexplored land. In [19] Leer and Connor tried to predict pa-
able to help the patient when performing exercises at home. tients adherence to treatment measuring three social-cognitive
Moreover, the system allows to record audio and text messages factors: self-efficacy, goal commitment and therapeutic al-
and to send them to the voice therapist. liance [1], [2]. Moreover, the authors study how much a
MP4 video support could influence the same triad of factors.
The system does not replace traditional therapy protocols, The study demonstrated that adherence to treatment can be
but help patients by remembering the therapist’s instructions, predicted by this three social-cognitive factors and that mo-
encouraging communication with therapist even when patients bile technology can extend treatment to extraclinical settings.
are at home, scoring their progress and giving them sub-goal
This is extremely important because, if we use technology to
that can be easily reached.
enhance self-efficacy, goal commitment, and the therapeutic
alliance, hypothetically we can reduce rate of dropouts.
Lavaissièri and Melo developed Q-Voz[12], a mobile applica-
RELATED WORK tion for voice therapists which reminds patients on the times
The use of modern technologies in medical practise is quickly for practicing and contains some tutorial for proposed tech-
increasing to assist both therapists and patients [5, 7, 6, 8, niques. The application was submitted to seven evaluators that
11, 16]. Kato explores the positive aspects of using existing reported that it can be considered an auxiliary tool for voice
commercial video games for health improvements or surgical speech therapy.
training, and tailor-made game for particular disease group
in order to improve recovery and rehabilitation of patients PROEL METHOD
[11]. She defines Re-Mission, a game to help young patients Human voice is a sound generated by an elastic system, air and
to understand and deal with cancer: game characters represent vibrations which resonate into the vocal tract. An euphonic
the drug which destroys cells with cancer. Moreover, the game voice production is granted if there is balance of the pneumo-
also provides patients a forum to discuss together. phonic system. Every alteration of this specific balance creates
HelpMe! [5] is a system which supports the rehabilitation muscles’ rigidity and hypertone, and results in a vocal disfunc-
process for children affected by CVI (Cerebral Visual Impair- tion called dysphonia. At the beginning of a dysfunctional
ment) through the use of two serious games. It also includes a dysphonia there are always rigidity, loss of vocal strength and
control panel to allow the therapist to adapt the rehabilitative lack of lubrication.
exercises to each particular child and to follow the improve- The Proprioceptive-Elastic (ProEl) method [3] is one of the
ments of the patient. Moreover the system can help doctors methods used to treat voice alterations. It is a technique used
to perform a good assessment of a patient and to create a in voice therapy that has been developed since the late 90’s in
rehabilitation program. a collaboration between the CFL (Centre of Phoniatrics and
Many papers reported evidence to support the efficacy of mo- Speech Therapy), Santander, Spain and the ORL operational
bile approaches for a variety of health problems like diabetes unit of Bufalini Hospital, Cesena, Italy. The basic principles
[13], smoking cessation maintenance [20], dyslexia [8, 16], of this method are the sensory “bombarding” of the phonatory
etc. [16] integrates dyslexia predictors in a tablet game, to cap- apparatus with the aim to make the patient more aware about
ture children attention to obtain a more accurate measurement. his/her phonatory structure and body in general, to distend the
The authors evaluate several existing games for preschoolers muscular tension and lubricate the mucosa.
to derive a set of guidelines to design an optimal tablet game These goals are reached facing different moments of the vocal
for 5 years old children. Then, these guidelines are used to therapy. In particular:
develop Diesel-X, a game about a robot dog, Diesel, which
has to fight against a gang of criminal cats. • lighting phase: activation of the learning system mainly
using motivational strategies;
Gaggi et al. [8] developed a set of serious games to identify
developmental dyslexia at early stages and immediately start a • voice modification: try to obtain vocal changes in order to
therapy. These serious games were designed to be accessible give different sensations to the patient;
from any devices, both tablet using a touch interaction or a PC
using a keyboard and a mouse, so the doctor can choose the • leave a mark: try to build a new motor pattern by remem-
best interaction method depending on the age and the ability bering the voice target;
of the child.
• compare: try to focus on the differences between the voice
Successful mobile applications typically help to maintain ad- before and after the exercises;
herence using push notifications, SMS, messages to reinforce
healthy behaviors, real time feedback and patient self moni- • automate: generalize the new vocal production.
toring process. As an example, Fish’n’Steps [14] is a serious Usually the therapy follows specific steps:
game that links the walking activity with the growth and mov-
ing activity of a fish in a fish tank. The system uses push • make the patient aware of vocal risk factors and teach him
notifications to keep attention of the user on the game. how to control them;
• make the patient more and more conscious about his/her
voice production, his/her muscle’s tension, breath, posture,
tongue and palate: the patient has to become proprioceptive;
• make the patient elastic and agile in voice production, elim-
inating rigidity;
• make the patient’s voice projected and resonant in order to
preserve the vocal instrument using less energy and obtain-
ing the best result;
• make the patient feel happy when using his/her voice: this
is the highest goal of this therapy.
In addition, this method also uses larynx and muscles manip-
ulation in order to eliminate tensions. The strengths of the Figure 1. Overall architecture of the system.
ProEl method are:
• it is very concrete in many theoretical aspects, allowing
the patient to understand better all the problems and the THE EUPHONI SYSTEM
solutions; The system is mainly composed by three components as de-
picted in Figure 1:
• it proposes quick and easy exercises which are powerful;
• a mobile application,
• it is motivating and inclusive for all kind of patients.
• a server and
Description of the Exercises • a panel for the doctor.
The main keywords in ProEl method are lubrication, elasticity
and resonance. The ProEl method proposes many exercises The server is the most important component, it contains the
that has been built in order to gain these goals: in the following description of the exercises and records data about user perfor-
only exercises that have been implemented in the application mances. Doctors can manage patients through a web interface,
are discussed. i. e., the EuphoniPanel: they can personalize the set of ex-
ercises for each user, i. e., they can choose among a set of
As already mentioned, some exercises are really very sim- proposed exercises the ones that are suitable for the disease of
ple: for example, to lubricate the vocal tract we suggest to a particular patient, and check data about user performances.
increase the amount of drunk water; an exercise which is pro-
posed every day is to drink at least two liters of water per day. The mobile application is named EuphoniApp: it retrieved the
This allows the vocal folds to have less friction and to work exercises that each patient should do every day and proposed
properly. them to the patient. Its main goal is to support patients during
therapy in two ways:
Other exercises have the goal of improving the elasticity, by
teaching the “easing Postures”, i. e., postures which mod- • it proposes and explains the exercises that patients are sup-
ify the body balance point to allow a better and easier voice posed to do during the day,
production with less muscles strength. An example is to ask • it encourages the user to continue the therapy through the
patients to put their hands on their knees, which must be a little use of sub-goals and badges’ achievements.
bit bent. This position reduces the neck’s muscles activity, put
the tongue in the right position and allows the voice to flow Moreover, it records patient’s performances, and sends data to
easier. the server.
In order to make the voice more resonant the ProEl method The first time a patient uses the application he, or she, must
uses the sensory bombarding with thermal or tactile stimula- log in, and the application retrieves the exercises program
tions in the oral cavity using different kind of tools. For ex- decided by the therapist for that particular patient. Practice
ample, an exercise asks patient to put his/her hands on his/her is organized in five weeks, which contain exercises organized
mouth to make him/her aware of his/her vocal vibrations and in seven days. Each day may contain many exercises. Some
power. exercises are present every day, i. e. hydration exercise, others
can change according to the day and the particular patient.
Exercises described so far are just a few examples of how
Each exercise must be performed in a precise interval of time,
ProEl method works on dysfunctional dysphonia. Patients which can be in the morning, in the evening, or during the day.
are usually very happy and enthusiastic in facing this kind
of therapy. The only real problem consists in repeating the Since patients are required to practise every day, motivation
exercises at home, between different therapy sessions, which is an important aspect to consider: the most important goal of
take place twice a week. For this reason, we implemented the EuphoniApp is to avoid dropouts. Another important aspect
Euphoni system to support both patients and therapist. is also to guarantee a correct practise of the therapy, i. e.,
Figure 2. Screenshots of EuphoniApp which show the weeks and the days
sometimes it may happen that patients doesn’t respect the few: Figure 2(b) shows the flower after one day that the patient
exercises list or the timetable of the exercises. For this reason, completely follows the therapy, Figure 2(c) is the result of six
we use the gamification approach, which uses elements of days of exercises done. To blossom the flower completely the
game design in non-game contexts. patients have to perform all the exercises all the days of the
corresponding week. In Figure 2(a) the patient has done all
The game uses the metaphor of a lotus flower; we made this
the exercises all the days in the first week, only for two days
choice since the shape of this flower recalls the shape of the
in the second week, for six days in the third week and never in
vocal chords. Figure 2(a) shows the interface after the lo-
the fourth week.
gin: there are five lotus flowers, one for each therapy week,
filled with different colors. The flower denoted by number 4 Each exercise is a sub-goal through which the patient can reach
in the figure represents the current week. Future weeks are the main goal. Before performing an exercise, the patient can
represented only with a leaf, e. g. leaf with number 5 in the watch a tutorial video or read some text instruction (see Figure
figure. 3(b)). When the exercise is complete, he, or she, must mark it
as completed.
We keep the graphic simple, to allow all types of patients,
different for age and gender, to find the application simple. The EuphoniApp does not reward the patient after each exer-
The patient selects a week by touching the corresponding cise but only when the patient performs all the exercises of
flower. Figure 2(b) shows the interface for the week: initially a day. So, to increase patient’s engaging and motivation, we
the current day, i. e., today, is shown, but the user can scroll added other intermediate goals to the application, i. e. the
up and down to select days after or before, numbered from 1, badges, simple subgoal which has the function to keep the
i.e, Monday, to 7, i. e., Sunday. patient focused in the therapy and to encourage him/her with
rewarding. Some examples of badges are the following:
The game uses a task/progress/motivation loop [21] in order to
increase motivation thus maximizing the number of performed • complete all the daily exercises;
exercises. Every time patient performs an exercise he, or she,
can see his/her level of completion through a progress bar, • complete all the daily exercises for two consecutive days;
as depicted in Figure 3(a) which shows the interface for a • drink 10 litres of water;
single day. Moreover, completed exercises are marked with
a check mark, while expired ones are denoted by a red cross. • drink 30 litres of water;
To engage and motivate the patients, EuphoniApp also send
them positive feedbacks when they complete an exercise or • send the first message to the therapist;
the whole list of a day. • send five messages to the therapist.
In the game, the user’s goal is to blossom all the five lotus Figure 3(c) shows the panel with badges showed to the patient.
flowers. Upon completion of the entire list of exercises for a
day, the flower, that correspond to the current week, opens a To maximize the support to the patients, EuphoniApp provides
a communication service between the patient and the speech
Figure 3. Screenshots of EuphoniApp which show: (a) the list of exercises, (b) the details of an exercise and (c) the panel of badges
therapist: the user can record an audio message or digit some therapist uses EuphoniPanel to manage all the patients and
text that will be recorded by the server. The therapist can listen exercises assigned to them.
to the audio messages and read the text messages using the
EuphoniPanel. EuphoniPanel is a web interface, which shows the patients
records. The therapist can read or listen the messages sent
The system reminds the user to perform the exercises using a by patients, check if a patient has completed the list of exer-
push notification system that could be customized according cises or not and eventually change it. It has been developed
to patient’s time preferences. using standard web technologies, i. e., PHP, HTML, CSS and
Javascript, with a particular attention to accessibility.
Both EuphoniApp and EuphoniPanel communicate with a
Implementation Details MySQL database situated at the server side, which collects
EuphoniApp has been developed only for Android operating data about patients and exercises. Personal data about patients
system, but we are currently working to support all available are stored separately to preserve their privacy, even in case of
operating systems, using a framework for cross-platform mo- malicious access.
bile development [4]. The application has been developed
using the Java language with Android SDK 25 and Android
CONCLUSION AND FUTURE WORKS
Studio. We choose the Android platform rather than other
operating systems since, according to Gartner statistics [9], it In this paper, we propose Euphoni, a system to support speech
is the most widespread among users (80,7%). The application therapy. This kind of therapy requires patients to perform
is developed for Android 7 (Nougat) but it is retrocompati- several exercises every day, so the main goal of the system
ble until Android 4.1 (Jelly Bean), so the 98,2% of Android is to support and encourage the patient to avoid the drop-out-
devices are supported [10]. from-therapy phenomenon.
The application communicates with the server only when nec- The strong points of our system are:
essary, in order to limit the network usage, i. e. it connects to • the use of gamification to involve and encourage the patient;
the server to synchronize exercises when the user log in and to
change their state of completion. The communication is made • the use of a mobile application, EphoniApp, since we use a
using some Json Web Services which are used to implement tool commonly available in the patients’ pockets and we do
the authentication system, the download of the exercises and not ask patients to buy other devices. Moreover, previous
the upload of exercises’ scores and messages of the patients. works have demonstrated that the use of modern mobile
Parameters, such as user’s id, exercise id, exercise status, etc., technologies is very efficient in the treatment of health prob-
are sent using POST method. lems and the use of push notification allows the system to
As already said, the therapy provides two appointments per easily remind the patients to perform the exercises,
week during which the therapist can adjust the exercises ac-
cording to the progress or failure of the patient. The speech • the use of the ProEl Method as treatment for dysphonia and
• the availability of video tutorial helps patients to understand 10. Inc. Google. 2017. Dashboards.
the exercises and practise them correctly. https://developer.android.com/about/dashboards/index.html.
(May 2017).
We plan to test our system with a large number of patients
divided into two groups, one using our system and the other 11. PM Kato, SW Cole, AS Bradlyn, and BH Pollock. 2008.
following the therapy without any additional help or tool. The A video game improves behavioral outcomes in
test will aim at understanding if the system is able to increment adolescents and young adults with cancer: A randomized
the number of performed exercises and therefore the level of trial. Vol. 122. 305–317 pages.
adherence to the protocol therapy, and to lower the number of
12. P. Lavaissièri and P. E. Damasceno Melo. 2017. Prototype
dropouts. At the time of writing, we are currently testing the
app for voice therapy: a peer review. CODAS 29, 1
efficacy and the level of acceptability of our system. A group
(2017).
of 5 patients affected by dysphonia is using the system. The
group is composed by 3 females and 2 males, all the patients 13. X. Liang, Q. Wang, X. Yang, J. Cao, J. Chen, X. Mo, J.
are aged between 45 and 65 and they have been using the Huang, L. Wang, and D. Gu. 2011. Effect of mobile
system for 30 days. The initial results are encouraging since phone intervention for diabetes on glycaemic control: A
they do not report problems in using the application and they meta-analysis. Diabetic Medicine 28 (2011), 455–463.
are performing the exercises daily.
14. J. J. Lin, L. Mamykina, S. Lindtner, G. Delajoux, and
In the future, we plan to develop the application for all the H. B. Strub. 2006. Fish’N’Steps: Encouraging Physical
available operating systems using a framework for cross- Activity with an Interactive Computer Game. In Proc. of
platform mobile development, to allow also users with other the International Conference on Ubiquitous Computing.
operating systems to use the application. Springer-Verlag, Berlin, Heidelberg, 261–278.
REFERENCES
15. C. Portone, M. M. Johns, and E. R. Hapner. 2008. A
1. A. Bandura. 1997. Self-efficacy: The exercise of control. review of patient adherence to the recommendation for
W. H. Freeman and Company, New York, USA. voice therapy. J. of Voice: Official Journal of the Voice
Foundation 22, 2 (2008), 192–196.
2. A. Bandura. 2001. Social cognitive theory: An agentic
perspective. Annual Review of Psychology 52, 1 (2001), 16. L. Van den Audenaeren, V. Celis, V. Vanden Abeele, L.
1–26. Geurts, J. Husson, P. Ghesquière, J. Wouters, L. Loyez,
and A. Goeleven. 2013. DYSL-X: Design of a tablet game
3. A. Borragan, E. Lucchini, A. Ricci Maccarini, and G. for early risk detection of dyslexia in preschoolers.
De Rossi. 2009. The proprioceptive-elastic method Springer Fachmedien Wiesbaden, Wiesbaden, 257–266.
(PROEL) for vocal therapy. In Proc. of the Pan European
Voice Conference. 17. E. Van Leer and N. P. Connor. 2010. Patient perceptions
of voice therapy adherence. J. of voice: official journal of
4. M. Ciman and O. Gaggi. 2017. An empirical analysis of the Voice Foundation 24, 2 (2010), 447–469.
energy consumption of cross-platform frameworks for
mobile development. Pervasive and Mobile Computing 18. E. Van Leer and N. P. Connor. 2012. Use of portable
39 (2017), 214–230. digital media players increases patient motivation and
practice in voice therapy. J. of voice: official journal of
5. M. Ciman, O. Gaggi, L. Nota, L. Pinello, N. Riparelli, the Voice Foundation 26, 4 (2012), 447–453.
and T. M. Sgaramella. 2013. HelpMe!: a Serious Game
for Rehabilitation of Children affected by CVI. In Proc. 19. E. Van Leer and N. P. Connor. 2015. Predicting and
of WEBIST2013. Aachen, Germany, 257–262. Influencing Voice Therapy Adherence Using
Social-Cognitive Factors and Mobile Video. American J.
6. A. De Bortoli and O. Gaggi. 2011. PlayWithEyes: a new of Speech-Language Pathology 24, 2 (2015), 164–176.
way to test children eyes. In Proc. of the IEEE SeGAH.
190–193. 20. R. Whittaker, H. McRobbie, C. Bullen, A. Rodgers, and
Y. Gu. 2016. Mobile phone-based interventions for
7. O. Gaggi and M. Ciman. 2016. The use of games to help smoking cessation. Cochrane Database of Systematic
children eyes testing. Multimedia Tools and Applications Reviews (2016).
75, 6 (2016), 3453–3478.
21. G. Zichermann and C. Cunningham. 2011. Gamification
8. O. Gaggi, C. E. Palazzi, M. Ciman, G. Galiazzo, S.
by Design: Implementing Game Mechanics in Web and
Franceschini, M. Ruffino, S. Gori, and A. Facoetti. 2017.
Mobile Apps (1st ed.). O’Reilly Media, Inc.
Serious Games for Early Identification of Developmental
Dyslexia. Comput. Entertain. 15, 2, Article 4 (apr 2017),
24 pages.
9. L. Goasduff and A. A. Forni. 2017. Gartner Says
Worldwide Sales of Smartphones Grew 7 Percent in the
Fourth Quarter of 2016.
http://www.gartner.com/newsroom/id/3609817.
(February 2017).