=Paper=
{{Paper
|id=Vol-1794/afcai16-paper8
|storemode=property
|title=Emotions Detection on an Ambient Intelligent System Using Wearable Devices
|pdfUrl=https://ceur-ws.org/Vol-1794/afcai16-paper8.pdf
|volume=Vol-1794
|authors=Angelo Costa,Jaime Andrés Rincon Arango,Carlos Carrascosa,Vicente Julián,Paulo Novais
|dblpUrl=https://dblp.org/rec/conf/afcai/CostaACJN16
}}
==Emotions Detection on an Ambient Intelligent System Using Wearable Devices==
https://ceur-ws.org/Vol-1794/afcai16-paper8.pdf
Emotions Detection on an Ambient Intelligent
System Using Wearable Devices
Angelo Costa1 , Jaime A. Rincon2 , Carlos Carrascosa2 , Vicente Julian2 , and
Paulo Novais1
1
Centro ALGORITMI, Escola de Engenharia, Universidade do Minho, Guimarães
acosta@di.uminho.pt, pjon@di.uminho.pt
2
Universitat Politècnica de València. D. Sistemas Informáticos y Computación
jrincon@dsic.upv.es, carrasco@dsic.upv.es, vinglada@dsic.upv.es
Abstract. In this paper we present an Ambient Intelligent System, the
iGenda, and the integration of a wearable device. The aim is to detect
emotional states through the wearable device and ultimately represent
and manage the social emotion of a group of entities. The advantage
of this action is that its usability is in line with retirement homes and
similar places, where the community is extended and an harmonious
environment is imperative. The iGenda serves has the visual interface
and the information centre, receiving the information from the wearable
device and managing the community emotion by sending information to
the care-receivers, caregivers, or changing home parameters (like music
or lighting) to achieve an specific emotion (such as calm or excitement).
Thus the goal is to provide an affective system that directly interacts
with humans by discreetly improving their lifestyle.
Keywords: Aging, Emotion, Simulation, Wearables
1 Introduction
Ambient Assisted Living (AAL) is an booming area where a large number of
projects are being developed with the aim of providing assistance to elderly and
disabled people [1,2,3]. What we can observe is that most of them have as a goal
to be simple and provide the least interaction with the users as possible. But
a common issue that they possess is that they rely solely on automatisms and
static user profiles. While they are effective to a certain degree (simple tasks
and basic likes) they do not encompass punctual changes that are part of the
complex human states (like boredom), which forces changes to the profile [4].
These changes are usually associated to certain emotional states [4,5,6] that
affect the human procedures, and asking every time the user for its consent to
every action/decision defeats the purpose of being discrete and ubiquitous [6].
A solution would be the adoption of an adaptive system that is able to per-
ceive these emotional changes and adjust to them. Furthermore, with sufficient
information, emotional profiles could be created in order to mimic the user com-
mon response and pre-emptively use this information to respond to decisions.
They way that humans perceive the world influences their emotional state
and that has a repercussion on the physical level. While most can hide facial
expressions and body movements (like hiding a state of surprise) there are low-
level signals that the human body sends inadvertently like skin/muscle tension-
ing, pupil dilatation and micro-movements. These most of these signals have a
corresponding bio-electrical impulse, thus they can be captured by sensors.
We propose the usage of a wearable device in form of a wristband, named
Emotional Smart Wristband (ESW) that feed information to the iGenda in order
to schedule new tasks according the emotional status.
The paper is structure as follows: section 2 presents the ESW architecture, the
emotional model and the virtual actor concept; section 3 presents the conclusion
and future developments.
2 Emotional Smart Wristband
The ESW monitors the Galvanic Skin Response and the Photoplethysmogram,
processes that information locally and sends that information to other systems
like the iGenda [7,8,9]. The platform that supports the ESW ontology is able
to receive the information of multiple ESW’s and process them individually or
in group. The group emotion detection is very useful to determine the general
emotion status of an environment that is composed of multiple people, like an
nursing home. In the AAL concept the individual is the most important factor
in the system, but that is applicable only when there is only one individual in a
smart home; when we consider multiple individuals we find that the best way to
maintain an harmonious ambient is when most of the people tend to an overall
emotion (considering varying degrees).
For instance, it is well known that there is typically one or a few more lead-
ers in elderly communities [10]. One issue with these leaders (specially in older
adults) is that they establish the likability of certain activities and the rest of the
group follows (even if they are internally in disagreement). This tends to result
in a very biased set of activities that the leader enjoy and the rest tolerate. By
using the ESW the users can respond unbiased of what their feelings are towards
an activity. Tracking the community responses results in the overall likeness and
in the social aspect identify if the leader is a good one or not.
2.1 Architecture
The ESW platform is composed of agents that calculate the group emotion (or
Social Emotion) of the participants, through the Social Emotion Agent (SEtA)
and resorting to machine learning techniques and makes it available for con-
sumption [11,12]. The voltage outputted from the sensors is captured locally (in
the wristband) and pre-processed there, thus the information fed to the agents
is already a high-level one. Additionally, the multi-agent system has dedicated
agents that grab that information and verify against the adopted models to ex-
tract the current emotion, joining it to the rest of the agents community that
represent other users. The overarching result is the availability of the immediate
emotion of each user and the global emotion of all users and the emotional trends
and evolution. The trend of the group emotion is important as it is easier to find
the culprits of the emotional changes.
For instance, consider a bar, it contains a group of friends and has music
playing. Now consider that some of these friends have different musical tastes.
If a music is playing that is enjoyable to 90% of the group we are able to no-
tice a emotional progression towards "satisfied", while if it is the opposite (90%
dislikes) we can observe movement in the opposite direction. One of the possi-
ble aim of this analysis is obtaining a playlist that is enjoyable to most of the
group with a specific alignment of songs that cater to each individual like. This
specificity is used to avoid reaching a tension point where one or more person
has reached a saturation point and refuses any further change or abandons the
environment.
The task of the iGenda is to consume this information a use its scheduling
feature to change the events of each participant, based on his/her profile, to
achieve a specific emotion, thus guiding the group to a common emotional state.
The emotional states are measured and placed in the PAD [13,14] model, out-
putting a representation of the emotional state, easing the task of the iGenda
on choosing new events. We have adopted the PAD model due to the simplicity
and the datasets available. Furthermore, we are able to project the information
captured into a 3D space, which facilitates the way the information is perceived.
2.2 The PAD model
The model seen in Figure 1 is a granular visual representation of the PAD model
where the Valence replaces the Pleasure as in the psychological area is more
representative and has a larger range of values [15]. The model has 12 sub-
quadrants that are 30 degrees from each other. The emotion is represented by
#»
the vector E(Ag) = [Arousal, V alence] and the representation of emotions uses
polar coordinates, constituted by the angle and the magnitude of the vector.
We use a fuzzy logic process that transforms quantitative values to qualita-
tive values that are able to be placed in the model showed in Figure 1. Thereon,
a neural network is trained using the DEAPdataset 3 as to achieve a real-world
result from real humans. The correlation between the data captured and the
dataset is established in the neural network meaning that each processed elec-
trical input can be translated into an emotion.
2.3 Virtual Actors
A new development is the inclusion of virtual actors in the system that emulate
the real participants. This feature aims to enhance of the decision-making process
of the system, detecting in advance the possible emotional states and prepar-
ing changes to the participants surrounding environment to accommodate these
3
http://www.eecs.qmul.ac.uk/mmv/datasets/deap/
Fig. 1. Emotions circle [16].
changes and proactively shift them to other states if the expected state is not
desirable [17,18].
In this case specifically, when related to the elderly there is little margin
for experimentation, as they have a large number of conditionings and are very
fragile. The emotional profile helps to preview the possible reactions to changes
on their daily routines and their daily emotion when performing specific activ-
ities. The group emotion is very useful in environments like nursing homes and
residential elderly communities. In this type of environment is very common to
perform group activities and generate teams of elderly people. One common is-
sue is the user verbal response and the real feeling, e.g., a person may tell that
they are enjoying an activity but that be only a "kind" response due to peer
pressure or to social engagement with the caregivers [19,20]. If the caregiver is
able to receive unbiased information about how one feels about an activity it is
easier to schedule similar or different activities or group that person with other
people [21].
Another benefit from using virtual actors is the possibility of introducing
Immersive Virtual Environments (IVE). In this specific case, the IVE’s would
be used to project the environments that the users reside and simulate all of the
possible components, meaning that every sensor/actuator is mapped as an agent
and has agency, thus it is able to directly interact with the human agents. This
forwards our research by having an safe environment were there is the ability of
testing multiple outcomes of real interactions in fractions of seconds and project
the optimal actions to achieve a certain outcome.
3 Conclusions
The main goal of this project is to create cohesive environments in which the
users are happy and feel accomplished. This is an arduous task as people is com-
posed of individuals that most of the time are very different to the rest. Further-
more, there are several different applications to the ESW ecosystem apart from
the healthcare area, like: crowd control, visual interfaces management, stress and
fatigue monitoring, virtual environments societies, among others.
4 Acknowledgements
This work is partially supported by the MINECO/FEDER TIN2015-65515-C4-
1-R and the FPI grant AP2013-01276 awarded to Jaime-Andres Rincon. This
work is supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT -
Fundação para a Ciência e Tecnología within the projects UID/CEC/00319/2013
and Post-Doc scholarship SFRH/BPD/102696/2014 (A. Costa)
References
1. Bravo, J., Cook, D., Riva, G.: Ambient intelligence for health environments. Jour-
nal of Biomedical Informatics 64 (December 2016) 207–210
2. Ramos, C.: Ambient Intelligence – A State of the Art from Artificial Intelligence
Perspective. In Neves, J., Santos, M.F., Machado, J.M., eds.: Progress in Arti-
ficial Intelligence. Volume 4874 of Lecture Notes in Computer Science., Berlin,
Heidelberg, Springer Berlin Heidelberg (2007) 285–295
3. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: Technologies,
applications, and opportunities. Pervasive and Mobile Computing 5(4) (2009)
277–298
4. Gratch, J., Marsella, S.: Tears and fears. In: Proceedings of the fifth international
conference on Autonomous agents - AGENTS’01, Association for Computing Ma-
chinery (ACM) (2001)
5. Maaoui, C., Pruski, A.: Emotion recognition through physiological signals for
human-machine communication. In: Cutting Edge Robotics 2010. InTech (Septem-
ber 2010)
6. van Hoof, J., Kort, H., Rutten, P., Duijnstee, M.: Ageing-in-place with the use of
ambient intelligence technology: Perspectives of older users. International Journal
of Medical Informatics 80(5) (May 2011) 310–331
7. Rincon, J.A., Costa, A., Novais, P., Julián, V., Carrascosa, C.: A dynamic emo-
tional model for agent societies. In Demazeau, Y., Ito, T., Bajo, J., Escalona,
M.J., eds.: Advances in Practical Applications of Scalable Multi-agent Systems.
The PAAMS Collection: 14th International Conference, PAAMS 2016, Sevilla,
Spain, June 1-3, 2016, Proceedings. Volume 9662. Springer International Pub-
lishing (2016) 169–182
8. Costa, A., Novais, P., Simoes, R.: A caregiver support platform within the scope of
an ambient assisted living ecosystem. Sensors (Basel, Switzerland) 14(3) (January
2014) 5654–5676
9. Costa, A., Julián, V., Novais, P.: Advances and trends for the development of
ambient-assisted living platforms. Expert Systems (2016) n/a–n/a EXSY-Mar-15-
075.R1.
10. Estabrooks, P.A., Munroe, K.J., Fox, E.H., Gyurcsik, N.C., Hill, J.L., Lyon, R.,
Rosenkranz, S., Shannon, V.R.: Leadership in physical activity groups for older
adults: A qualitative analysis. Journal of Aging and Physical Activity 12(3) (jul
2004) 232–245
11. Rincon, J.A., Julián, V., Carrascosa, C.: Representing social emotions in MAS.
In Demazeau, Y., Decker, K.S., Pérez, J.B., de la Prieta, F., eds.: Advances in
Practical Applications of Agents, Multi-Agent Systems, and Sustainability: The
PAAMS Collection - 13th International Conference, PAAMS 2015, Salamanca,
Spain, June 3-4, 2015, Proceedings. Volume 9086 of Lecture Notes in Computer
Science., Springer (2015) 308–311
12. Rincon, J.A., Julián, V., Carrascosa, C.: An emotional-based hybrid application
for human-agent societies. In Herrero, Á., Sedano, J., Baruque, B., Quintián, H.,
Corchado, E., eds.: 10th International Conference on Soft Computing Models in
Industrial and Environmental Applications, SOCO 2015, Burgos, Spain, June 2015.
Volume 368 of Advances in Intelligent Systems and Computing., Springer (2015)
203–213
13. Mehrabian, A.: Analysis of affiliation-related traits in terms of the PAD tempera-
ment model. The Journal of Psychology 131(1) (January 1997) 101–117
14. Nanty, A., Gelin, R.: Fuzzy controlled PAD emotional state of a NAO robot.
In: 2013 Conference on Technologies and Applications of Artificial Intelligence,
Institute of Electrical and Electronics Engineers (IEEE) (dec 2013)
15. Gupta, R., Hur, Y.J., Lavie, N.: Distracted by pleasure: Effects of positive versus
negative valence on emotional capture under load. Emotion 16(3) (2016) 328–337
16. Thayer, R.E.: The biopsychology of mood and arousal. Oxford University Press
(1989)
17. Minhat, H.S., Mohd Amin, R.: Sociodemographic Determinants of Leisure Partic-
ipation Among Elderly in Malaysia. Journal of Community Health 37(4) (2011)
840–847
18. Hartgerink, J.M., Cramm, J.M., Bakker, T., van Eijsden, A.M., Mackenbach, J.P.,
Nieboer, A.P.: The importance of multidisciplinary teamwork and team climate
for relational coordination among teams delivering care to older patients. Journal
of Advanced Nursing 70(4) (2014) 791–799
19. Fernández-Mayoralas, G., Rojo-Pérez, F., Martínez-Martín, P., Prieto-Flores,
M.E., Rodríguez-Blázquez, C., Martín-García, S., Rojo-Abuín, J.M., Forjaz, M.J.:
Active ageing and quality of life: factors associated with participation in leisure
activities among institutionalized older adults, with and without dementia. Aging
& Mental Health 19(11) (2015) 1031–1041
20. Muñoz, D., Gutierrez, F., Ochoa, S.F., Baloian, N.: Enhancing social interaction
between older adults and their families. In: International Workshop on Ambient
Assisted Living, Springer (2013) 47–54
21. Chang, P.J., Wray, L., Lin, Y.: Social relationships, leisure activity, and health in
older adults. Health Psychol 33(46) (2014) 516–523