=Paper=
{{Paper
|id=Vol-2068/symcollab7
|storemode=property
|title=Implicit Ambient Surface Information: From Personal to Interpersonal
|pdfUrl=https://ceur-ws.org/Vol-2068/symcollab7.pdf
|volume=Vol-2068
|authors=Katsumi Watanabe,Makio Kashino,Kimitaka Nakazawa,Shinsuke Shimojo
|dblpUrl=https://dblp.org/rec/conf/iui/WatanabeKNS18
}}
==Implicit Ambient Surface Information: From Personal to Interpersonal==
https://ceur-ws.org/Vol-2068/symcollab7.pdf
Implicit Ambient Surface Information:
From Personal to Interpersonal
Katsumi Watanabe Makio Kashino
Waseda University NTT Communication Science Laboratories,
Tokyo, Japan Kanagawa, Japan
kw@waseda.jp kashino.makio@lab.ntt.co.jp
Kimitaka Nakazawa Shinsuke Shimojo
University of Tokyo California Institute of Technology
Tokyo, Japan Pasadena, USA
nakazawa@idaten.c.u-tokyo.ac.jp sshimojo@caltedh.edu
ABSTRACT INTRODUCTION
We have proposed a novel concept: “Implicit Ambient Wisdom computing and the consequential harmonious
Surface Information” (IASI), which is based on the notion collaborations between humans and machines can
that information on the surface of an agent (e.g., bodies and contribute meaningfully to many potential application fields,
machines) is implicitly processed and affects collaborations such as learning and teaching [1], enhancing working
between agents. To utilize IASI, it is necessary to develop experience, and promoting sports and cultural activities.
technologies and analysis methods that can recode and During such processes, dynamic, mutual interactions are
decode implicit signals that appear on the surface of the embedded as implicit and embodied knowledge [2], which
body without disrupting the intended actions of users. We are hard to realize or understand from the first-person
sought to gain insight into IASI and to utilize it to establish perspective of humans (or machines). Cognitive science has
intelligent information processing systems by measuring investigated some interactions between more than two
implicit body movements, physiological responses, and persons (e.g., [3,4]). However, the workings of implicit
mental states and thereby accumulate scientific knowledge processes of embodied knowledge are largely unknown,
for theoretical advances. We have applied this concept to and technologies utilizing such embodied knowledge will
measuring physiological states and body movement of a be vital for development of wisdom computing and
single athlete, and here describe a few studies, and then harmonious human−machine collaborations.
propose future directions, with greater focus on the mental
and interpersonal aspects of IASI. Based on research projects on implicit information, we have
proposed a novel concept: “Implicit Ambient Surface
Author Keywords Information” (IASI). This is based on the notion that
Ambient; Argumentation; Implicit; Interpersonal; Sport information on the “surface of an agent (e.g., bodies and
ACM Classification Keywords machines)” is implicitly processed and influences
H.5.m. Information interfaces and presentation (e.g., HCI): interactions. In order to utilize IASI, it is important to
Miscellaneous. develop technologies that recode and decode implicit body
movements and physiological responses without disrupting
the intended actions of users. A recent advance in
performance material capable of measuring biometric
information will provide a good point of initiation.
© 2018. Copyright for the individual papers remains with the In the project (“Intelligent Information Processing Systems
authors. Copying permitted for private and academic purposes. based on IASI,” we aim to gain understanding of
SymCollab '18, March 11, Tokyo, Japan.
information that exists on the surfaces of the human body
and machines but are largely ignored. We intend to utilize
this information to establish intelligent information
processing systems for creative human−machine
collaboration.
In particular, we have tested technologies that recode and
decode implicit body movements and physiological
responses in the actual field, and have accumulated
scientific knowledge for theoretical advances. While we
have discovered multiple important and/or interesting
findings and there have been numerous outputs from our
research at this point, we introduce the results of a selected
set of such studies in the following sections.
IMPLICIT AMBIENT SURFACE INFORMATION IN
ACTUAL SPORTS FIELDS
In order to test the developed technologies and proposed
theories and to aim for higher quality activities of humans
in collaborations with machines, we first targeted practical
Figure 1. Model obtained by subtracting heart rates (HR) in
fields (e.g., sports). Among other fields, we focused on
practice sessions from those in the actual field in baseball play.
sports, because implicit, embodied knowledge has been Note that HRs are generally higher and more variable in the
claimed to be important in sports, but we think it has not actual field. The prediction model is used for computing the
been well examined. One potential use of such information HR difference online in the actual field. Physical activities
is to provide feedback of implicit processes to athletes were measured by the output of acceleration sensors.
and/or to coaches, to facilitate physical and mental
regulation, and metacognition of their bodies and minds.
The recent development of hydrophilic high body-
compatibility sensors has enabled us to measure body
activities and heart rate continuously and stably. We have
used these in several procedures to separate mental states
from body states and activities. In essence, this involves
measuring physical activities by means of acceleration
sensors, and heart rate by means of hydrophilic sensors, and
then developing a specific model to predict heart rate from
physical activities during practice sessions, which is then
subtracted from the actual heart rate. This concept is simple, Figure 2. Difference in HR in the actual field during baseball
but is has been quite difficult to test and obtain a sufficient play from single player. The blue line represents the predicted
amount of data in the real sports field. HR from the physical activities at a given time. The orange
regions indicate the time when the player was under mild
In order to resolve this lack of data and test fields, we pressure and the pink regions indicate high pressure. Note
formed and registered a baseball team specifically for that HR difference was high even when he was not on the
testing the devices and the system. This allowed us to ground and watching the game, which had a big chance at the
identify and select possible physiological signals in the end (one hit would lead to win the game after defeat seems
certain).
actual field and to accumulate knowledge, technology, and
know-how for physiological measurement on the actual
sports field (Figure 1). We successfully extracted mental
states, including tension, anxiety, etc. (Figure 2). However,
the exact mapping of the objective data to the subjective
mental state still needs to be achieved by obtaining more
data in the actual field.
Moreover, we have developed a system for visualizing the
balance between sympathetic and parasympathetic nervous
states, in order to measure the mental state during active
movements. We have also developed a system for Figure 3. Example of sonification. Lower (higher) frequencies
translating muscle activity into sounds (i.e., sonification of are assigned to lower (higher) parts of the body. Even when
surface electromyography) and providing intuitive feedback the ball speeds were the same (i.e., 100 km/h), the difference in
coordination between lower and higher parts of the body and
to a user to adjust pitching form [5] (Figure 3). As the
body movement (which were also translated into auditory
auditory feedback is given in real time (as compared with feedback) were more articulated in professional baseball
visual feedback, which is given offline), we expected that players. Note that the sounds are mapped more discrete in the
sonification and online feedback would improve learning of baseball player.
pitching forms. These technologies and their fine-tuning in IMPLICIT REALTIME MODULATION OF EMOTION BY
the actual sports fields could have various applications, OWN VOICE
including the analysis of whole-body synergy, the virtual Another implicit process that appears on the surface of our
reality system for implicit brain function analysis, and the bodies is emotion. Emotion is difficult to control, because
Sports Brain Project initiative at NTT Communication unconscious processes underlie both understanding and
Science Laboratories. expression of emotions. In order to examine the effect of
IASI on emotion, we have developed software that changes
To date, our research has involved studies of single athletes, the emotional tone of voices online without noticeable
and has mostly focused on physical activities while we delay (Da Amazing Voice Inflection Device: DAVID,
attempted to measure pressure, anxiety, tension, etc. Figure 4, [6]).
However, we think that in order to achieve human-
harmonized information systems, scientific knowledge and To test the effect of voice tone on the subjective evaluation
the technologies of measuring, decoding, and controlling of one’s own emotion, we let participants read a story alone
human behaviors, in particular, implicit “interpersonal” and aloud and let them listen to their own voice (altered
information, are vital. Our project is based on the concept into happy, sad, or afraid). We found that people did not
that and effective interpersonal communication depends detect the manipulation of their own voices, but that their
strongly on implicit, non-symbolic information that emotional state was changed toward the expected emotion
emerges from dynamic interactions among agents. The [6,7]. In the field of psychology, the relationship between
other studies in our project clearly provide other scientific the expression and subjective feeling of emotions has been
bases for future research and technological development. debated for a long time. The present study is the first to
These objectives include delayed compensation by using provide evidence of direct auditory feedback effects on
implicit visuomotor responses, a screening procedure for emotional experience.
autism spectrum disorders based on auditory and gaze
processing, and objective measurements of immersiveness
by using autonomic nervous and hormonal responses.
One example is the interpersonal interactions between an
athlete and a coach. There are many potential pathways
though which interpersonal information is conveyed: direct
conversations, explicit and implicit feedback from the
coach after observing the athlete’s movement, explicit and
implicit bodily feedback from the athlete’s own actions,
explicit and implicit feedback from observations of his/her
own performance, social encouragement and
discouragement, etc. Patterns appearing on the body
surfaces would convey much information about the state of
the athlete in the presence of the coach. In addition, it
would be clear that some atmospheres could lead to either
good or bad performances, often consequential and Figure 4. Implicit modulation of emotion can be achieved by
sometimes independent of the consequence of interactions using DAVID (Da Amazing Voice Inflection Device), a digital
among agents (for example, flow experience). This can be audio platform to modify the emotional tone of people’s voices
felt by those involved in the interaction; however, it is while they are talking, and to make them sound more happy,
difficult to describe what these actually are, and even more sad, or fearful. Participants remained unaware that their
difficult to implement processes that mediate such contexts. voices were being manipulated.
This is mainly because the agents do not notice these We believe that DAVID may be used for both basic and
aspects, because they are focused on their activities. applied research in many new fields. To date, emotional
Here, our technologies and the analysis method for implicit manipulation has not been done only on recorded but not on
surface information would help to examine these aspects. running speech (i.e., online). For example, DAVID could
That is, by examining the implicit surface information and be used for mood disorders by inducing a positive
their interactions among more than two athletes, coaches, emotional change or by redescribing traumatic events in a
and spectators, we might be able to detect, decode, and different tone of voice. It might also be possible to alter the
even utilize implicit surface ambient information. emotional atmosphere of conversations in online meetings
and sports games (e.g., in American football). DAVID can
also be used for advanced modulation of mental states and
can be combined with the wearable systems we proposed in
the previous section for effective self-coaching, by
modulating the emotion of the user.
Since the basic theoretical concept is based on the A more recent study has also shown that we could produce
James−Lange theory of emotion [8] (i.e., emotion initially a situation we termed “interpersonal flow,” where flow
comes from bodily responses and interpretation of such experience appears to be shared by two persons, by letting
responses by the brain), which holds that emotion arises two persons play a music game together, and confirmed this
from the interpretation of bodily signals, such surface by both subjective ratings and performance scores.
information could also be used interpersonally. For example, Furthermore, although preliminarily, we have measured two
changes in emotional and affective atmosphere might be brains simultaneously (i.e., hyper-scanning) by using EEG,
induced by implementing the voice filters between two or and found that auditory evoked potentials are reduced
more persons during casual conversations, business during the subjective experience of flow (Figure 5). This
meetings, and teaching in a classroom. Such changes in observation is yet another expression of IASI.
atmosphere are considered as manifestations of IASI,
because individuals are not aware that such information is
being presented, and the effect is only noted interpersonally. CONCLUSION
The present project aims at decoding and utilizing IASI, in
order to both gain scientific understanding and to expand
INDUCING AND DETECTING “FLOW” AND ITS NEURAL the appropriate use of this concept.
CORRELATES
To date, we have shown that IASI can be measured and
As in the implicit modulation of emotion, subjective
possibly used in individual athletes. While this is a
experience may be modulated and induced externally, and
significant advance in and expansion of the concept of such
therefore may be shared by multiple persons. People play
information and its range of application, we would like to
sports not only for achieving better performances, but also
advance and expand it even further. In particular, we
for experiencing special feelings. “Flow” is defined as a
propose to apply IASI analysis methods that we have
peculiar mental state and/or experience when a person plays
established to decode and control “interpersonal IASI”
sports, music, games, etc. with high-level performance. It is
often characterized as a highly coordinated sensory-motor Most collaborative (or collective) behaviors occur during
performance, extreme concentration, alteration of space and dynamic, reciprocal interactions [1]. This is particularly
time, euphoria, etc. It has been suggested to be related to true when such activities involve many pieces of implicit
activation of the reward system, improved performance, knowledge [2]. Recent advancements in neuroscience and
and better team playing [9]. However, it has been difficult cognitive science have examined the multifaceted and
to replicate and/or simulate a state of flow. Our team has dynamic processes in explicit and implicit interpersonal
successfully found an experimental setup that induces a interactions [3, 4, 10-16].
flow state by using computer games and has begun to
measure behavioral, physiological, and neural correlates of To detect and decode interpersonal IASI, the reading and
flow. By starting from the experimental setup, we found analyzing of information on the body surfaces would be
valuable and increase knowledge and technological
that auditory evoked potentials (i.e., brain responses for
task-irrelevant sounds) could be an index of flow state. This advances in implicit interpersonal information. We expect
simple index could be used to detect a flow state in the that scientific and technological advances in IASI will open
actual field, including during sports, playing music, or in a new field of harmonious collaboration between humans
conversations, etc. and machines and lead to wisdom computing.
ACKNOWLEDGMENTS
This work has been supported by a grant from Japan
Science and Technology Agency, CREST (JPMJCR14E4,
14529247, Intelligent Information Processing Systems
Creating Co-Experience Knowledge and Wisdom with
Human-Machine Harmonious Collaboration).
REFERENCES
1. Katsumi Watanabe. 2013. Teaching as a dynamic
phenomenon with interpersonal interactions. Mind
Brain Educ 7, 2: 91-100.
2. Michael Polanyi. 1966. The Tacit Dimension.
University of Chicago Press.
Figure 5. Measuring interpersonal flow by hyper-scanning
3. Ivana Konvalinka and Andreas Roepstorff. 2012. The
(EEG) electroencephalogram.
two-brain approach: how can mutually interacting
brains teach us something about social interaction? 16. Judee K. Burgoon, Beth A. Le Poire, and Robert
Front Hum Neurosci 6, 215. Rosenthal. 1995. Effects of preinteraction expectancies
4. Kyongsik Yun, Katsumi Watanabe, and Shinsuke and target communication on perceiver reciprocity and
Shimojo. 2011. Interpersonal body and neural compensation in dyadic interaction. J Exp Soc Psychol
synchronization as a marker of implicit social 31, 4: 287-321.
interaction. Sci Rep 2, 959.
5. Toshitaka Kimura, Takemi Mochida, Tetsuya Ijiri, and
Makio Kashino. 2016. Body-mind sonification to
improve player’s actions in sports. NTT Technical
Review 14, 1.
6. Jean-Julien Aucouturier, Petter Johansson, Lars Hall,
Rodrigo Segnini, Lolita Mercadié, and Katsumi
Watanabe. 2016. Covert digital manipulation of vocal
emotion alter speakers’ emotional state in a congruent
direction. Proc Natl Acad Sci U S A 113, 948-953.
7. Laura Rachman, Marco Liuni, Pablo Arias, Andreas
Lind, Petter Johansson, Lars Hall, Daniel Richardson,
Katsumi Watanabe, Stéphanie Dubal, and Jean-Julien
Aucouturier. 2017. DAVID: An open-source platform
for real-time transformation of infra-segmental
emotional cues in running speech. Behav Res Methods
1, 21.
8. James, W., & Lange, C. G. 1922. The Emotions.
Baltimore: Williams & Wilkins Co.
9. Mihaly Csikszentmihalyi. 2014. Flow and the
Foundations of Positive Psychology. Springer.
10. Emmanuelle Tognoli, Julien Lagarde, Gonzalo C. De
Guzman, and J. A. Scott Kelso. 2007. The phi complex
as a neuromarker of human social coordination. Proc
Natl Acad Sci U S A 104, 19: 8190-8195.
11. Natalie Sebanz, Harold Bekkering, and Günther
Knoblich. 2006. Joint action: bodies and minds moving
together. Trends Cogn Sci 10, 2: 70-76.
12. Lior Noya, Erez Dekel, and Uri Alon. 2011. The mirror
game as a paradigm for studying the dynamics of two
people improvising motion together. Proc Natl Acad
Sci U S A 108, 52: 20947-20952.
13. P. Read Montague, Gregory S. Berns, Jonathan D.
Cohen, Samuel M. McClure, Giuseppe Pagnoni,
Mukesh Dhamala, Michael C. Wiest, Igor Karpov,
Richard D. King, Nathan Apple, and Ronald E. Fisher.
2002. Hyperscanning: simultaneous fMRI during
linked social interactions. Neuroimage 16, 4: 1159-
1164.
14. Günther Knoblich and Natalie Sebanz. 2006. The
social nature of perception and action. Curr Dir
Psychol Sci 15, 3: 99-104.
15. Jean Decety and Jessica A. Sommerville. 2003. Shared
representations between self and other: a social
cognitive neuroscience view. Trends Cogn Sci 7, 12:
527-533.