We have proposed a novel concept: “Implicit Ambient Surface Information” (IASI), which is based on the notion that information on the surface of an agent (e.g., bodies and machines) is implicitly processed and affects collaborations between agents. To utilize IASI, it is necessary to develop technologies and analysis methods that can recode and decode implicit signals that appear on the surface of the body without disrupting the intended actions of users. We sought to gain insight into IASI and to utilize it to establish intelligent information processing systems by measuring implicit body movements, physiological responses, and mental states and thereby accumulate scientific knowledge for theoretical advances. We have applied this concept to measuring physiological states and body movement of a single athlete, and here describe a few studies, and then propose future directions, with greater focus on the mental and interpersonal aspects of IASI.
© 2018. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. SymCollab '18, March 11, Tokyo, Japan.
INTRODUCTION
Wisdom computing and the consequential harmonious
collaborations between humans and machines can
contribute meaningfully to many potential application fields,
such as learning and teaching [
Based on research projects on implicit information, we have proposed a novel concept: “Implicit Ambient Surface Information” (IASI). This is based on the notion that information on the “surface of an agent (e.g., bodies and machines)” is implicitly processed and influences interactions. In order to utilize IASI, it is important to develop technologies that recode and decode implicit body movements and physiological responses without disrupting the intended actions of users. A recent advance in performance material capable of measuring biometric information will provide a good point of initiation. In the project (“Intelligent Information Processing Systems based on IASI,” we aim to gain understanding of information that exists on the surfaces of the human body and machines but are largely ignored. We intend to utilize this information to establish intelligent information processing systems for creative human−machine collaboration.
In particular, we have tested technologies that recode and decode implicit body movements and physiological responses in the actual field, and have accumulated scientific knowledge for theoretical advances. While we have discovered multiple important and/or interesting findings and there have been numerous outputs from our research at this point, we introduce the results of a selected set of such studies in the following sections.
IMPLICIT AMBIENT SURFACE INFORMATION IN ACTUAL SPORTS FIELDS In order to test the developed technologies and proposed theories and to aim for higher quality activities of humans in collaborations with machines, we first targeted practical fields (e.g., sports). Among other fields, we focused on sports, because implicit, embodied knowledge has been claimed to be important in sports, but we think it has not been well examined. One potential use of such information is to provide feedback of implicit processes to athletes and/or to coaches, to facilitate physical and mental regulation, and metacognition of their bodies and minds. The recent development of hydrophilic high bodycompatibility sensors has enabled us to measure body activities and heart rate continuously and stably. We have used these in several procedures to separate mental states from body states and activities. In essence, this involves measuring physical activities by means of acceleration sensors, and heart rate by means of hydrophilic sensors, and then developing a specific model to predict heart rate from physical activities during practice sessions, which is then subtracted from the actual heart rate. This concept is simple, but is has been quite difficult to test and obtain a sufficient amount of data in the real sports field.
In order to resolve this lack of data and test fields, we formed and registered a baseball team specifically for testing the devices and the system. This allowed us to identify and select possible physiological signals in the actual field and to accumulate knowledge, technology, and know-how for physiological measurement on the actual sports field (Figure 1). We successfully extracted mental states, including tension, anxiety, etc. (Figure 2). However, the exact mapping of the objective data to the subjective mental state still needs to be achieved by obtaining more data in the actual field.
Moreover, we have developed a system for visualizing the balance between sympathetic and parasympathetic nervous states, in order to measure the mental state during active movements. We have also developed a system for translating muscle activity into sounds (i.e., sonification of surface electromyography) and providing intuitive feedback to a user to adjust pitching form [5] (Figure 3). As the auditory feedback is given in real time (as compared with visual feedback, which is given offline), we expected that sonification and online feedback would improve learning of pitching forms. These technologies and their fine-tuning in the actual sports fields could have various applications, including the analysis of whole-body synergy, the virtual reality system for implicit brain function analysis, and the Sports Brain Project initiative at NTT Communication Science Laboratories.
To date, our research has involved studies of single athletes,
and has mostly focused on physical activities while we
attempted to measure pressure, anxiety, tension, etc.
However, we think that in order to achieve
humanharmonized information systems, scientific knowledge and
the technologies of measuring, decoding, and controlling
human behaviors, in particular, implicit “interpersonal”
information, are vital. Our project is based on the concept
that and effective interpersonal communication depends
strongly on implicit, non-symbolic information that
emerges from dynamic interactions among agents. The
other studies in our project clearly provide other scientific
bases for future research and technological development.
These objectives include delayed compensation by using
implicit visuomotor responses, a screening procedure for
autism spectrum disorders based on auditory and gaze
processing, and objective measurements of immersiveness
by using autonomic nervous and hormonal responses.
One example is the interpersonal interactions between an
athlete and a coach. There are many potential pathways
though which interpersonal information is conveyed: direct
conversations, explicit and implicit feedback from the
coach after observing the athlete’s movement, explicit and
implicit bodily feedback from the athlete’s own actions,
explicit and implicit feedback from observations of his/her
own performance, social encouragement and
discouragement, etc. Patterns appearing on the body
surfaces would convey much information about the state of
the athlete in the presence of the coach. In addition, it
would be clear that some atmospheres could lead to either
good or bad performances, often consequential and
sometimes independent of the consequence of interactions
among agents (for example, flow experience). This can be
felt by those involved in the interaction; however, it is
difficult to describe what these actually are, and even more
difficult to implement processes that mediate such contexts.
This is mainly because the agents do not notice these
aspects, because they are focused on their activities.
Here, our technologies and the analysis method for implicit
surface information would help to examine these aspects.
That is, by examining the implicit surface information and
their interactions among more than two athletes, coaches,
and spectators, we might be able to detect, decode, and
even utilize implicit surface ambient information.
IMPLICIT REALTIME MODULATION OF EMOTION BY
OWN VOICE
Another implicit process that appears on the surface of our
bodies is emotion. Emotion is difficult to control, because
unconscious processes underlie both understanding and
expression of emotions. In order to examine the effect of
IASI on emotion, we have developed software that changes
the emotional tone of voices online without noticeable
delay (Da Amazing Voice Inflection Device: DAVID,
Figure 4, [
To test the effect of voice tone on the subjective evaluation
of one’s own emotion, we let participants read a story alone
and aloud and let them listen to their own voice (altered
into happy, sad, or afraid). We found that people did not
detect the manipulation of their own voices, but that their
emotional state was changed toward the expected emotion
[
We believe that DAVID may be used for both basic and applied research in many new fields. To date, emotional manipulation has not been done only on recorded but not on running speech (i.e., online). For example, DAVID could be used for mood disorders by inducing a positive emotional change or by redescribing traumatic events in a different tone of voice. It might also be possible to alter the emotional atmosphere of conversations in online meetings and sports games (e.g., in American football). DAVID can also be used for advanced modulation of mental states and can be combined with the wearable systems we proposed in the previous section for effective self-coaching, by modulating the emotion of the user.
Since the basic theoretical concept is based on the
James−Lange theory of emotion [
A more recent study has also shown that we could produce a situation we termed “interpersonal flow,” where flow experience appears to be shared by two persons, by letting two persons play a music game together, and confirmed this by both subjective ratings and performance scores. Furthermore, although preliminarily, we have measured two brains simultaneously (i.e., hyper-scanning) by using EEG, and found that auditory evoked potentials are reduced during the subjective experience of flow (Figure 5). This observation is yet another expression of IASI.
CONCLUSION The present project aims at decoding and utilizing IASI, in order to both gain scientific understanding and to expand the appropriate use of this concept.
To date, we have shown that IASI can be measured and
possibly used in individual athletes. While this is a
significant advance in and expansion of the concept of such
information and its range of application, we would like to
advance and expand it even further. In particular, we
propose to apply IASI analysis methods that we have
established to decode and control “interpersonal IASI”
Most collaborative (or collective) behaviors occur during
dynamic, reciprocal interactions [
To detect and decode interpersonal IASI, the reading and analyzing of information on the body surfaces would be valuable and increase knowledge and technological advances in implicit interpersonal information. We expect that scientific and technological advances in IASI will open a new field of harmonious collaboration between humans and machines and lead to wisdom computing.
ACKNOWLEDGMENTS This work has been supported by a grant from Japan Science and Technology Agency, CREST (JPMJCR14E4, 14529247, Intelligent Information Processing Systems Creating Co-Experience Knowledge and Wisdom with Human-Machine Harmonious Collaboration). brains teach us something about social interaction? Front Hum Neurosci 6, 215.