=Paper=
{{Paper
|id=Vol-2068/symcollab5
|storemode=property
|title=A Study on Skill Acquisition Mechanism and Development of Skill Transfer Systems
|pdfUrl=https://ceur-ws.org/Vol-2068/symcollab5.pdf
|volume=Vol-2068
|authors=Hideki Koike,Jun Rekimoto,Junichi Ushiba,Shinichi Furuya,Asa Ito
|dblpUrl=https://dblp.org/rec/conf/iui/KoikeRUFI18
}}
==A Study on Skill Acquisition Mechanism and Development of Skill Transfer Systems==
https://ceur-ws.org/Vol-2068/symcollab5.pdf
A Study on Skill Acquisition Mechanism and Development
of Skill Transfer Systems
Hideki Koike Jun Rekimoto Junichi Ushiba
Tokyo Institute of Technology The University of Tokyo Keio University
koike@acm.org rekimoto@acm.org ushiba@brain.bio.keio.ac.jp
Shinichi Furuya Asa Ito
Sony Computer Science Tokyo Institute of Technology
Laboratory ito.a.ah@m.titech.ac.jp
furuya@csl.sony.co.jp
ABSTRACT The conventional augmented reality system displays super-
This paper describes our project which studies skill acqui- imposed virtual worlds in the real world. In acquiring skills,
sition mechanism and develops skill transfer systems. To it has been pointed out that not only such simple additional
clarify skill acquisition mechanism, we analyze top athletes, information display but also an augmented reality system in
elite music performers and handicapped persons who have which people see themselves from the outside in a third-
advanced skills which are not found in ordinary people. Then person’s view [4] is effective. Also, there are many cases that
we will develop skill transfer systems by using advanced the motion is so fast that visual feedback is not appropriate.
computer vision, robotics and artificial intelligence. In this case, it is necessary to integrate other sensory channels
such as auditory sense, tactile sense, and haptic sensation and
Author Keywords to provide teaching data in multi-channel.
augmented human; augmented reality; cyber training
The advanced skills of elite music performers, top athletes,
system; skill acquisition; skill transfer;
disabled people are acquired by many years of training and
experience. It is difficult to externalize such skills, and there-
ACM Classification Keywords
fore it is difficult to transfer the skills to others. For example,
H.5.m. Information Interfaces and Presentation (e.g. HCI):
in sports science, it is possible to measure body motion using
Miscellaneous
the latest equipment such as special video equipment, small
sensors, etc. However, the feedback to the athlete is limited
INTRODUCTION
to video playback after training and presentation of the mea-
Human augmentation is regarded as an important research
surement data. It is desirable that the coach intervenes in real
field with a view to the future society in which computer
time during training, and teach how to use the body, gaze di-
technology, artificial intelligence technology, robot technol-
rection, or psychological guidance etc. However, it cannot be
ogy, etc. are highly integrated. There is a powered suit such
done because (1) the essence of skill is not understood, (2) the
as HAL1 by CYBERDYNE Inc. as a representative human
difference with others is unclear, (3) it is difficult to present
augmentation technology. This is used to carry heavy objects
differences in real time using haptic tactile awareness during
that human cannot possess or to complement the lost body
exercise.
functions due to injuries and diseases. However, too much de-
pendence on such complementary technologies may worsen In this project, we aim to develop the foundation of the tech-
human’s physical ability and cognitive ability. In other words, nology of skill acquisition support system that acquires (i.e.
there is a need for a human augmentation that can strengthen copies) advanced skills from people and transfers (i.e. pastes)
human’s ability regardless of healthy persons or persons with them to others using advanced image processing technology,
disabilities. Of course, application to strengthening physi- augmented reality technology, robotics technology, and arti-
cal ability of powered suit is also conceivable, but many of ficial intelligence technology. To that end, it is first necessary
the conventional orthotics are mechanical type or those us- to clarify the principle of skill acquisition mechanism. There-
ing McKibben type artificial muscle. However, they are not fore, we analyze elite musicians, elite athletes and disables
suitable for teaching sensitive delicate movements. people those have special skills that are not found in ordinary
1 people. We try to abstract the essence of skills by reducing the
https://www.cyberdyne.jp/english/products/HAL/ dimension of the multidimensional big data obtained by using
statistical and machine learning methods. On the other hand,
it was difficult to transfer skills until now because it is dif-
ficult to measure gaze, body movement, environment in real
time, without disturbing the free and natural movement of hu-
mans. In order to solve this problem, we develop the follow-
ing skill acquisition support system in this project. The first
is a wearable gaze, body motion and environment recogni-
⃝2018.
c Copyright for the individual papers remains with the authors. Copying per-
mitted for private and academic purposes.
SymCollab ’18, March 11, 2018, Tokyo, Japan
Figure 1. An overview of the project.
tion device using a small omnidirectional camera. The omni- Data accumulation and analysis of top athlete skills will be
directional camera shoots an entire scene including human’s conducted. Motion measurement, gaze measurement, etc.
face, body and environment. By applying image stabilization, for specific sports are performed. Obtained data are multi-
feature extraction, machine learning to this omnidirectional dimensional time series data. We apply principal compo-
video, recognition of gaze, body motion and environment is nent analysis (PCA) and machine learning to extract the
achieved at the same time. The second is a device for present- essence of skills. At the same time, we will develop a
ing third person’s view using a compact HMD. Based on the gaze, body motion and environment recognition device us-
body movement and the environment recognition, real-time ing a small omnidirectional camera. The image stabiliza-
feedback is provided based on the difference between model tion method [6] is applied to the obtained omnidirectional
data and the body movement. The third is a real-time force video. Then, we will perform gaze estimation and mo-
feedback device using ultra-fine artificial muscles. Conven- tion recognition by using deep learning algorithms such as
tional force presentation devices were mechanical type like OpenPose library [1], and also perform environment recog-
CYBERDYNE’s HAL or McKibben type artificial muscles, nition by SLAM [2].
which hindered users’ natural and free movement. In con-
trast, we develop a lightweight, wearable haptic feedback de- • Cognitive study on handicapped person
vice using ultra-fine artificial muscle. Teaching data is pre- In order to master artificial limbs, we need to acquire
sented to the user in real time. Ultimately, we validate the another body control sensation different from that when
effectiveness of each developed device by applying it to each not wearing. This process of acquiring shifts from “con-
field of music performance, sports, disabled persons and re- scious control” to “unconscious control (automatic con-
habilitation. trol)” along with proficiency. However, depending on
physical condition and specifications or prosthesis, uncon-
PROJECT DESCRIPTION scious control goes back to conscious control. It is im-
In this research, we mainly focus on two research themes: portant to know how the prosthetic user gains and distin-
(1) abstraction of skills and clarification of skill acquisition guishes multiple body senses. We analyze what kind of
mechanism, and (2) development of skill acquisition support cause exists in handicapped persons through interviews.
technology. Both are done in parallel, but scientific knowl- The findings obtained here not only include implications
edge in (1) is fed back to (2). Also, the prototype of the ac- for support of people with disabilities but also help to clar-
quisition support system developed in (2) will be provided in ify the promoting factors and inhibiting factors when the
(1) and will be used for new experiments to clarify the skill human body wears artificial objects such as artificial de-
acquisition mechanism. vices. Also, in the human augmentation technology, it is
possible to think of situations where a person and a person
actively interact through artifacts.
A study on skill acquisition
• Biomechanical study on top athletes • Study on rehabilitation
• A system providing third person’s view
In order to acquire body control ability, it is effective to
provide means for seeing learner’s body from the third per-
son’s view. In addition, we think that it is effective to pro-
vide means for providing the difference between the phys-
ical condition of the user and the model. For this purpose,
we develop a system that projects its own body image in
front of himself by augmented reality and emphasizes the
difference with the model image as a model. Furthermore,
a tactile feedback device is arranged on the body surface
of the learner so that the difference portion can be experi-
enced not only by sight but also by tactile sense. In order
to present the difference in a more comprehensible manner,
Figure 2. Cyber training system. a method of compressing multidimensional information of
the skeleton shape with a neural net (e.g. auto encoder,
etc.) will be studied.
We will develop “visual reprogramming glasses” that can
integrate HMD with the omnidirectional gaze motion mea- • Future prediction
suring device developed in other groups and program the
Excellent athletes and performers are expected to predict
visual characteristics of the subjects [8, 5]. In addition,
the event occurring at the next moment from the current
we develop a “haptic reprogramming device” which can
exercise condition and the environmental recognition re-
change the motion resistance by incorporating a magnetic
sult. Also, in rehabilitation, if it becomes possible to pre-
fluid active joint whose viscosity can be controlled to the
dict accidents such as falling of a practitioner in advance,
brace attached to the wrist joint or the elbow joint. We will
measures such as supporting the principal can be taken. For
use “target/reaching experiment” to adjust the fingertip to
this reason, time series measurement information including
the target position. As described above, by mathematically
human body shape displacement and barycentric position
modeling the adaptation process of the brain that occurs
change etc. is learned and predicted using Recurrent Neu-
when modulating the physical environment information at
ral Network. In combination with a force feedback system,
a strength that is not conscious using a wearable visual pre-
an injury prevention interface can be constructed.
sentation device or a force sense presentation device. We
formulate a general extension law that is not dependent on
generality. DISCUSSION
The first scientific impact expected from the project is a
• Study on elite music performers recognition method of gaze, body and environment using om-
nidirectional video. This will open up a new field of computer
We will clarify elite musician’s skills and biological infor-
vision including a new image stabilization method, gaze esti-
mation processing behind them [3, 7]. Firstly, we will tar-
mation method by deep learning and three-dimensional body
get professional pianists in college of music and perform
shape restoration. The third person’s view system requires
’archive of transcendence skill’ to measure the movement
advanced image processing technology and real-time image
and muscle activity of various bodily skills. Therefore, we
synthesis technology, and creates a new augmented reality
develop a sensing system detachable to the instrument and
study. The force sense presentation system proposes a new
high precision data glove. Secondly, using the multivari-
way of using ultra-fine artificial muscle and it is expected to
ate analysis such as NMF and LASSO regression and deep
be applied to the field of robotics. Also, externalization of
learning, we extract individual skills based on the obtained
skills makes it possible to discuss skills scientifically.
big data.
From the viewpoint of creating new industries, cyber train-
Development of skill transfer system ing systems for players and athletes, support devices for per-
• A tactile feedback suit using ultra-fine artificial muscles sons with disabilities, and rehabilitation systems will be de-
veloped. In particular, a third person’s view system and a
Tactile feedback is very important for actual skill trans- force feedback system using ultra-fine artificial muscles be-
fer. In this study, we develop a lightweight tactile feedback comes a new technology in HCI.
suit incorporating ultra-fine artificial muscle [9]. This is an
ultra-fine artificial muscle with a diameter of about 2 to 4 As a social contribution, this project contributes not only to
mm, it is possible to make a suit that does not disturb free the transfer of skills, but also to the realization of a soci-
movement by sewing into clothes. We have already con- ety where healthy people and disabled people coexist, and
ducted preliminary experiments using this artificial muscle contribution to the elderly problem can be considered. At
and confirmed that sufficient torque can be obtained with the same time as supporting the maintenance of the physical
a small number of artificial muscles by devising a knitting health of the elderly, by lowering the threshold of acquiring
pattern. We will first develop a device that can be worn on the musical instrument performance, it is possible to give the
the arm, and then develop upper body or whole body suit. living worth of hobby and also to prevent dementia by fine
In Proc. on Computer Vision and Pattern Recognition
(2017).
2. Durrant-White, H., and Bailey, T. Simultaneous
localization and mapping. IEEE Robotics and Automation
magazine 13, 2 (2006), 99–108.
3. Furuya, S., Klaus, M., Nitsche, M., Paulus, W., and
Altenmulle, E. Ceiling effects prevent further
improvement of transcranial stimulation in skilled
musicians. The Journal of Neuroscience 34, 41 (2014),
13834 13839.
4. Higuchi, K., Ishiguro, Y., and Rekimoto, J. Flying eyes:
Figure 3. Motion capture with a small omnidirectional camera.
Free-space content creation using autonomous aerial
vehicles. In CHI ’11 Extended Abstracts on Human
Factors in Computing Systems, CHI EA ’11, ACM
(2011), 561–570.
5. M.Kawakami, T.Fujiwara, J.Ushiba, A.Nishimoto,
K.Abe, K.Honaga, A.Nishimura, K.Mizuno, M.Kodama,
Y.Masakado, and M.Liu. A new therapeutic application of
brain-machine interface (bmi) training followed by hybrid
assistive neuromuscular dynamic stimulation (hands)
therapy for patients with severe hemiparetic stroke: A
proof of concept study. Restrative Neurology and
Neuroscience 34, 5 (2016), 789–797.
Figure 4. Third person’s view. 6. Nakazawa, M., and Koike, H. Synthesizing fixed point of
views from a spinning omnidirectional ball camera. In
finger movement. Human augmentation technology is ubiq- Proceedings of the 8th Augmented Human International
uitously invoked for everyday life at elderly care facilities and Conference, AH ’17, ACM (New York, NY, USA, 2017),
at home, realizing a human-machine coexistence life that pre- 32:1–32:5.
vents, maintains, and strengthens declines in physical ability 7. Oku, T., and Furuya, S. Skillful force control in expert
and cognitive ability. pianists. Experimental Brain Research 235, 5 (2017),
1603–1615.
CONCLUSION
In this paper, we described our project of a study on skill ac- 8. S.Kasuga, S.Telgen, J.Ushiba, D.Nozaki, and
quisition and skill transfer. By using augmented human tech- J.Diedrichsen. Learning feedback and feedforward
nology which integrates computer vision, robotics and artifi- control in a mirror-reversed visual environment. Journal
cial intelligence, we aim to develop a framework and systems of Nurophysiology 114, 4 (2015), 2187–2193.
to support humans collaborating with embedded environmen- 9. Takahashi, N., and Koike, H. Shin-tai: Design for
tal intelligence. controlling personalities of a humanoid robots body using
artificial muscles and fats. In Proc. on IEEE Int l Symp.
REFERENCES
on Robot and Human interactive communication
1. Cao, Z., Simon, T., Wei, S., and Sheikh, Y. Realtime (RO-MAN 2017) (2017), 1400–1405.
multi-person 2d pose estimation using part affinity fields.