This paper describes our project which studies skill acquisition mechanism and develops skill transfer systems. To clarify skill acquisition mechanism, we analyze top athletes, elite music performers and handicapped persons who have advanced skills which are not found in ordinary people. Then we will develop skill transfer systems by using advanced computer vision, robotics and artificial intelligence.
INTRODUCTION Human augmentation is regarded as an important research field with a view to the future society in which computer technology, artificial intelligence technology, robot technology, etc. are highly integrated. There is a powered suit such as HAL1 by CYBERDYNE Inc. as a representative human augmentation technology. This is used to carry heavy objects that human cannot possess or to complement the lost body functions due to injuries and diseases. However, too much dependence on such complementary technologies may worsen human’s physical ability and cognitive ability. In other words, there is a need for a human augmentation that can strengthen human’s ability regardless of healthy persons or persons with disabilities. Of course, application to strengthening physical ability of powered suit is also conceivable, but many of the conventional orthotics are mechanical type or those using McKibben type artificial muscle. However, they are not suitable for teaching sensitive delicate movements.
⃝c 2018. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes.
SymCollab ’18, March 11, 2018, Tokyo, Japan
The conventional augmented reality system displays
superimposed virtual worlds in the real world. In acquiring skills,
it has been pointed out that not only such simple additional
information display but also an augmented reality system in
which people see themselves from the outside in a
thirdperson’s view [
The advanced skills of elite music performers, top athletes, disabled people are acquired by many years of training and experience. It is difficult to externalize such skills, and therefore it is difficult to transfer the skills to others. For example, in sports science, it is possible to measure body motion using the latest equipment such as special video equipment, small sensors, etc. However, the feedback to the athlete is limited to video playback after training and presentation of the measurement data. It is desirable that the coach intervenes in real time during training, and teach how to use the body, gaze direction, or psychological guidance etc. However, it cannot be done because (1) the essence of skill is not understood, (2) the difference with others is unclear, (3) it is difficult to present differences in real time using haptic tactile awareness during exercise.
In this project, we aim to develop the foundation of the technology of skill acquisition support system that acquires (i.e. copies) advanced skills from people and transfers (i.e. pastes) them to others using advanced image processing technology, augmented reality technology, robotics technology, and artificial intelligence technology. To that end, it is first necessary to clarify the principle of skill acquisition mechanism. Therefore, we analyze elite musicians, elite athletes and disables people those have special skills that are not found in ordinary people. We try to abstract the essence of skills by reducing the dimension of the multidimensional big data obtained by using statistical and machine learning methods. On the other hand, it was difficult to transfer skills until now because it is difficult to measure gaze, body movement, environment in real time, without disturbing the free and natural movement of humans. In order to solve this problem, we develop the following skill acquisition support system in this project. The first is a wearable gaze, body motion and environment recognition device using a small omnidirectional camera. The omnidirectional camera shoots an entire scene including human’s face, body and environment. By applying image stabilization, feature extraction, machine learning to this omnidirectional video, recognition of gaze, body motion and environment is achieved at the same time. The second is a device for presenting third person’s view using a compact HMD. Based on the body movement and the environment recognition, real-time feedback is provided based on the difference between model data and the body movement. The third is a real-time force feedback device using ultra-fine artificial muscles. Conventional force presentation devices were mechanical type like CYBERDYNE’s HAL or McKibben type artificial muscles, which hindered users’ natural and free movement. In contrast, we develop a lightweight, wearable haptic feedback device using ultra-fine artificial muscle. Teaching data is presented to the user in real time. Ultimately, we validate the effectiveness of each developed device by applying it to each field of music performance, sports, disabled persons and rehabilitation.
PROJECT DESCRIPTION In this research, we mainly focus on two research themes: (1) abstraction of skills and clarification of skill acquisition mechanism, and (2) development of skill acquisition support technology. Both are done in parallel, but scientific knowledge in (1) is fed back to (2). Also, the prototype of the acquisition support system developed in (2) will be provided in (1) and will be used for new experiments to clarify the skill acquisition mechanism.
A study on skill acquisition
Biomechanical study on top athletes
Data accumulation and analysis of top athlete skills will be
conducted. Motion measurement, gaze measurement, etc.
for specific sports are performed. Obtained data are
multidimensional time series data. We apply principal
component analysis (PCA) and machine learning to extract the
essence of skills. At the same time, we will develop a
gaze, body motion and environment recognition device
using a small omnidirectional camera. The image
stabilization method [
In order to master artificial limbs, we need to acquire another body control sensation different from that when not wearing. This process of acquiring shifts from “conscious control” to “unconscious control (automatic control)” along with proficiency. However, depending on physical condition and specifications or prosthesis, unconscious control goes back to conscious control. It is important to know how the prosthetic user gains and distinguishes multiple body senses. We analyze what kind of cause exists in handicapped persons through interviews. The findings obtained here not only include implications for support of people with disabilities but also help to clarify the promoting factors and inhibiting factors when the human body wears artificial objects such as artificial devices. Also, in the human augmentation technology, it is possible to think of situations where a person and a person actively interact through artifacts.
We will develop “visual reprogramming glasses” that can
integrate HMD with the omnidirectional gaze motion
measuring device developed in other groups and program the
visual characteristics of the subjects [
We will clarify elite musician’s skills and biological
information processing behind them [
Development of skill transfer system
A tactile feedback suit using ultra-fine artificial muscles
Tactile feedback is very important for actual skill
transfer. In this study, we develop a lightweight tactile feedback
suit incorporating ultra-fine artificial muscle [
In order to acquire body control ability, it is effective to provide means for seeing learner’s body from the third person’s view. In addition, we think that it is effective to provide means for providing the difference between the physical condition of the user and the model. For this purpose, we develop a system that projects its own body image in front of himself by augmented reality and emphasizes the difference with the model image as a model. Furthermore, a tactile feedback device is arranged on the body surface of the learner so that the difference portion can be experienced not only by sight but also by tactile sense. In order to present the difference in a more comprehensible manner, a method of compressing multidimensional information of the skeleton shape with a neural net (e.g. auto encoder, etc.) will be studied.
Excellent athletes and performers are expected to predict the event occurring at the next moment from the current exercise condition and the environmental recognition result. Also, in rehabilitation, if it becomes possible to predict accidents such as falling of a practitioner in advance, measures such as supporting the principal can be taken. For this reason, time series measurement information including human body shape displacement and barycentric position change etc. is learned and predicted using Recurrent Neural Network. In combination with a force feedback system, an injury prevention interface can be constructed. DISCUSSION The first scientific impact expected from the project is a recognition method of gaze, body and environment using omnidirectional video. This will open up a new field of computer vision including a new image stabilization method, gaze estimation method by deep learning and three-dimensional body shape restoration. The third person’s view system requires advanced image processing technology and real-time image synthesis technology, and creates a new augmented reality study. The force sense presentation system proposes a new way of using ultra-fine artificial muscle and it is expected to be applied to the field of robotics. Also, externalization of skills makes it possible to discuss skills scientifically. From the viewpoint of creating new industries, cyber training systems for players and athletes, support devices for persons with disabilities, and rehabilitation systems will be developed. In particular, a third person’s view system and a force feedback system using ultra-fine artificial muscles becomes a new technology in HCI.
As a social contribution, this project contributes not only to the transfer of skills, but also to the realization of a society where healthy people and disabled people coexist, and contribution to the elderly problem can be considered. At the same time as supporting the maintenance of the physical health of the elderly, by lowering the threshold of acquiring the musical instrument performance, it is possible to give the living worth of hobby and also to prevent dementia by fine finger movement. Human augmentation technology is ubiquitously invoked for everyday life at elderly care facilities and at home, realizing a human-machine coexistence life that prevents, maintains, and strengthens declines in physical ability and cognitive ability.
CONCLUSION In this paper, we described our project of a study on skill acquisition and skill transfer. By using augmented human technology which integrates computer vision, robotics and artificial intelligence, we aim to develop a framework and systems to support humans collaborating with embedded environmental intelligence.