Digital Twin for Shooting Sports Training

Abstract

Digital twins are innovative tools that are only recently finding difusion, mostly in industrial fields. Among the possible applications on humans, Digital Twins have been employed in sports training for performance enhancement, correction of gesture errors, or injury prevention. The objective of this research project is the evaluation and application of this tool to create a smart training system for shooting disciplines that still strongly rely on traditional training methods.

1. Introduction

Digital Twins are virtual models for systems monitoring, optimization, simulation, and management, capable of evolving alongside the physical system during its operative life. Such a physical system can be a machine, an entire industrial plant, or a human being. The term was introduced in 2002 in the Product Lifecycle Management field for physical systems monitoring through a virtual counterpart.

According to literature [ 1 ], the term Digital Twin is not bound to a unique definition even today but can overall be described as composed of three fundamental entities: the physical twin (the real system), the digital twin (the model of the system) and a continuous connection for bi-directional information exchange between the system and its virtual counterpart.

In its simpler implementations, a Digital Twin is a virtual model that describes a real system. Given the capabilities we aim to employ, in this research, a digital twin is more a living, intelligent, and continuously evolving model that follows the life cycle of a real system and allows monitoring, control, and optimization [ 1 ].

This project regards specifically the Human Digital Twin, in which the physical system is a human being, and the most common applications are monitoring and simulation of organs and physiological functions [ 1 ], and monitoring of physical activity for rehabilitation, wellness, and sport [ 2 ]. CEUR

2. Motivation and objectives

Among Human Digital Twin applications, we find Digital Twin Coaching [ 3 ], in which the subject is a human being and the objective is performance enhancement in sports disciplines. The Digital Twin can, in this case, be a substitute for the trainer or support for it.

The design and implementation of a Digital Twin are not currently bound by common standards, and a variety of applications and methods are possible. While dealing with a Human Digital Twin, ethical and social issues are also introduced, such as the necessity to handle the security and privacy of users’ personal information [ 1, 3 ], or technical issues like the dificulty in instrumenting and connecting online a human being compared to a static machine [ 1, 2 ].

In reported papers, authors developed semi-autonomous systems for user assistance during physical activity, covering a variety of disciplines, such as football [ 4 ], tennis [ 5 ], swimming [ 6 ], and weightlifting [ 7 ].

Several issues need to be covered to obtain a system fit for the specific application field, but also to cover the main issues highlighted in scientific literature.

I identified three main themes that are relevant to my research project: objectives, measurement instruments, and interaction design and user experience.

There is a variety of objectives a training system can pursue. In disciplines that rely on a well-executed athletic gesture, the athlete’s Digital Twin is employed for comparison between the subject gesture and an optimal gesture through graphical comparison [ 8 ] or purposely defined quantities [ 9 ]. The subject’s Digital Twin can also be employed for the computation of customized suggestions on the training routine, to maximize the performance improvement [ 4, 10 ], or to identify behaviours that can cause an injury [ 11 ]. During the development of my project, I will investigate user modeling and identify metrics for performance evaluation, with the objective of computing customised suggestions for the trainee.

Modeling of the user requires the collection of data during training sessions and the choice of adequate measurement instruments. Among the most measured quantities, there are kinematics of certain body segments through vision systems [ 8 ] and Inertial Measurement Units [ 6 ]. The most commonly measured quantities for stress evaluation [ 12 ], or the fatigue level [ 10 ], are physiological quantities like skin temperature and humidity, heart rate, and respiratory rate [ 12 ]. Other examples also report the measurement of dynamic quantities, as suggested in [ 2 ], and the collection of user data through surveys [ 4 ], which also comprehend information such as eating and sleeping routines.

Scientific literature highlights the need for appropriate user interface and interaction [ 1, 13 ] towards the system. A variety of mediums and modes are employed, starting from a PC GUI [ 4, 12 ], smartphone applications [ 9, 11 ], that also doubles as data input platform, up to virtual reality or mixed reality [ 8 ], where an accurate user representation and placement in the virtual environment are basic requirements for user experience [ 14, 15 ].

My study will focus on the design of an adequate user interface, employing end-user development techniques.

3. Project description

Within the three years of this Ph.D. research project1, I intend to study the concept of Human Digital Twin and its application to athletes’ training in those disciplines that involve firearms as a means for performance enhancement.

My current proposal for the overall system can be summarised as in Figure 1, in which we ifnd the three main components of the system: the physical world, in which we have a training environment where multiple users operate and a trainer monitors the activity from outside. In this architecture, the users are fitted with sensors and interact with the environment and other users. Each user is monitored by its own Digital Twin, which computes customized suggestions. The training system is the virtual counterpart of the subject. Its main components represent the tasks the system is asked to perform: feature computation on input data, storage, modeling of the users through machine learning algorithms, and correction of the model based on new input data. Finally, the user feedback system comprehends algorithms for user-suggestions computation and a graphical interface that allows interaction between the user and the system.

One of the primary aspects of the project is the necessity for an appropriate interaction design and user experience. The user interface must report information in a clear and efective way but also involve the user, push it towards interacting with the system, and 1Advisors: Prof. Barbara Rita Barricelli and Prof. Federico Cerutti build trust in the suggestions by highlighting the involvement of field experts during design, especially if manual information input is required. Regarding interaction mediums, we could hypothesize a multimodal system that also employs haptic or acoustic feedback. I will also consider virtual reality and augmented reality systems as long as they’re compatible with the training environments, with the activity, and with the user’s comfort. In this context, some of the aspects that require to be investigated concern the players’ avatars. User experience is directly impacted by the quality of players’ representation on multiple levels: in literature, it is reported that incoherent user tracking in the virtual environment can be a cause of discomfort, can lead to a break of immersion and motion sickness, and can also unconsciously influence the user’s movements [ 15 ]. To compensate for the lack of adaptability in traditional methods that rely on inverse kinematics of a few tracked points [ 16 ], custom avatar scaling methods are developed. Researchers also note that the avatar itself can influence the behaviour of the user [ 14 ], depending on the way it is perceived as a more or less external entity.

At the core of a Digital Twin, there is a model of the real system that emulates its behaviour in a specific context and given the necessity to follow its physical counterpart during its natural evolution, it is required to employ machine learning algorithms for the modeling process. Both the model choice among the most performing [ 17 ] and the choice of its parameters are bound to the specific case and will be covered during this study.

Regarding measurement instrumentation, it is necessary to collect information that allows the system to monitor the subject and its performance. For user monitoring, kinematics and physiological measurements appear to be the most common and can be performed with both wearable and on-field instruments or with smart devices. Finally, athletes and trainers themselves could be the source of manual input information. Performance evaluation will instead rely whenever possible on the obtained score, but for tactical and defensive shooting, it is necessary to define a customised evaluation metric.

An adequate data collection and management is mandatory. The system must continuously collect and store data from sensors to maintain the model up to date with the real world; this could also require some pre-elaboration, such as feature extraction, and fill-in of missing values, which are a concrete possibility in case of manual data input or unstable network connection, sensor fusion to improve the informative content or Principal Components Analysis for dimensionality reduction [ 17 ]. In the case of Human Digital Twin, we also need to observe the privacy and security of collected information and face the dificulty of establishing a continuous and robust communication network.

Finally, the information obtained from the classifier must be properly adapted and returned as feedback for the users of the system, each of which has diferent necessities and require a diferent format. Scientific literature highlighted how adequate information formatting is fundamental, while application examples show that if the object of monitoring is not a physical quantity, but a behaviour, it is necessary to define one or more easily readable quantities, to bridge the algorithm world with the physical one.

4. Conclusions and future research

In this paper, I presented my Ph.D. project about the application of the Human Digital Twin concept to the training for shooting disciplines.

Scientific literature reports some intrinsic open issues that will require evaluation. Among those are the necessity for a robust connection, the need to handle a large quantity of data, the implementation cost, and the lack of standardization and regulation [ 1 ]. Several studies list the fundamental characteristics that should be covered during the development of this kind of system. For example, autonomy and interactivity are common to almost all reported papers, but intelligibility and flexibility are only rarely covered [ 2 ]. Finally, certain characteristics are specific to Human Digital Twins and are bound to the necessity to collect and handle users’ personal data and to interact with them; for example, privacy and data security and user involvement to stimulate interaction and trust towards the system [ 1, 2, 13, 3 ].

During the development of the project, I will follow an approach similar to the one described in [ 18 ]. I am currently in the first of three years, dedicated to problem acknowledgment, exploring literature review, and performing user research to identify open issues and opportunities. During the second year, I will cover the suggestion and development phases, where the former regards study and iterative evaluation of implementation proposals while the latter is the implementation of a high-fidelity prototype. Finally, the third year will allow evaluation of the prototype and thesis writing.

References

[1] B. R. Barricelli , E. Casiraghi , D. Fogli , A survey on digital twin: Definitions, characteristics, applications, and design implications , IEEE Access 7 ( 2019 ). doi: 10 .1109/ACCESS. 2019 . 2953499 .

[2] R. G. Díaz , Q. Yu , Y. Ding , F. Laamarti , A. E. Saddik , Digital twin coaching for physical activities: A survey , Sensors (Switzerland) 20 ( 2020 ) 1 - 21 . doi: 10 .3390/s20205936.

[3] R. Ferdousi , F. Laamarti , M. A. Hossain , C. Yang , A. E. Saddik , Digital twins for well-being: an overview , Digital Twin 1 ( 2022 ) 7 . doi: 10 .12688/digitaltwin.17475.2.

[4] B. R. Barricelli , E. Casiraghi , J. Gliozzo , A. Petrini , S. Valtolina , Human digital twin for fitness management , IEEE Access 8 ( 2020 ) 26637 - 26664 . doi: 10 .1109/ACCESS. 2020 . 2971576 .

[5] B. Bačić , Towards the next generation of exergames: Flexible and personalised assessmentbased identification of tennis swings , 2022 . URL: www.espn.com/esports/.

[6] U. Jensen , F. Prade , B. M. Eskofier , Classification of kinematic swimming data with emphasis on resource consumption , in: 2013 IEEE International Conference on Body Sensor Networks , 2013 , pp. 1 - 5 . doi: 10 .1109/BSN. 2013 . 6575501 .

[7] A. Yasser , D. Tariq , R. Samy , M. A. Hassan , A. Atia , Smart coaching: Enhancing weightlifting and preventing injuries , International Journal of Advanced Computer Science and Applications 10 ( 2019 ) 686 - 691 . doi: 10 .14569/ijacsa. 2019 . 0100789 .

[8] F. Hülsmann , J. P. Göpfert , B. Hammer , S. Kopp , M. Botsch , Classification of motor errors to provide real-time feedback for sports coaching in virtual reality - a case study in squats and tai chi pushes , Computers and Graphics (Pergamon) 76 ( 2018 ) 47 - 59 . doi: 10 .1016/j.cag. 2018 . 08 .003.

[9] J. Wang , K. Qiu , H. Peng , J. Fu , J. Zhu, Ai coach: Deep human pose estimation and analysis for personalized athletic training assistance , in: Proceedings of the 27th ACM International Conference on Multimedia, Association for Computing Machinery (ACM) , 2019 , pp. 374 - 382 . doi: 10 .1145/3343031.3350910.

[10] J. Vales-Alonso , D. Chaves-Dieguez , P. Lopez-Matencio , J. J. Alcaraz , F. J. Parrado-Garcia , F. J. Gonzalez-Castano , Saeta: A smart coaching assistant for professional volleyball training , IEEE Transactions on Systems, Man, and Cybernetics: Systems 45 ( 2015 ) 1138 - 1150 . doi: 10 .1109/TSMC. 2015 . 2391258 .

[11] L. Rothkrantz , Personalized digital fitness coach , in: Proceedings of the 22nd International Conference on Computer Systems and Technologies, Association for Computing Machinery , 2021 , pp. 123 - 127 . doi: 10 .1145/3472410.3472412.

[12] C. Scheuermann , T. Binderberger , N. V. Frankenberg , A. Werner , Digital twin: A machine learning approach to predict individual stress levels in extreme environments , in: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, Association for Computing Machinery , 2020 , pp. 657 - 664 . doi: 10 .1145/3410530.3414316.

[13] M. W. Lauer-Schmaltz , P. Cash , J. P. Hansen , A. Maier , Designing human digital twins for behaviour-changing therapy and rehabilitation: A systematic review , Proceedings of the Design Society 2 ( 2022 ) 1303 - 1312 . doi: 10 .1017/pds. 2022 . 132 .

[14] D. Possler , N. Carnol , C. Klimmt , I. Weber-Hofmann , A. Raney , A Matter of Closeness: Player-Avatar Relationships as Degree of Including Avatars in the Self , 2022 , pp. 171 - 182 . doi: 10 .1007/978-3- 031 -20212-4_ 14 .

[15] L. Boban , L. Strauss , H. Decroix , B. Herbelin , R. Boulic , Unintentional synchronization with self-avatar for upper- and lower-body movements , Frontiers in Virtual Reality 4 ( 2023 ) 1073549 . doi: 10 .3389/frvir. 2023 . 1073549 .

[16] J. L. Ponton , V. Ceballos Inza , L. Acosta , A. Rios , E. Monclús , N. Pelechano , Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality , Virtual Reality ( 2023 ) 1 - 20 . doi: 10 .1007/s10055-023-00821-z.

[17] C. M. Bishop , Pattern Recognition and Machine Learning (Information Science and Statistics) , Springer-Verlag, Berlin, Heidelberg, 2006 .

[18] V. Vaishnavi , W. Kuechler , Design Science Research Methods and Patterns: Innovating Information and Communication Technology , 2007 . doi: 10 .1201/9781420059335.