Real-time Hand Tracking Using Leap Motion Controller Roumaissa Bekiri 0 Mohamed Chaouki Babahenini 0 LESIA Laboratory, Computer Science Department, University of Biskra , Algeria 64 69

Hand tracking has become a crucial topic in recent years due to the advancement in various applications of human-computer interaction (HCI). The primary objective of this research is to investigate the capabilities and limitations of the Leap Motion Controller in real-time hand tracking scenarios. We begin by providing an overview of the Leap Motion technology, its hardware architecture, and the software stack used for hand tracking. We delve into the underlying algorithms and methods employed by the controller to analyze and interpret hand and finger movements. Then, we propose several scenes to interact with it real-hand through leap motion camera. Our experimental results demonstrate that our approach, relying on tracking data from the Leap Motion Controller, accurately identifies hand gestures in the absence of any occlusion.

 eol>Leap Motion hand tracking gesture real-time interaction 1. Introduction

tems. The Leap Motion Controller (LMC) translates human intentions into robotic interactions, excelling Within the realm of gesture recognition systems, tracking at discerning human gestures through hand and finger stands out as a pivotal component. Over the course of the movements[ 16, 17, 18 ]. LMC supports diverse virtual past year, there has been a noteworthy surge in the devel- tasks, such as engaging in virtual games, making selecopment of various techniques and methodologies aimed tions, exercising virtual control, and running applicaat enhancing the overall performance of gesture recogni- tions in virtual environments[ 19, 20 ]. Its precision in tion systems. These eforts have predominantly focused tracking intricate hand and finger movements within on four key aspects: segmentation, tracking, modeling, three-dimensional space enhances user engagement and and recognition’. Tracking is an essential element in the immersion within augmented reality (AR) environments. sequence as it involves monitoring and following the By enabling lifelike representations of user hands in AR, movements of the segmented gestures over time. Achiev- Leap Motion technology facilitates intuitive interaction ing robust and accurate tracking is vital to ensure the with virtual elements. In this paper, we focus on real-time continuity and coherence of gesture recognition, espe- hand tracking using a Leap Motion Controller. We used cially in dynamic environments. the Leap Motion Unity Module, a software development

The progress in the development of cutting-edge ges- kit (SDK) provided by Leap Motion, to integrate hand ture recognition sensors has found common use in the tracking and gesture recognition into Unity-based apcontrol of robotic hands. Robotics has increasingly be- plications. We created multiple virtual scenes to enable come a substitute for human labor across a broad spec- interactive and immersive real-time experiences. The trum of fields[ 1 ]. One noteworthy example is the Leap paper’s outline is as follows: Section 2 reviews recent Motion Controller, which generates a virtual screen for research on hand tracking using Leap Motion Controller visualizing hand gestures and subsequently transmits and its intersection with computer vision, robotic conthis information to a robotic hand through wireless de- trol, and EEG-based BCI systems. Section 3 describes our vice support[ 2, 3 ]. Beyond gesture recognition, such proposed methodology for hand tracking across diferent technology also finds applications in computer vision scenarios. Section 4 outlines the experimental setup and tasks[ 4 ], robotic control[ 5, 6, 7, 8 ], and EEG-based BCI presents results. Section 5 concludes the paper. systems[ 9, 10, 11 ]. These systems leverage data from MEMS accelerometers, EMG sensors, gyroscopes, flex sensors, and pressure sensors[ 12, 13 ], and other sen- 2. Related Work sorial devices [ 14, 15 ] enabling the accurate detection and interpretation of human hand movements to facilitate seamless interaction between users and robotic sysand evaluating system performance through user testing. Satheeshkumar et al.[ 22 ] introduced a touchless communication system using Leap Motion, combining virtual environments with real-time settings by interfacing the device with a robotic arm through a microcontroller. Similarly, Sharma et al.[ 23 ] demonstrated Leap Motion’s ability to track hand movements using infrared LEDs and cameras, ofering valuable support for tasks like 3D modeling and animation. In another study, Marin et al.[24] focused on static gesture recognition using Leap Motion and Kinect devices. They designed custom features based on fingertip positions and orientations and utilized a multi-class SVM classifier for gesture classification. Depth features from Kinect were combined with Leap Motion data to improve recognition accuracy. However, their work excluded dynamic gestures. Cooper et al.[25] employed a color-coded glove for hand tracking, which, while efective, reduced user experience due to the need for wearing gloves. In addition to Leap Motion applications, recent advancements in computer vision, such as convolutional neural networks (CNNs)[26, 27, 28], have significantly contributed to improving gesture recognition systems. These techniques are widely adopted in robotic control[29, 30], enabling precise manipulation tasks. Furthermore, EEG-based BCIs[31, 32, 33] have opened new avenues for integrating brain activity signals with gesture control systems, enhancing the interaction between humans and machines. In this paper, we focus on using the Leap Motion Controller for real-time hand tracking and position monitoring, utilizing its SDK to explore applications in dynamic scenarios. 3. System Overview • B. Virtual Hand: Subsequently, on the laptop or PC, a 3D image, representing a virtual hand, is generated. This virtual hand is accompanied by relevant information and data associated with its movements and positions. This visual and informational representation of the virtual hand is then wirelessly transmitted to another component or device for further use. • C. Interaction with the virtual hand: In Unity 3D, we create three scenes with virtual objects, the hand can grasp and move in diferent positions.

3.1. Hardware

The Leap Motion controller, when used in conjunction with the current Application Programmer Interface (API), provides positional data in Cartesian space for predefined objects such as fingertips and pen tips. These positional values are given relative to the central point of the Leap Motion controller, which is situated at the position of the second infrared emitter. As depicted in Figure 2, the controller comprises three infrared emitters and two infrared cameras, classifying it as an optical tracking system based on Stereo Vision. Due to the absence of a point cloud for the scene and the presence of predefined detectable objects, traditional calibration methods are not suitable for Leap Motion. Nevertheless, it is essential to establish a precise reference system for evaluating the accuracy and consistency of the Leap Motion controller. The information is subsequently transmitted to the Leap Motion tracking software via a USB connection. This data materializes as a grayscale representation resembling a stereo image of the nearby infrared light spectrum, captured separately by the left and right cameras. To utilize the Leap Motion Controller for hand tracking, we need specialized software. The Leap Motion Controller’s Software Development Kit (SDK) is available in various programming languages, including C++, Java, JavaScript, Objective C, C#, and Python. It ofers compatibility with diferent operating systems such as Windows, Linux, and macOS. The benefit of this versatility is that it

In our paper, we propose a method using a Leap motion

controller to detect and track the human hand using a Software Development Kit(SDK) enabled by Unity 3D. 3.2. Software The following are the main steps of our proposed approach: • A. Real Hand: In this initial step, the camera of the Leap Motion Controller(LMC) captures real hand and finger gestures. These captured gestures are then transmitted to a laptop or PC for processing and analysis. 64–69 enables the development of custom software applications capable of tracking human hand movements, empowering developers to create their own unique software solutions tailored to their specific needs.

Following the transmission of photo data to your computer, the real computational heavy lifting takes place.

Contrary to common misconceptions, the Leap Motion Controller does not directly generate a depth image; rather, it employs sophisticated algorithms to process the raw sensor data.

 3.3. Image captured from LMC

The Leap Motion Controller captures images through its built-in camera arranged in the configuration of an inverted pyramid shape[34]. This controller operates at a frame rate of 120 frames per second, allowing for rapid and continuous image acquisition. Moreover, it ofers a wide field of view, spanning 135 degrees, which ensures comprehensive coverage for tracking hand movements and gestures.

 3.4. Hand skeletal

As mentioned in Figure 3, hand skeletal. It consists of a total of 27 bones. Eight are located in the wrist region, known as the carpals. The palm, referred to as the metacarpals, contains five bones, with each digit being associated with one metacarpal. The remaining fourteen bones are referred to as digital bones, which make up the ifngers and thumb.

Within the palm, there are five metacarpal bones, each featuring a head, a shaft, and a base. These metacarpals correspond to the five digits of the hand.

In the fingers and thumb, there are a total of fourteen digital bones, also known as phalanges or phalanx bones. 4. Experimental Results: The thumb difers from the other fingers as it lacks a middle phalanx. In contrast, the four fingers possess three phalanges each: the distal phalanx, the middle phalanx, and the proximal phalanx.

Additionally, there are small ossified nodes known as sesamoid bones, embedded within the tendons to provide added leverage and alleviate pressure on underlying tissues. These sesamoid bones are typically found at the bases of the digits around the palm, although the precise number can vary among individuals. In this study, the leap motion controller utilizes a conventional skeletal model for tracking the human hand.

 3.5. Convert captured human hand into virtual hand

The Leap Motion Controller functions by using human hand motion as input. It achieves this by employing an infrared(IR) sensor integrated into the Leap Motion Controller to detect the motion of the human hand. Additionally, two cameras within the Leap Motion Controller capture the hand movements that are detected by the IR sensors. These hand gestures are then transmitted to the computer through a USB communication port[35]. As depicted in Figure 4. On the computer, there is a Software Development Kit(SDK) that processes the data related to hand gestures. This processing is typically carried out using the Visual C++ programming language, allowing for the manipulation of hand gesture data for virtual simulation. This simulated hand gesture information is then relayed from the computer to a robot hand using a ZigBee wireless device.

To implement this approach, we construct three distinct virtual environments that are seamlessly integrated with the Leap Motion Controller. The integration between the LMC and these virtual environments is crucial for enabling precise and intuitive hand tracking within the digital space. By combining the tracking abilities of the LMC with the interactive capabilities of Unity 3D, we aim to create immersive and interactive experiences that bridge the gap between the physical and digital worlds. We create three virtual environments that interact in real-time with virtual objects. The first scenario is composed of virtual hands that connect with real hands captured from LMC cameras and show the outputs in unity3D cameras. Figure 6. shows that dynamic hand movements rely on ifngertip and palm velocity to detect complex movement patterns, contrasting with simpler static gestures. Our approach starts by analyzing global hand movements, including translation, rotation, and circular motion. We focus on individual finger movements, with particular emphasis on the index finger, which holds significant importance in communication and interactions.

 5. Conclusion In this research work, hand tracking and its interaction

in diferent scenarios in real-time. LMC observes hand and finger gestures. It displays a three-dimensional image of a hand. The proposed concept of a linked virtual environment with real-time the environment is found to perform better.

In future, the latency of leap motion controller can be reduced by the new version of SDK. The speed of wireless communication might be improved by the new types of wireless communication. The proposed concept can be implemented in remote-operated robotic applications.

 6. Declaration on Generative AI During the preparation of this work, the authors used

ChatGPT, Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. means of touch detection for retail stores, in: CEUR Workshop Proceedings, volume 3092, 2021, p. 89 – 94.

 [1] A. Saudabayev , H. A. Varol , Sensors for robotic hands: A survey of state of the art , IEEE Access 3 ( 2015 ) 1765 - 1782 . [2] J. Kofman , X. Wu , T. J. Luu , S. Verma , Teleoperation of a robot manipulator using a vision-based humanrobot interface , IEEE transactions on industrial electronics 52 ( 2005 ) 1206 - 1219 . [3] R. Avanzato , C. Randieri , Advances and recent applications of 5g: the model of 5g infrastructure and ecosystem in india , in: CEUR Workshop Proceedings , volume 3869 , 2024 , p. 78 - 83 . [4] A. Tibermacine , S. M. Amine , An end-to-end trainable capsule network for image-based character recognition and its application to video subtitle recognition ., ICTACT Journal on Image & Video Processing 11 ( 2021 ). [5] V. Marcotrigiano , G. D. Stingi , S. Fregnan , P. Magarelli , P. Pasquale , S. Russo , G. B. Orsi , M. T. Montagna , C. Napoli , C. Napoli , An integrated control plan in primary schools: Results of a field investigation on nutritional and hygienic features in the apulia region (southern italy) , Nutrients 13 ( 2021 ). doi: 10 .3390/nu13093006. [6] C. Napoli , V. Ponzi , A. Puglisi , S. Russo , I. Tibermacine , et al., Exploiting robots as healthcare resources for epidemics management and support caregivers , in: CEUR Workshop Proceedings , volume 3686 , CEUR-WS , 2024 , pp. 1 - 10 . [7] N. Boutarfaia , S. Russo , A. Tibermacine , I. E. Tibermacine , Deep learning for eeg-based motor imagery classification: Towards enhanced human-machine interaction and assistive robotics 3695 ( 2023 ) 68 - 74 . [8] A. Tibermacine , N. Djedi , Gene regulatory network to control and simulate virtual creature's locomotion ( 2015 ). [9] V. Ponzi , S. Russo , V. Bianco , C. Napoli , A. Wajda , Psychoeducative social robots for an healthier lifestyle using artificial intelligence: a case-study , in: CEUR Workshop Proceedings , volume 3118 , 2021 , p. 26 - 33 . [10] R. Brociek , G. D. Magistris , F. Cardia , F. Coppa , S. Russo , Contagion prevention of covid-19 by [11] S. Russo , I. E. Tibermacine , A. Tibermacine , D. Chebana , A. Nahili , J. Starczewscki , C. Napoli , Analyzing eeg patterns in young adults exposed to motion controller, Procedia computer science 132 diferent acrophobia levels: a vr study, Frontiers in ( 2018 ) 551 - 556 . Human Neuroscience 18 ( 2024 ) 1348154 . [24] G. Marin , F. Dominio , P. Zanuttigh , Hand gesture [12] N. S. Chu , C.-L. Tai , Real-time painting with an recognition with jointly calibrated leap motion and expressive virtual chinese brush , IEEE Computer depth sensor, Multimedia Tools and Applications Graphics and applications 24 ( 2004 ) 76 - 85 . 75 ( 2016 ) 14991 - 15015 . [13] R. Satheeshkumar , Real time robotic arm control [25] T. B. Moeslund , A. Hilton , V. Krüger , L. Sigal , Visual using human hand gesture measurement , Journal analysis of humans , Springer, 2011 . of Advanced Research in Dynamical and Control [26] B. Ladjal , I. E. Tibermacine , M. Bechouat , M. SeSystems 12 ( 2020 ) 984 - 996 . draoui, C. Napoli, A. Rabehi , D. Lalmi , Hybrid mod- [14] G. Zimatore , C. Serantoni , M. C. Gallotta, els for direct normal irradiance forecasting: A case L . Guidetti , G. Maulucci, M. De Spirito , Auto- study of ghardaia zone (algeria), Natural Hazards matic detection of aerobic threshold through re - 120 ( 2024 ) 14703 - 14725 . currence quantification analysis of heart rate time [27] E. Iacobelli , D. Pelella , V. Ponzi , S. Russo , C. Napoli, series, International Journal of Environmental Re- et al., A fast and accessible neural network based search and Public Health 20 ( 2023 ). doi: 10 .3390/ eye-tracking system for real-time psychometric and ijerph20031998. hci applications , in: CEUR WORKSHOP PROCEED- [15] G. Zimatore , M. Cavagnaro , Recurrence anal- INGS , volume 3870 , CEUR-WS , 2024 , pp. 32 - 41 . ysis of otoacoustic emissions, Understanding [28] A. TIBERMACINE , W. GUETTALA, I. E. TIBERMAComplex Systems ( 2015 ) 253 - 278 . doi: 10 .1007/ CINE, Eficient one-stage deep learning for text 978-3-319-07155-8_8. detection in scene images ., Electrotehnica , Elec- [16] D. Kruse , J. T. Wen , R. J. Radke , A sensor-based tronica , Automatica 72 ( 2024 ). dual-arm tele-robotic system , IEEE Transactions [29] A. Tibermacine , N. Djedi , Neat neural networks to on Automation Science and Engineering 12 ( 2014 ) control and simulate virtual creature's locomotion, 4-18 . in: 2014 International Conference on Multimedia [17] C. Randieri , A. Pollina , A. Puglisi , C. Napoli , Smart Computing and Systems (ICMCS) , IEEE, 2014 , pp. glove : A cost-efective and intuitive interface for 9-14. advanced drone control , Drones 9 ( 2025 ). doi:10. [30] G. Capizzi , C. Napoli , S. Russo , M. Woźniak, Lessen3390/drones9020109. ing stress and anxiety-related behaviors by means [18] A. Tibermacine , I. E. Tibermacine , M. Zouai, of ai-driven drones for aromatherapy, in: CEUR A. Rabehi, Eeg classification using contrastive learn- Workshop Proceedings , volume 2594 , 2020 , p. 7 - ing and riemannian tangent space representations, 12 . in: 2024 International Conference on Telecommuni- [31] S. Russo , S. Ahmed , I. E. Tibermacine , C. Napoli , Encations and Intelligent Systems (ICTIS) , IEEE, 2024 , hancing eeg signal reconstruction in cross-domain pp. 1-7. adaptation using cyclegan , in: 2024 International [19] H. Jiang , B. S. Duerstock , J. P. Wachs, A machine Conference on Telecommunications and Intelligent vision-based gestural interface for people with up- Systems (ICTIS) , IEEE, 2024 , pp. 1 - 8 . per extremity physical impairments , IEEE Transac- [32] I. Naidji , A. Tibermacine , W. Guettala , I. E. Tibertions on Systems, Man, and Cybernetics: Systems macine , et al., Semi-mind controlled robots based 44 ( 2013 ) 630 - 641 . on reinforcement learning for indoor application ., [20] A. Tibermacine , D. Akrour , R. Khamar , I. E . Tiber- in : ICYRIME , 2023 , pp. 51 - 59 . macine, A. Rabehi, Comparative analysis of svm [33] N. Brandizzi , V. Bianco , G. Castro, S. Russo , A. Waand cnn classifiers for eeg signal classification in jda, Automatic rgb inference based on facial emoresponse to diferent auditory stimuli, in: 2024 tion recognition , in: CEUR Workshop Proceedings , International Conference on Telecommunications volume 3092 , 2021 , p. 66 - 74 . and Intelligent Systems (ICTIS), IEEE, 2024 , pp. 1 - 8 . [34] A. Sarkar , K. A. Patel , R. G. Ram , G. K. Capoor , Ges- [21] L. Shao , Hand movement and gesture recogni- ture control of drone using a motion controller, in: tion using leap motion controller , Virtual Reality , 2016 international conference on industrial inforCourse Report ( 2016 ). matics and computer systems (ciics) , IEEE, 2016 , pp. [22] R. Satheeshkumar , R. Arivoli , Real time virtual 1-5. human hand using leap motion controller , Interna- [35] S. Hirche , M. Buss , Human-oriented control for tional Journal of Electronics and Telecommunica- haptic teleoperation , Proceedings of the IEEE 100 tions 67 ( 2021 ). ( 2012 ) 623 - 647 . [23] A. Sharma , A. Yadav , S. Srivastava , R. Gupta , Analysis of movement and gesture recognition using leap