Gesture recognition with millimetre wave radar has been extensively researched and many datasets are publicly available. However, datasets with raw data (including voltage levels) are almost not available. These are essential as they allow for recreation of all signal representations, thus allowing evaluation of gesture detection pipelines on the same datasets. This paper presents the design, implementation, and testing of a data recording prototype for hand gesture recognition coupling the Texas Instruments IWR6843ISK radar sensor and magic leap motion sensor. The study emphasizes on the system architecture, system apparatus and implementation to optimize recorded data quality and format flexibility (including voltage levels).
CEUR
ceur-ws.org
Radar Sensor and Leap
In recent years, gesture recognition with millimetre-wave (mmWave) radars has garnered
significant interest among researchers [
Current gesture recognition datasets with mmWave radars often provide processed data in the form of Range-Doppler, Range-Angle, or point-cloud representations. While these formats ofer certain benefits, they lack the flexibility and granularity of raw Voltage data, which are essential for understanding and comparing diferent gesture recognition pipelines. Further still these datasets predominantly only ofer one sensing modality, thus failing to provide reference data CEUR Workshop Proceedings streams. For example, radar sensor can be used to reconstruct point-cloud data, however to the best of our knowledge no dataset exists where reference point-cloud data (ground truth) would be available besides the radar data. To address this gap, we present the design, implementation, and testing of a data recording prototype for hand gesture recognition coupling the Texas Instruments (TI) IWR6843ISK radar sensor and magic leap motion sensor.
Even though Texas Instruments provides extensive documentation and user guides for conifguring the IWR6843ISK radar sensor with the DCA1000EVM data capture card, the process is not always straightforward. This is especially true when users need to develop their own applications and customize the sensor to suit specific use cases. Moreover, integrating multiple sensors together can be challenging. In this paper, we delve into the process of configuring the radar sensor and integrating it with the LeapMotion sensor to achieve robust data capture. We also explore the dificulties we encountered during the prototype setup and how we addressed them, providing insights into our decision-making process.
This data capture tool incorporates two sensors: IWR6843ISK radar sensor1 and an Ultraleap LeapMotion sensor2. The tool captures mid-air gesture trajectories in real-time from both sensors. The radar sensor operates at a frame rate of 100 frames per second (fps), while the LeapMotion sensor operates at a frame rate of 115 fps. Both sensors are triggered simultaneously via two sockets, and the timestamps of each frame are logged. This allows the tool to align the corresponding radar frames with the LeapMotion hand captures.
The user interface (UI) of the application was implemented using the React3 JavaScript library and pre-built semantic components from the Semantic UI4 front-end development framework. React’s component-based approach and Semantic UI’s intuitive and reusable components made it
an ideal choice for building a user-friendly and visually appealing interface. React’s componentbased structure allowed for modular development, ensuring that UI elements could be easily reused across the application. This approach simplified code maintenance and contributed to a maintainable and scalable codebase. Semantic UI’s pre-built semantic components provided a consistent and modern design language, ensuring that the UI elements were well-organized, accessible, and visually appealing. This library helped to streamline the design process and ensured that the application’s UI conformed to industry standards. The results is an interface shown in Figure 2
The backend was developed using Node.js5, Express.js6, and Atlas MongoDB cloud database7. The database stores gesture metadata and metadata about recordings such as number of frames captured, number of zero filled frames, number of Bytes recieved, and recorded temperatures of the antennas. The Express.js routing framework was utilized to eficiently handle incoming user requests and data from the sensors. Locally running backend was responsible for communicating with both the radar sensor and the LeapMotion sensor via sockets to acquire and store gesture
data in the Atlas MongoDB database and local disk drive. The asynchronous and non-blocking I/O capabilities of Node.js, coupled with the eficient routing capabilities of Express.js, enabled the backend to handle a high volume of data streams from the sensors with minimal latency. The NoSQL structure of Atlas MongoDB provided flexibility in storing and retrieving gesture metadata, making it suitable for the dynamic nature of mid-air gesture data.
In our research, we employed the Texas Instruments (TI) IWR6873ISK, an advanced on-antenna radar module, as our primary radar sensor. TI provides the mmWave Studio application, a comprehensive software tool designed for communication and configuration of the radar sensor. We utilized this application for both configuring and triggering the sensor. A notable feature of mmWave Studio is its command-line accessibility, facilitated through built-in Lua shell. This feature allows for the configuration of the radar sensor via a custom Lua script.
The process of sensor activation is initiated through a Lua script within mmWave Studio. This script serves two primary functions: firstly, it enables the detailed configuration of the sensor through a series of specific commands. Once the sensor is configured, the Lua script then establishes a socket server, actively listening for trigger commands issued from the backend of our application. Upon receiving a trigger command from the backend, the Lua script proceeds to activate the radar sensor.
The operational duration of the radar sensor is set to a precise two-second window upon activation. During this period, the sensor transmits a total of 200 frames. Following the transmission of each frame, the radar sensor relays the raw voltage data of the radar receiving antennas ( ) to the DCA1000EVM data capture card8. This transmission occurs through an LVDS (Low-Voltage Diferential Signaling) interface, ensuring high-speed data transfer and integrity. Upon reception, the DCA1000EVM data capture card sends the captured radar data to the computer via an Ethernet port using User Datagram protocol (UDP), ensuring eficient and timely transfer.
In our setup, the antennas of the radar sensor capture data at a bandwidth exceeding the capabilities of standard UART COM port communication. To address this, Texas Instruments provides the DCA1000EVM data capture card, designed specifically for streaming radar sensor data. This card connects to the radar sensor’s LVDS port and transmits the data to a computer via an Ethernet port. While the DCA1000EVM receives operational commands through UART COM ports, the data transfer is executed through Ethernet for higher bandwidth eficiency.
Texas Instruments also ofers a Command-Line Interface (CLI) control program for the DCA1000EVM9. This program manages the data flow from the capture card, filling in any missing frames with zeros and reordering the data before storing it as binary files. Additionally, it compiles the statistics of the received data into a separate CSV file.
In our research, we employed the Ultraleap Leap Motion sensor for tracking hand movements. Ultraleap provides the Gemini software 10, a C#-based SDK, to facilitate the sensor’s integration with computers for data capture. To integrate the Leap Motion sensor with our systemb́ackend, we developed a custom Leap Motion controller script, leveraging the LeapC Python wrapper11.
The Leap Motion controller initiates a socket server to listen for interruption signals from the backend. Upon receiving an interruption signal, the controller starts the recording of hand events. The Leap Motion sensor transmits the detected hand coordinates as events via USB-C to the computer. These events are then captured by our Leap Motion controller using the LeapC Python wrapper, ensuring seamless and accurate acquisition of hand movement data.
9https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/1023/TI_DCA1000EVM_CLI_ Software_DeveloperGuide.pdf 10https://leap2.ultraleap.com/gemini-downloads/ 11https://github.com/ultraleap/leapc-python-bindings
Upon acquiring the raw data, our initial step is to evaluate its utility and interpretability.
This assessment is conducted by visualising the data using several established representation
methods in radar signal processing. Figure 3 shows Range-Doppler[
Range-Doppler Map
Velocity Range-Angle Map
Meters Complex signal Time-domain
In our experimental setup, two distinct approaches were employed for triggering the radar sensor. The first approach involves configuring the radar sensor to emit an indefinite number of frames, wherein data recording occurs exclusively during the user’s gesture performance. This method eliminates the need to activate the radar sensor for each gesture repetition.
The alternative approach necessitates the activation of the radar sensor concurrently with the commencement of the user’s gesture, thereby initiating data recording. This method requires the radar sensor to be invoked for every gesture repetition. However, the comparative advantages of these two approaches – finite frame versus infinite frame configurations – are not immediately apparent. To investigate this, we conducted a focused within-subject user study.
The study’s objective was to assess the sensor’s eficacy across these two operational modes by monitoring the incidence of data packet loss and variations in sensor temperature.
For a robust evaluation, we conducted a controlled within-subject experiment, maintaining consistent external variables while comparing the frequency of missing data frames and temperature fluctuations between the two sensor operation modes.
We selected a set of five commonly used gestures that are relatively simple to execute:
SwipeLeft, Swipe-Right, Swipe-Up, Swipe-Down, and Hold[
Initially, we set the sensor to Infinite mode, during which the participant completed 50 gestures repetition in total. After each repetition, we recorded the antenna temperatures and the count of any missing data frames.
Subsequently, we powered down the sensor for one hour to allow it to return to baseline temperature conditions. Following this cool-down period, we reconfigured the sensor to Finite mode, ensuring that startup temperatures were consistent with the initial trials.
We analysed the mean antenna temperatures and number of missing radar frames in two sensor configurations. Since the mean antenna temperatures ( = 0.958, < 0.01 ) and number of missing radar frames ( = 0.53375, < 0.00001 ) were not normally distributed, we opted for Wilcoxon matched pairs [15] test with false discovery rate (FDR)[16] correction.
The mean temperatures and number of missing frames across study conditions are shown in Figure 4. Significant efect could be detected between Infinite and Finite sensor configurations when analysing the distribution of the mean temperatures. ( = 0, < 0.0001 ) Similarly, significant efect could be detected between the same same sensor configurations when analysing the distributions of the number of missing radar frames. ( = 229, < 0.05 ) mode finite infinite Finite frames
Infinite frames Mode
Finite frames
Infinite frames Mode
The evaluation demonstrates that the sensor efectively captures the raw voltage signal, and the acquired data can be successfully interpreted using standard radar signal processing techniques.
Our initial setup involved setting the radar frame frequency to 100 fps, with the radar continuously radiating frames until manually stopped. This approach, however, led to a significant rise in sensor temperature, reaching approximately 100°C within 30 minutes of activation. To mitigate this issue and maintain temperature control, we modified the radar operation to activate only during gesture execution. The radar now radiates frames for a precise two-second duration aligned with gesture performance, followed by deactivation. This intermittent activation efectively regulates the sensor temperature. Moreover, we observed that at temperatures exceeding 90°C, there was a notable increase in data frame loss. This error was substantially reduced with our revised method of operation. section 4 shows that the number of missing radar frames and the antenna temperatures are significantly diferent in the two sensor configurations. However, we should acknowledge that the findings are based on a single-participant study.
The TI Radar Toolbox provides various applications such as Out-of-Box-Demo12, Area-Scanner13, and Gesture-Recognition14, designed for deployment within the IWR6843ISK radar sensor. These applications leverage the radar’s inbuilt DSP chip for signal processing, outputting data formats such as point cloud, Range-Doppler, Angle-Doppler, or recognized gestures and objects via UART COM ports. Additionally, by setting ‘lvdsStreamCfg -1 1 1 1‘ in the configuration file, the sensor can be configured to output raw raw Voltage data through the LVDS interface.
However, this approach presents limitations when operating at higher frame rates, such as 100 fps, rendering it unsuitable for our specific needs. Therefore, in our study, we opt for processing the signal externally on a computer, post capturing the raw Voltage data. This method not only circumvents the frame rate limitation but also ofers greater flexibility in data analysis and processing.
This study presents the development of a prototype system designed for capturing raw voltage data from the IWR6843ISK radar sensor during hand gesture performance. In our innovative approach, we integrated two distinct sensors: the IWR6843ISK radar sensor and Ultraleap’s Leap Motion sensor. We developed a specialised application capable of simultaneously recording both the radar’s raw voltage data and the Leap Motion sensor’s hand snapshots during gesture execution.
Furthermore, we delved into the operational dynamics of the sensor, examining how diferent configurations afect antenna temperatures and the occurrence of missing radar frames. This exploration is crucial for optimising sensor performance and ensuring reliable data capture.
However, our research also encountered challenges, particularly in achieving synchronisation when capturing data from multiple sensors. Addressing these challenges is critical for enhancing the accuracy and eficiency of the data collection process. This study not only contributes valuable knowledge to the field of gesture recognition using radar technology but also opens avenues for future research in improving multi-sensor integration and synchronisation.
We acknowledge support from Slovenian Research Agency, grant number P1-0383, P5-0433 and J5-1796 and support from research program CogniCom (0013103) at the University of Primorska. 12https://dev.ti.com/tirex/explore/node?node=A__AJwREK3NVquf175i533giA__radar_toolbox__1AslXXD_ _LATEST 13https://dev.ti.com/tirex/explore/node?node=A__AOxzzy5SR.-h2AygLnSknw__radar_toolbox__1AslXXD_ _LATEST 14https://dev.ti.com/tirex/explore/node?node=A__ABBjTJ7Y0H4LgRlsbV2S7g__radar_toolbox__1AslXXD__LATEST on Human-Computer Interaction 7 (2023) 1–25. [15] M. Marusteri, V. Bacarea, Comparing groups for statistical diferences: how to choose the right statistical test?, Biochemia medica 20 (2010) 15–32. [16] J. D. Storey, R. Tibshirani, Statistical significance for genomewide studies, Proceedings of the National Academy of Sciences 100 (2003) 9440–9445.