Human activity recognition (HAR) has garnered significant attention owing to its potential applications in ifelds such as healthcare, automated driving, and personalized user interfaces. The use of wearable devices for collecting real-time motion data has spurred ongoing research in activity detection and prediction. However, the development of robust HAR models faces a critical challenge: the dificulty in acquiring diverse and comprehensive datasets of human activities. Ethical considerations and practical limitations often impede the collection of realworld motion data, particularly for sensitive or uncommon activities. To address this challenge, we introduce a novel system-the Unreal Data Generator-that leverages the capabilities of Unreal Engine 5. This tool facilitates the synthesis of time series data for a wide range of activities and scenarios by simulating data collection from multiple wearable device locations using virtual human models. Experiments demonstrated that augmenting real-world datasets with synthetic data generated by the Unreal Data Generator significantly improves the performance of deep learning-based HAR models. Specifically, by training our deep convolutional neural network model with both real-world and synthetic data, the accuracy in the WISDM benchmark test improved from 0.784 to 0.823. This approach ofers a promising solution for improving the robustness and generalizability of HAR models by providing access to a rich and diverse range of training data.
Human activity recognition (HAR) is a rapidly evolving field that enables machines to automatically
recognize and interpret human actions and movements. HAR systems have numerous applications in
healthcare [
Recent advancements in wearable device technology have significantly accelerated progress in HAR
[
Despite advancements fueled by readily available data, significant challenges persist in developing robust HAR systems. Acquiring suficient data for accurate predictions, particularly for infrequent yet critical activities such as falls and syncope, remains a major obstacle. Conducting large-scale studies with diverse participants further complicates data collection. Furthermore, ensuring data quality and mitigating sensor noise are crucial considerations. Future research must focus on addressing these limitations while improving the generalizability of HAR systems to unseen scenarios and diverse populations.
To address these data acquisition challenges, we developed Unreal Data Generator, the architecture of which is illustrated in Figure 1. Our methodology comprises two main stages. First, as detailed in Figure 1a, we leverage Unreal Engine, a real-time 3D creation tool, to simulate human movements. The world coordinates are recorded and then converted into raw sensor signals, which are subsequently refined through a dedicated processing pipeline. Second, as depicted in Figure 1b, this processed synthetic data is used to augment real-world datasets. By simulating a wide range of activities and scenarios under controlled parameters, this approach efectively overcomes the limitations and ethical concerns of collecting large-scale real-world data, especially for rare events. The primary objective is to leverage synthetic data generated within Unreal Engine to enhance the accuracy and robustness of deep learning-based HAR models by augmenting real-world datasets and improving performance on human activity recognition tasks.
Numerous studies have been conducted on HAR using multi-channel time series data from sensors
such as accelerometers and gyroscopes, obtained from wearable devices and smartphones [
In model-centric improvement, HAR using neural networks has been actively studied. For example,
convolutional neural networks (CNNs) have been employed to efectively capture local patterns in
sensor data [
While these deep learning models have shown promising results and continue to be actively developed,
they often require large amounts of training data to achieve optimal performance. This requirement
for extensive datasets poses a significant challenge for the practical implementation of HAR systems.
Therefore, a data-centric approach, focusing on improving data quality and exploring efective data
augmentation techniques, becomes crucial for building robust and generalizable HAR models with
limited resources. Kim et al. [
The application of GANs has shown promise in generating synthetic data for HAR tasks. However, a
potential limitation of relying solely on GANs lies in ensuring the diversity and comprehensiveness
of the generated data. Simulation-based approaches ofer a complementary strategy by allowing for
the creation of varied datasets through the manipulation of virtual environments and parameters. For
instance, Waqar et al. [
UE5.4 was selected as the standard engine for this study. The motion-matching template was downloaded,
and all necessary plugins and configurations were set up following the oficial documentation [
Because UE5’s default mannequin ofers only basic actions—running, walking, standing, and jumping—custom animations need to be added for specific actions, such as climbing stairs and sitting. To implement the stair animations, a stair detection capsule and a Boolean variable were first added to indicate when a character was "on stairs." Then, stairs were configured as a separate collision layer that overlapped only with the character’s stair capsule. Logic was implemented so that the Boolean variable switched to “true” when the stair capsule overlapped with the stair collision. Stair-specific animations (e.g., "running upstairs" and "ascending stairs") from Mixamo were integrated into the motion-matching system. A condition in the animation player triggered the appropriate animations when the Boolean variable was true. The sitting action was implemented using a montage system. Fifteen types of sitting animations were downloaded from Mixamo and used to create diferent montage ifles, with data captured and updated iteratively.
The data collection system was developed by placing sockets on designated bone locations and using the Break Vector and Break Rotator math functions to obtain world coordinates and rotation angles, respectively. In the Unreal Data Generator, the sampling rate, measurement time, and measurement points can be freely configured. For this study, time series data were collected at a sampling rate of 50 Hz for approximately 300 s for each activity across diferent animations (default mannequins, preset female mannequins, and three custom animation layers). Details of the dataset are presented in Table 1.
The Unreal data generated by UE5.4 contain time series data of unreal world coordinates (, , ) and rotation information as roll, pitch, and yaw ( , , ). The relationships between the coordinates (), rotation angles ( ), velocity (), acceleration (), and angular velocity () are defined by the following equations (1–4): = (, , ), = ( , , ) = ≈ + Δ∆ − = ≈ + Δ∆ − = ≈ + Δ∆ − Since the acceleration data measured by wearable devices in the real world are afected by gravity and changes in device posture, we analyzed the efects of gravity on real-world data and applied gravity correction to the Unreal data as follows: (1) (2) (3) (4) • Calculation of average acceleration per axis for each motion. • Determination of gravity compensation values: For each axis, the average acceleration was compared to the gravitational acceleration (9.81 m/s²). • If the absolute diference between the average acceleration and 9.81 (or -9.81 for downward acceleration) was within a threshold of 2.0 m/s², then 9.81 or -9.81 was used as the gravity compensation value for that axis. It was assumed that the sensor was primarily aligned with gravity. • Handle outliers: If the average acceleration on any axis fell outside the threshold of ± 2.0 m/s² from 9.81 or -9.81, the median acceleration for that axis was used as the gravity compensation value instead. This approach is more robust for handling outliers and noisy data. • Calculation of the rotation matrices: Euler-ZYX rotation matrices were computed from the rotation angle data at each time step of the Unreal data. • Apply rotation and gravity compensation: The gravity compensation vector (obtained in steps 2 and 3) was multiplied by the rotation matrix (calculated in step 4) for each time step. This process transformed the gravity compensation vector from the sensor coordinate system into the world coordinate system, accounting for the sensor’s orientation. • The resulting compensated gravity vector was added to the raw acceleration readings on each axis to account for the real-world efects of gravity.
Finally, a Butterworth low-pass filter [
As real benchmark datasets, we selected two types: WISDM [
The learning task was defined as a six-activity classification problem using 10 s windowed data sampled
at 20 Hz (i.e., a window size of 200). For HAR model development, we implemented a one-dimensional
convolutional neural network (1D-CNN) based on a previous study [
We developed Unreal Data Generator for synthetic data generation to improve the HAR system with accelerometers and gyroscopes. By utilizing UE5 as the standard engine, diverse motion data can be generated across various scenarios through intuitive key operations, much like controlling a game character. For instance, the data collected while walking on a paved road difers from that collected while walking on a bumpy road. Movement speed is also adjustable, allowing it to run straight and fast or to keep circling around the area. The Motion Matching in UE5 enables smooth animation transitions, thus ofering natural and realistic motion responses to complex user operation inputs. Additionally, the sensor position for data collection and the sampling rate can be specified arbitrarily. This level of customizability is particularly useful for optimizing sensor positions on wearable devices and for other related applications.
We developed an unreal data generation system using the real-time 3D creation tool UE5. To extract unreal data, we utilized the real-world coordinate system in UE5 and recorded six activities—walking, jogging, upstairs, downstairs, sitting, and standing—using five types of mannequins sampled at 50 Hz. Activity animations were obtained from the Fab website (Unreal Engine marketplace). The values for accelerometers and gyroscopes were calculated based on the time diferences in the coordinate dynamics for each record. We recorded each motion for 5 minutes, resulting in a total of 150 minutes of motion capture data (5 mannequin types × 6 motion types).
Figure 2 compares the real (top subplot) and unreal (middle and bottom subplots) datasets. As shown in Figure 2, the raw synthetic data generated by Unreal Engine exhibited significant noise and deviated considerably from the real-world data. This discrepancy arises primarily because the synthetic acceleration is derived by numerically diferentiating position data twice, a method highly susceptible to amplifying noise. In contrast, real-world accelerometers measure acceleration directly as a physical force and often include hardware-level filters, resulting in inherently smoother signals.
To address this, we applied a two-step processing method. First, a Butterworth low-pass filter was used to remove the high-frequency noise inherent in the diferentiation process. Second, to account for the influence of gravity present in real-world measurements, we implemented the gravity correction detailed in Section 3.2. The processed unreal data were significantly closer to the measured real-world values (see bottom subplots in Figure 2).
To validate the improvement in HAR accuracy for diferent datasets, we trained separate 1D-CNN based activity recognition models for WISDM and DSADS datasets, using Unreal Engine-generated synthetic data to augment each. Model performance was evaluated using k-fold cross-person validation in terms of accuracy, precision, recall, F1-score, and AUC. We compared the performance of models trained solely on real-world data with those trained on augmented datasets incorporating synthetic data. The results are summarized in Table 2.
On the WISDM dataset, the baseline model trained solely on real-world data achieved an accuracy of 78.41%. Augmenting the dataset with Unreal data generated by UE5 improved HAR performance. Specifically, applying a low-pass filter and gravity correction to the Unreal data further enhanced performance, resulting in an accuracy of 82.55%. Similarly, on the DSADS dataset, the addition of processed unreal data also improved HAR performance, whereas raw Unreal data negatively impacted model accuracy. These findings highlight the importance of incorporating low-pass filtering and gravity correction during data preprocessing when developing HAR models using Unreal Engine-generated data.
The confusion matrix for HAR classification in the WISDM benchmark is shown in Figure 3. The accuracy of the HAR model improved significantly for motions involving movement when Unreal data were included. Although classification accuracy for diferentiating between the static postures of sitting and standing did not show improvement, the accuracy in dynamic/static motion classification demonstrated noticeable gains. Furthermore, the confusion matrix revealed improved classification accuracy for similar motions, such as walking/jogging and climbing upstairs/downstairs.
Unreal Data Generator facilitates the acquisition of acceleration and gyroscopic sensor data across a wide range of motions and scenarios. Typically, 3D animations in game engines operate by repeatedly playing back pre-recorded animation data, which results in identical sensor data being generated from the same animation. However, the ALS in UE5 employs a dynamic procedural animation system that adapts to the environment and player inputs in real time. This approach blends animations and adjusts them based on factors such as terrain, speed, and direction, creating more fluid and realistic character movements. Compared with traditional animation techniques in game engines, these advanced methods provide a more immersive and responsive character movement system. In this study, we developed a more flexible movement animation by integrating ALS, motion matching, and Foot IK techniques.
As mentioned previously, generating natural and diverse movements using default mannequins can be easily achieved. However, generating data for specific custom actions requires the addition of new animations. For instance, in this study, a sitting animation montage was downloaded from FAB to capture sitting motion data. The FAB database currently ofers a wide variety of motions, providing a diverse range of animations that can be utilized. When specific motion data are not available in the FAB database, new animations must be created. This process can be streamlined by leveraging software that generates motions from videos (DeepMotion [29]) or using generative AI tools specialized in video generation (Runway Gen-3 Alpha [30]).
The primary diferences between Unreal Engine-generated data and real-world sensor data lie in the nature of the measurements. The real data generator provides world-space coordinates, dynamics, and angular rotation, whereas real-world wearable devices measure acceleration, angular velocity, and other digital biomarkers, such as heart rate, using their own algorithms. Notably, while real-world sensors measure acceleration as the force applied to the device—afected by gravity—acceleration derived from world coordinate dynamics in Unreal Engine does not account for gravity. This distinction is crucial when comparing or integrating data from these two sources.
Our results demonstrate the profound impact of correcting for this gravitational efect. In the real world, the gravitational force registered by a 3-axis accelerometer changes based on the sensor’s orientation. Our method of applying a gravity vector transformed by the sensor’s rotation matrix successfully embeds this orientation-dependent component into the synthetic data. The efectiveness of this is most apparent in static activities like ’Standing’ and ’Sitting’ (Figure 2e, 2f). For these poses, the raw synthetic acceleration is nearly zero, as position coordinates are constant. In contrast, a real sensor registers non-zero values corresponding to the constant pull of gravity. Our gravity correction replicates this real-world phenomenon, transforming the unrealistic near-zero signals into realistic static ofsets, thus creating much more faithful training data.
These specific corrections for gravity and signal noise are part of a broader challenge in
simulationbased research known as the "Sim-to-Real Gap"—the discrepancy between simulated and real-world
data [31]. Our study demonstrates that it is essential to mitigate this gap through a dedicated data
processing pipeline to improve the data’s fidelity. We addressed this by implementing a low-pass filter
for noise and using Euler rotation matrices for gravity correction. While this direct signal processing
approach proved efective, future research could explore alternative methods to bridge this gap. For
instance, incorporating domain adaptation techniques or leveraging Generative Adversarial Networks
(GANs) could ofer ways to learn the transformation from simulated to realistic data automatically
[
While this study demonstrates the potential of using Unreal Engine for data augmentation, it has several limitations. First, the scope of our evaluation was limited. The experiments were conducted on two public datasets, covering only six basic daily activities such as walking, running, and sitting, and used a single backbone architecture (1D-CNN). Furthermore, our study focused on validating a simulation-based approach and did not include a direct comparative analysis against other categories of data augmentation, such as model-based techniques like GANs. Second, our methodology for bridging the Sim-to-Real Gap was confined to a direct signal processing pipeline (i.e., low-pass filtering and gravity correction). The exploration of other advanced methods, such as automated domain adaptation, was not covered in this work.
We thank Epic Games Japan for the complimentary use of Unreal Engine for this research.
During the preparation of this work, the authors used Gemini in order to: draft content, Grammar and spelling check, Paraphrase and reword, Text translation, Improve writing style, and Literature review generation. After using this tool, the authors reviewed and edited the content as needed and took full responsibility for the content of the publication. [29] Deep Motion Inc., Deep motion, 2025. URL: https://www.deepmotion.com/. [30] Runway AI Inc., Introducing gen-3 alpha: A new frontier for video generation, 2025. URL: https: //runwayml.com/research/introducing-gen-3-alpha. [31] E. Salvato, G. Fenu, E. Medvet, F. A. Pellegrino, Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning, IEEE Access 9 (2021) 153171–153187. doi:10.1109/ACCESS.2021.3126658. [32] A. Akbari, R. Jafari, Transferring activity recognition models for new wearable sensors with deep generative domain adaptation, in: Proceedings of the 18th International Conference on Information Processing in Sensor Networks, ACM, New York, USA, 2019, pp. 85–96. doi:10.1145/ 3302506.3310391.