We propose a robust method for estimating the pose of occluded objects by hand during user interaction in a Head-Mounted Display (HMD) environment. Existing approaches to the occlusion problem often predict the hand and object jointly to improve efficiency, but their applicability in HMD environments is limited by high computational cost and poor generalization to occluded objects. Our approach applies hand pose changes to object pose changes based on the confidence levels of both the hand and the object. Evaluation conducted on 20 distinct grasping pose types demonstrated a lower Mean Per-Vertex Position Error (MPVPE) compared to conventional interpolation methods. Consequently, the proposed method enables effective estimation of occluded objects using fewer computational resources.
Object recognition plays a critical role in scenarios where users interact with objects via
HeadMounted Display (HMD) devices. When real-world objects held by users are not accurately tracked
and such information fails to be transmitted to the device, natural interaction becomes impaired [
Current approaches to occlusion mitigation predominantly employ deep learning and generative
models to simultaneously predict hand and object states, often improving temporal efficiency
compared to sequential prediction methods [
We proposes a method that estimates the current pose and position of objects occluded beyond a certain threshold by the user’s hand by leveraging hand motion information. Our approach enables robust inference of occluded object movement with low computational overhead by applying hand pose changes to object pose estimation. The method integrates object recognition results from both current and previous frames, assessing the sufficiency of available information. When data confidence is adequate, the method heavily relies on current frame information for pose estimation; otherwise, it references prior frames’ data to compensate for missing or unreliable inputs. This temporal data utilization ensures applicability to time-series data and supports real-time object pose estimation on resource-constrained HMD platforms.
Quantitative evaluation was performed using multiple hand-object interaction scenarios [
The contributions of this study are threefold. First, it presents an efficient pose estimation method tailored for real-time HMD interaction environments under limited computational resources, specifi cally addressing occlusion caused by the hand. Second, it enhances robustness by leveraging both current and past frame recognition data, enabling compensatory estimation when immediate informa tion is insufficient or unreliable. Third, it validates improved tracking performance and generalization through comprehensive quantitative experiments involving realistic hand-object grasping and pose scenarios. Consequently, this work demonstrates the feasibility of reliable, real-time occluded object pose estimation for collaborative and interactive applications utilizing HMD devices.
2.1.
Research on pose estimation in hand-object interaction scenarios from images or videos continues
to advance. The H+O framework [
Other research efforts have sought to apply alternative AI models to hand-object pose estimation.
Semi-supervised frameworks have been proposed to enhance pose estimation performance under
occlusion and limited 3D labeled data from single images [
Research on detecting how human-object interactions (HOI) occur continues to advance. The
UnionDet framework [
3.1.
In real-world interactions, the hand and object typically move independently until contact occurs. Therefore, temporal information is utilized to set the initial pose of the stationary object, and subsequent hand pose variations are applied to update the object pose when it becomes occluded. Because the occluded object’s pose is predicted using the remaining recognition results, this approach operates efficiently without requiring additional computational resources.
The system is broadly divided into two stages. The first stage recognizes the hand and object separately based on 2D images from the camera viewpoint. We assume that 3D information of the target object is provided beforehand, and an RGB-D image captured either immediately at contact onset or while in contact is supplied as input. This enables pose estimation of the object in its initial static state, as well as the detection of the hand pose at the instant of contact. From the moment handobject interaction begins, an object detection model is employed to evaluate the recognition confidence of both hand and object. To improve reliability, cropped regions based on the estimated hand and object locations are fed into the object detection model. These detection results subsequently inform the application of hand pose changes to update the object pose.
The second stage estimates the pose based on the detection confidence scores from the first stage. When the object detection confidence is sufficiently high, indicating accurate recognition, the object pose is updated using a dedicated pose estimation model. Conversely, if the object confidence is low, the most recently recognized object pose is updated by applying hand pose variations. This update process also considers the confidence of the hand pose estimation as well as the temporal interval since the last object pose update. the previous time step V O t -1, scaled by a weighting coefficient α t . Formally:
V O t =α t ∙ M H t ∙V O t -1 .
Here, α t quantifies the degree of trust in the previous object pose estimate when computing the current pose update.
α t = {No update , 1 , (λ Δt ∙S hand ), t if S t obj ≥ τ obj if S t obj <τ obj and S t hand <τ hand otherwise (1) (2)
Equation (2) defines the calculation of the weighting coefficient α t . Let S denote the recognition confidence score for the object or the hand at time t , and let τ be a predefined confidence threshold. If the recognition confidence for either the object or the hand falls below τ, it is considered that the respective entity is insufficiently visible, and the system either fully references or disregards the previ ous frame’s information accordingly. In other cases, the hand confidence is used with a decay factor proportional to the number of frames elapsed since the last reliable object pose update to calculate α t . When both the object confidence and the hand confidence drop below respective minimum thresholds, the current frame’s pose estimation becomes unreliable. Therefore, the system preserves the pose from the most recent reliable frame to minimize estimation error.
4.1.
To consider scenarios in which objects are partially occluded by the hand, the SHOWME dataset
[
The evaluation metric employed in this study is the Mean Per-Vertex Position Error (MPVPE). MPVPE quantifies the average positional discrepancy between the vertices of the ground truth (GT) mesh and those of the estimated mesh. For both the proposed method and the interpolation baseline, predicted object meshes are separately saved as obj files to compute this metric. Specifically, the MPVPE is calculated by comparing the obj files of the predicted object mesh against the corresponding ground truth object mesh provided in the SHOWME dataset. Lower MPVPE values indicate smaller deviations between the predicted and actual vertex positions, thus representing higher estimation accuracy. The results are analyzed by plotting graphs for each grasp type and rotation direction to provide detailed performance insights. 4.3.
Prior to deployment on HMD devices, the original dataset values are treated as ground truth and used to evaluate the prediction accuracy. The method follows the previously described system workflow. The model is applied to cropped images focusing exclusively on the hand and object regions within each frame of the dataset. This cropping aims to isolate the hand-object interaction, preventing interference from other objects in the scene and ensuring that confidence scores reflect only the scenario-specific hand and object. The crops were generated using the rendered results provided by the SHOWME dataset.
As a baseline, an interpolation method was considered. When the confidence scores for the hand and object in the current frame fall below the threshold used in the proposed method, the object pose is estimated as the midpoint between the previous and subsequent frames. This interpolation approach is analogous to the proposed idea of predicting object pose based on the hand pose change between consecutive frames. The interpolation is applied specifically at the point where the weighting coefficient is calculated during object pose estimation at time t in the system. Otherwise, all other processing steps remain identical. This setup enables direct comparison of evaluation metrics between the proposed method and the interpolation baseline.
Significant variability exists in object detection rates across scenarios, heavily influenced by the training quality of the detection model. Experiments were conducted not only using the raw detection results but also by artificially adjusting detection rates per scenario. This allowed assessment of which method performs better relative to the detection model’s effectiveness. When using unmodified detection results, object pose prediction is triggered only if the object fails to be detected by the detection model, which in this study is Mediapipe. When detection rates are artificially manipulated, undetected frames are randomly prioritized for pose prediction using the respective methods. Hand pose confidence for prediction always relies on Mediapipe model outputs.
When utilizing the dataset, the grasping method, object type, and rotation sequence/direction are not consistent. Therefore, we additionally grouped the data into rotation units of 10 frames, which served as the minimum rotation segment for classification. In other words, one unit corresponds to a 100frame video (10 data points). For all 20 scenarios, these units were grouped, and the corresponding rotation vectors were classified into rotation types using the k-means clustering algorithm. Based on these rotation types, we examined which rotation directions each scenario is more specialized in, thereby enabling more accurate estimation. The number of clusters was experimentally adjusted by varying k until clusters with identical directions and motion tendencies no longer appeared. Consequently, the six clusters obtained represent distinct directions or tendencies (i.e., consistency of rotation).
5.1.
First, when specifying different detection rate ratios, we compared the Mean Per-Vertex Position Error (MPVPE) results for each scenario and detection rate using both the proposed method and the interpolation baseline. The results demonstrated that the proposed method consistently achieved lower MPVPE values across all scenarios. Scenario 14 (Medium Wrap) exhibited a substantial performance gap favoring the proposed method regardless of the object detection rate. In contrast, Scenarios 4(Inferior Pincer), 8(Tripod), and 13(Quadpod) showed relatively minor differences between methods, irrespective of detection rates. These scenarios involve grasps on smaller objects, which may contribute to smaller absolute errors in both methods. Additionally, although these scenarios feature longer durations with diverse rotations, the smaller radius of object rotation results in relatively low errors even when using conventional interpolation.
Across all detection rate variations, the proposed method consistently outperformed the interpolation approach. Furthermore, as detection rates decreased, the performance gap widened, indicating that the proposed method is particularly effective when recognition confidence is low. Conversely, performance stabilized when detection rates surpassed a certain threshold. This plateau is likely due to the decay factor in the weighting coefficient α t , which diminishes proportionally with the number of frames elapsed since the last reliable object pose update. Higher detection rates reduce the number of frames over which decay applies, leading to more stable pose estimations.
To further analyze results by rotation type, scenarios were grouped into six rotation clusters. Clusters 0 and 2 have opposite rotation directions, so they revealed significant differences in MPVPE despite the symmetrical rotation axes. For example, Scenarios 9 (Parallel Extension), 10 (Power Sphere), and 11 (Precision Sphere) frequently exceeded an MPVPE of 0.002 in Cluster 0, whereas in Cluster 2, most scenarios remained below this threshold. This discrepancy is hypothesized to result from longer occlusion durations caused by the rotation direction. Longer occlusions increase the number of frames over which decay in pose confidence is applied, thereby reducing prediction reliability and increasing error. Thus, the proposed method demonstrates better performance when occlusion occurs in shorter, repeated intervals rather than in prolonged continuous segments.
Next, the comparison between Clusters 4 and 5 focused on whether rotation direction remained consistent or changed midway. Cluster 4 generally exhibited higher MPVPE values, with Scenario 13 (Quadpod) showing a twofold increase compared to Cluster 5. This result suggests that predicting object pose from hand pose changes is more straightforward when rotation direction remains constant. When rotation direction changes, the hand’s rotational velocity typically decreases, resulting in longer occlusion intervals and greater difficulty in accurate prediction.
11.88
According to a recent study analyzing the impact of frame rate on user experience in virtual reality
environments [
In addition, the FLOPs value of the proposed method is relatively small when compared to the
computational capabilities of current HMD devices and smartphones, thereby confirming its feasibility
for deployment on such platforms. Finally, the maximum memory consumption of approximately 2
GB further indicates that the system is well within the RAM capacity of modern HMD devices,
ensuring its practical applicability [
We proposed a method to estimate the pose of objects occluded by the hand through the utilization
of hand pose changes. To evaluate the performance advantage of our method compared to the
conventional interpolation approach, tests were conducted on 20 grasp types from the SHOWME
dataset[
In addition, we calculated the step-by-step processing time and computational resource usage to examine whether the proposed method could be utilized on real-time HMD devices. Since our method requires very little time per frame, we demonstrated that it can be applied in terms of processing time, provided that the object detection and hand pose estimation models to be used together are appropriately selected for real-time operation. Furthermore, in terms of FLOPs, we confirmed that the method is applicable when considering the performance levels of HMD devices and general smartphones.
Although the SHOWME dataset used in this study contains certain instances of directional changes, frames exhibit predominantly linear tendencies. Hence, it remains necessary to evaluate whether the proposed approach can be generalized effectively to datasets characterized by more complex motion patterns. Furthermore, since the SHOWME dataset is limited to single-hand manipulation of an object, additional validation is required in scenarios that align more closely with the research objective— namely, multi-user interaction with objects in immersive HMD environments, where multiple users may manipulate a single object simultaneously. In addition, to comprehensively evaluate different grasping methods, we utilized a dataset that can be classified into 20 grasp types and performed experiments under the assumption that the hand and object in the images are observed from the user’s perspective. To further validate the user’s direct manipulation of objects, we plan to conduct additional user studies.
In this regard, future work may consider incorporating a weighting term that accounts for complex movements. For example, there is determining which hand’s confidence level should be applied when estimating object pose changes in multi-hand scenarios. Such an extension would enhance the robustness of the proposed estimation method for interaction with virtual avatars, which constitutes the final objective of this research. Additionally, as this study has prioritized performance validation of the proposed method, relatively less attention has been devoted to the selection of the hand pose estimation model. Further investigations into model selection could thus provide a more comprehensive guideline for the effective application of the proposed framework.
This paper was supported by Korea Institute for Advancement of Technology(KIAT) grant funded by the Korea Government(MOTIE) (RS-2025-02304167, HRD Program for Industrial Innovation)
During the preparation of this work, the author used GPT-5 in order to: Grammar and spelling check. After using these tools, the author reviewed and edited the content as needed and takes full responsibility for the publication’s content.