TasksSotADatasetsPapersMethodsSubmitAbout
Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable BenchmarksAll SotADatasetsPapersMethods

Community

Submit ResultsAbout

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Papers/Towards Accurate Alignment in Real-time 3D Hand-Mesh Recon...

Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction

Xiao Tang, Tianyu Wang, Chi-Wing Fu

2021-09-03ICCV 2021 103D Hand Pose Estimation
PaperPDF

Abstract

3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR). However, this requires not only real-time speed and accurate hand pose and shape but also plausible mesh-image alignment. While existing works already achieve promising results, meeting all three requirements is very challenging. This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages: a joint stage to predict hand joints and segmentation; a mesh stage to predict a rough hand mesh; and a refine stage to fine-tune it with an offset mesh for mesh-image alignment. With careful design in the network structure and in the loss functions, we can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions. Extensive quantitative and qualitative results on benchmark datasets demonstrate that the quality of our results outperforms the state-of-the-art methods on hand-mesh/pose precision and hand-image alignment. In the end, we also showcase several real-time AR scenarios.

Results

TaskDatasetMetricValueModel
HandFreiHANDPA-F@15mm0.981Tang et al.
HandFreiHANDPA-F@5mm0.724Tang et al.
HandFreiHANDPA-MPJPE6.7Tang et al.
HandFreiHANDPA-MPVPE6.7Tang et al.
Pose EstimationFreiHANDPA-F@15mm0.981Tang et al.
Pose EstimationFreiHANDPA-F@5mm0.724Tang et al.
Pose EstimationFreiHANDPA-MPJPE6.7Tang et al.
Pose EstimationFreiHANDPA-MPVPE6.7Tang et al.
Hand Pose EstimationFreiHANDPA-F@15mm0.981Tang et al.
Hand Pose EstimationFreiHANDPA-F@5mm0.724Tang et al.
Hand Pose EstimationFreiHANDPA-MPJPE6.7Tang et al.
Hand Pose EstimationFreiHANDPA-MPVPE6.7Tang et al.
3DFreiHANDPA-F@15mm0.981Tang et al.
3DFreiHANDPA-F@5mm0.724Tang et al.
3DFreiHANDPA-MPJPE6.7Tang et al.
3DFreiHANDPA-MPVPE6.7Tang et al.
3D Hand Pose EstimationFreiHANDPA-F@15mm0.981Tang et al.
3D Hand Pose EstimationFreiHANDPA-F@5mm0.724Tang et al.
3D Hand Pose EstimationFreiHANDPA-MPJPE6.7Tang et al.
3D Hand Pose EstimationFreiHANDPA-MPVPE6.7Tang et al.
1 Image, 2*2 StitchiFreiHANDPA-F@15mm0.981Tang et al.
1 Image, 2*2 StitchiFreiHANDPA-F@5mm0.724Tang et al.
1 Image, 2*2 StitchiFreiHANDPA-MPJPE6.7Tang et al.
1 Image, 2*2 StitchiFreiHANDPA-MPVPE6.7Tang et al.

Related Papers

ExtPose: Robust and Coherent Pose Estimation by Extending ViTs2025-06-18Monocular 3D Hand Pose Estimation with Implicit Camera Alignment2025-06-10OccRobNet : Occlusion Robust Network for Accurate 3D Interacting Hand-Object Pose Estimation2025-03-27Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation2025-03-25SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training2025-02-21HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos2025-01-06BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting2025-01-01MMHMR: Generative Masked Modeling for Hand Mesh Recovery2024-12-18