Arxiv 2024

ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion

 

Jiajun Zhang1, Yuxiang Zhang2, Liang An2, Mengcheng Li2, Hongwen Zhang3, Zonghai Hu 1, Yebin Liu2

1Beijing University of Posts and Telecommunications
2Tsinghua University
3Beijing Normal University

 


We present ManiDext, which takes a sequence of object motions as input and generates dexterous bimanual hand manipulations. The motions are depicted through snapshots at several key frames. Our hierarchical, diffusion-based pipeline first generates contact probability maps and continuous correspondence maps on the object's surface. These contact details then guide the subsequent stage of hand pose generation.

 

Abstract

Dynamic and dexterous manipulation of objects presents a complex challenge, requiring the synchronization of hand motions with the trajectories of objects to achieve seamless and physically plausible interactions. In this work, we introduce ManiDext, a unified hierarchical diffusion-based framework for generating hand manipulation and grasp poses based on 3D object trajectories. Our key insight is that accurately modeling the contact correspondences between objects and hands during interactions is crucial. Therefore, we propose a continuous correspondence embedding representation that specifies detailed hand correspondences at the vertex level between the object and the hand. This embedding is optimized directly on the hand mesh in a self-supervised manner, with the distance between embeddings reflecting the geodesic distance. Our framework first generates contact maps and correspondence embeddings on the object's surface. Based on these fine-grained correspondences, we introduce a novel approach that integrates the iterative refinement process into the diffusion process during the second stage of hand pose generation. At each step of the denoising process, we incorporate the current hand pose residual as a refinement target into the network as a condition, guiding the network to correct inaccurate hand poses. Introducing residuals into each denoising step inherently aligns with traditional optimization process, effectively merging generation and refinement into a single unified framework. Extensive experiments demonstrate that our approach can generate physically plausible and highly realistic motions for various tasks, including single and bimanual hand grasping as well as manipulating both rigid and articulated objects.

 

Overview


Method Overview. Given a sequence of object motion trajectory, we adopt a hierarchical diffusion-based framework to gradually generate the hand poses that manipulate the object. First, we generate contact information on the object's surface, which includes a contact probability map and a continuous correspondence embedding map. These information provides dense correspondence, allow us to compute the geometric residual error at each diffusion timestep T. Subsequently, we use the generated contact information and the computed residual error as conditions to generate the manipulation hand poses.

Video

 

Citation

                @misc{zhang2024manidext,
                  title={ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion},
                  author={Jiajun Zhang and Yuxiang Zhang and Liang An and Mengcheng Li and Hongwen Zhang and Zonghai Hu and Yebin Liu},
                  eprint={2409.09300},
                  archivePrefix={arXiv},
                  primaryClass={cs.CV},
                  url={https://arxiv.org/abs/2409.09300}, 
                  year={2024}
                }