A High-speed 3D Vision-based High-speed Multi-fingered Hand Manipulation System
Driven by significant advancements in structure, sensors, and control algorithms, the multi-fingered hand and their applications have increasing attention from both academia and industry.
While most researchers have focused on the interaction between multi-fingered hands and static, rigid objects, many real-world tasks, such as catching moving objects, are dynamic.
Furthermore, everyday environments contain numerous non-rigid objects like paper and clothes.
Therefore, our focus is on designing a multi-fingered hand manipulation system capable of successfully interacting with dynamic, non-rigid objects.
We set our task goal as catching a falling paper sheet.
To achieve this, we are breaking down the task into teleoperation, reinforcement learning-based control, and deformation feedback.
In this study, we successfully achieved these three sub-goals, culminating in the high-speed 3D visionbased high-speed multi-fingered hand manipulation system.
This system demonstrated a latency of less than 1ms, an average paper deformation of 0.54 mm, and a success rate of 100%.
Movie
If you want to use the original video, please send an e-mail for copyright permission to yamakawalabo
.
References
- Xiaohang Shi, Qitong Guo, Ruoyu Jia, Chunxin Yang, Kenichi Murakami and Yuji Yamakawa: A 3D Vision-based Framework for Teleoperation and Dynamic Catching with A High-speed Multi-fingered Hand, IEEE Robotics and Automation Letters, Vol.11, Issue 4, pp.5230-5237 (2026) [link]
- Xiaohang Shi, Qitong Guo, Ruoyu Jia, Chunxin Yang, Kenichi Murakami, and Yuji Yamakawa: Ultra-low-latency 3D measurement with a 1D approximate de Bruijn pattern, Optics Letters, Vol.50, Issue 9, pp.3142-3145 (2025) [link]