電腦視覺與具身 AI
HOPE Predicts Hand Pressure from Monocular Video, Reaching 0.660 Vertex-Level Contact F1 on OpenTouch
HOPE maps tactile-glove data, planar sensor readings, and contact-only annotations onto a unified MANO hand mesh. It can output per-vertex pressure and contact from ordinary RGB video, although its public page currently still lists the code and models as “coming soon.”
Vision models can usually determine only whether a hand is touching an object. Estimating the magnitude of applied force has typically required planar pressure mats, single images, or coordinates tied to specific sensors. HOPE reformulates the task as hand-centric video prediction: regardless of object shape or sensor layout, its outputs are per-vertex contact probabilities and normal pressure on a MANO hand mesh, enabling training across data sources using a shared representation.
The system first projects taxels from the OpenTouch tactile glove, planar readings from PressureVisionDB, and distance-based contact annotations from DexYCB and ARCTIC onto a common mesh. VertexFormer treats each hand vertex as a token that persists across frames, alternating among vertex self-attention, cross-attention to DINOv3 image patches, and temporal attention. Hand poses are supplied by external reconstructors such as HaWoR. The pressure head uses a contact gate, `p̂ = ĉ ⊙ p̃`, encoding “no contact, no pressure” directly into the architecture while also allowing contact data without force values to constrain the pressure distribution.
On the held-out OpenTouch test set, HOPE achieves a frame-level contact F1 of 0.874 and a vertex-level F1 of 0.660, compared with 0.835 and 0.361, respectively, for HACO. Its per-vertex pressure MAE is 1.808 kPa, lower than PressureVision’s 1.93 kPa. After adding contact-only HOI data, per-vertex contact F1 for bare hands on ARCTIC rises from 0.069 to 0.498, while pressure performance on OpenTouch remains largely unchanged, indicating that the shared mesh can indeed absorb heterogeneous supervision.
The limitations are equally clear. On PressureVisionDB’s pixel-level contact IoU metric, HOPE scores 0.328, still below the native planar model’s 0.547, because mesh projection compounds hand-reconstruction errors. In-the-wild videos are demonstrated only qualitatively, with no ground-truth pressure measurements available for validation. Left hands are still mirrored and processed by the right-hand model. Although the project page provides interactive results, the code and weights remain marked as “soon.” Researchers in embodied manipulation should next focus on generalization across hand shapes, tangential forces, object reaction forces, and closed-loop validation in real robot control.