Bridging the Frame Gap: Robot-Centric Pointmaps for VLA Models

Short summary

Robot-centric pointmaps transform visual input into 3D coordinate grids aligned with the robot's frame, solving the persistent camera-to-robot frame mismatch in Vision-Language-Action models. By preserving the H×W grid structure, pointmaps integrate into existing ViT-based architectures with minimal modification. This approach reduces dependence on heavy data augmentation and improves generalization across diverse camera placements.

• •Pointmaps convert RGB input to 3D coordinates relative to the robot's base or end-effector
• •Maintains standard image grid format so existing ViT encoders work without redesign
• •Reduces need for expensive viewpoint augmentation; requires accurate camera calibration and depth estimation

Generated with AI, which can make mistakes.