黑料福利网

Topic 

Beyond tele-operation; generating large-scale data for training humanoid VLAs

Abstract

Recent advances in foundation models have shown great promise in imitating teleoperation demonstrations for complex manipulation tasks. These models are built on a vision-language model (VLM) pre-trained on large-scale internet (non-robot) data and are connected to an action module that maps the output of the VLM to robot actions. While very successful, current methods mostly focus on static manipulation problems and fall short in providing a scalable path towards general-purpose humanoid loco-manipulation, mainly because it is impractical to generate a large amount of tele-operation demonstrations for humanoid robots.

In my talk, I outline three main developments that, when combined, provide a scalable framework for generating large-scale data required for training humanoid VLAs. The first component is a general optimization-based task and motion planning (TAMP) framework that generates diverse strategies for achieving different tasks. To enable fast and efficient search, second component uses pre-trained VLMs to provide various manipulation sequence proposals (subgoals) given a desired task and environment. These two components together generate a detailed interaction graph between the robot and the environment. Finally, the third component is a generalist RL policy trained to realize any given desired interaction sequence and enable robust sim-to-real execution of the generated behaviours on the real robot.

About the speaker

Majid Khadiv is an assistant professor in the school of Computation, Information and Technology (CIT) at TUM. He leads the chair of AI Planning in Dynamic Environments and is also a member of the Munich Institute of Robotics and Machine Intelligence (MIRMI). Prior to joining TUM, he was a research scientist at the Empirical Inference Department at the Max Planck Institute for Intelligent systems. Before that he was a postdoctoral researcher in the Machines in Motion, a joint laboratory between New York University and Max Planck Institute. Since the start of his PhD in 2012, he has been performing research on motion planning, control and learning for legged robots ranging from quadrupeds, lower-limb exoskeleton up to humanoid robots.

On March 12 professor Khadiv will give a lecture about Safe Robot learning in the real world.

Your host

Alessandro Saccon, Associate Professor at the department of Mechanical Engineering.

is required but free of charge.