Seminar: Watch, Predict, Act: Robot Learning Meets Web Videos

Abstract

Developing robots that can help us in our daily activities has been a guiding aspiration in AI for decades. Yet, general-purpose robots that work out of the box, are able to perform tasks without manual interventions, and are safe to interact with remains an elusive goal. Since collecting robot interaction data in diverse scenarios is difficult due to operational constraints, a  key challenge in robotics is being able to perform new tasks in novel scenes without requiring robot data collection in every scenario. Going beyond the standard of end-to-end imitation learning, in this talk I will describe an alternate paradigm towards this goal: combining robot-specific data with predictive planning from diverse web videos such as YouTube clips of humans doing daily chores.

By learning to predict motion and contextual cues from naturally diverse web data for robotic policy learning, I will demonstrate how this recipe enables training common goal/language-conditioned policies capable of multiple in-the-wild manipulation tasks in unseen offices and kitchens, across different robotic embodiments. I will conclude by laying out my vision for the coming years, encompassing the entire spectrum of learning structure from diverse web datasets, developing robot learning algorithms that can scalably use such structured predictions for broad generalization, and deploying robotic systems in-the-wild by being adaptable to deployment-time constraints and compliant with human preferences. 

_Biography

Homanga Bharadhwaj is a final-year PhD student in Carnegie Mellon University. His research goal is to develop embodied AI systems capable of helping us in the humdrum of everyday activities within messy rooms, offices, and kitchens, in a reliable, compliant, and scalable manner without requiring significant robot-specific data collection and task-specific heuristics. Homanga was named a Future Leader in Robotics and AI by the University of Maryland in 2025 and was selected as a Meta AI Mentorship (AIM) Fellow in 2022. His research has received a Best Paper in Robot Manipulation Finalist at ICRA '24, the Best Conference Paper Award at ICRA '24, the Outstanding Presentation award at NeurIPS '23 Robot Learning Workshop, and has been covered by several media outlets like TechCrunch, IEEE Spectrum, and VentureBeat among others.