Virginia Tech®home

Seminar: Human-Centered Interactive Systems for Configuring, Extending, and Developing AI Applications

Toby Li

PhD Candidate, Carnegie Mellon University

Monday, January 25, 2021
9:30am - 10:30am
Zoom Only


Artificial intelligence (AI) technologies are emerging and affecting our lives in many aspects.  However, the majority of individuals are merely users of AI with little capability to adapt AI to their  own long-tail tasks, preferences, and interests that are not covered by the existing AI solutions.  Democratizing AI to empower the individuals to create, configure, and extend AI-powered systems requires a new human-centered approach to guide the users to provide more effective inputs to the AI systems, and to enable the systems to better understand and to generalize from inputs in the  user’s natural style. In this talk, I will present my research on designing, developing, and studying (1) a new smartphone interactive task learning agent that empowers end-users to teach new tasks,  concepts, and automation rules using the combination of natural language instructions and  demonstrations on existing third-party app GUIs; and (2) a new developer tool that allows  developers without natural language processing expertise to bootstrap a task-oriented  conversational bot from existing mobile apps using the task model extracted from the app  interaction traces. These projects examplify my approach of combining human-centered methods  with AI techniques to create new novel interfaces and system capabilities that facilitate effective human-AI collaboration, lowering the barriers to customizing, extending, and developing AI  applications for end-users and non-expert developers.


Toby Jia-Jun Li is a final year Ph.D. candidate in the Human-Computer Interaction Institute at  Carnegie Mellon University, advised by Brad A. Myers. He also works closely with Tom M. Mitchell.  Toby works at the intersection of HCI, applied AI, and human factors in software engineering, where he uses human-centered methods to design, build, and study interactive systems to empower end- users and novice programmers to create, configure, and extend AI-powered computing systems. Toby has authored more than 15 publications in premier academic venues across HCI, NLP, and  systems (e.g., CHI, UIST, CSCW, ACL, MobiSys, VL/HCC), including three award-winning papers.  Toby’s work is supported by a Yahoo! Fellowship through the InMind project, by J.P. Morgan, and by  NSF. You can find out more about Toby at