In-Context Learning for Robots

Methods and Applications

Haojian Huang1,2Zexi Li1,3Junhao Guo1Yehang Zhang1,2Wenxuan Peng1,4Bohan Zhou1,3Weilin Ruan1,3Leyi Wu1,2Chenxu Wang1,5Jianchong Su1,2Binghui Xie1,3Wosong Chen1,2Yingjie Xu1,2Tianhao Zhou1,2Suzeyu Chen1,2Pukun Zhao1Jiaqi He1Xinyi Li1,3Runze Li7Peiran Dong1,3Shaoxiang Dang1Jing Huang1Yingbing Chen1Yifan Chang1Tianyi Zhang1Shiyuan Deng1Haozhi Wang1Yangkai Wei1Wenqian Li1Han Yang1Kaiwen Zhou1Huaping Liu5James Cheng3Rui Shao6Donglin Wang7Yaochu Jin7Jianye Hao8Ying-Cong Chen1,2,*Yinchuan Li1,*
1Knowin AI2HKUST(GZ)3The Chinese University of Hong Kong4Tongji University5Tsinghua University6Harbin Institute of Technology, Shenzhen7Westlake University8Tianjin University

* Corresponding authors

huanghaojian · lizexi @ knowin.ai

Read paper GitHub arXiv Forthcoming
From evidence to action

Four method families

How context shapes action.

Context-conditioned policies

Infer / retrieve actions

Geometric demonstration transfer

Align motion & contact

World-model-based control

Predict futures → plan / decode

Skill- and agent-based execution

Compose skills, programs & tools

The central question

What does the robot learn
from context?

01 / ACQUIRE

A new task.

What does the demonstration teach?

02 / TRANSFER

A new situation.

Does it work with different objects and scenes?

03 / RETAIN

The next task.

What experience remains useful?

Physical recursive self-improvement Browse the collection