Separate the taught requirement from its realization.
Learning from demonstration can mean training a policy from recorded behavior or using a new example to direct an existing controller. The practical distinction is what changes when that example arrives: neural weights, contextual inputs, or an explicit reference for execution. A single demonstration can support any of these routes.
The Robot ICL Survey connects demonstration learning to the intermediate consumed by the robot: actions, motion references, predicted consequences, or executable skills. This companion guide focuses on how that choice changes the information a demonstration must provide and the assumptions behind transfer.
Consider teaching a packing order. The robot must preserve the order while changing its reaches and grasps for the current scene. For a handle interaction, the demonstrated contact may itself be essential. Deciding which relation must survive is the starting point for choosing a representation.
Match the representation to the interaction.
| Route | What the demonstration supplies | What execution must preserve |
|---|---|---|
| Context-conditioned policy | Video, sensorimotor records, keypoints, or spatial relationships read by an action generator. | The taught distinction through a learned mapping to current robot actions. |
| Geometric transfer | Object-relative motion, contact keyframes, poses, or a tracked reference. | An applicable interaction after alignment and a feasible motion for the receiving robot. |
| Predictive control | Evidence that specifies a desired future or conditions the prediction of action effects. | Task-consistent predicted evolution that a decoder or planner can realize. |
Observation-only video leaves the demonstrator’s commands implicit. A sensorimotor trace exposes them, but their coordinates, timing, and gripper semantics must be compatible with the target interface. Geometry can establish correspondence explicitly. It can either condition a learned policy or produce a reference that a separate controller tracks.
Preparation therefore includes more than collecting examples. Support and query executions must express the same task requirement. Calibration connects recorded commands to physical effects, and temporal selection connects the current decision to the relevant demonstration phase.
Four ways to use a new example.
Vid2Robot: reading visual progression
Vid2Robot conditions robot actions on demonstration video through cross-attention. The support video need not contain recorded robot commands; query execution still supplies action supervision during training. This separates what the teacher shows from what trains the action output.
Instant Policy: relational context
Instant Policy uses a graph-diffusion policy conditioned on robot demonstrations. Spatial representations make relationships available to action generation. The survey also discusses how additional demonstrations can resolve task interpretations that remain ambiguous in a single example.
DITTO: transferring object motion
DITTO reconstructs manipulated-object motion from an RGB-D human demonstration, transfers the trajectory to objects in the robot scene, and realizes it through grasping and motion planning. The embodiment problem appears in how the robot produces the demonstrated object effect.
MT3: retrieving an applicable interaction
MT3 combines language filtering, geometric retrieval, and pose alignment before interaction replay. Its retrieval branch can add task references without new task-policy training. Reuse still depends on restoring each interaction’s preconditions; longer procedures require appropriate stage references.
Test intent, correspondence, and execution separately.
First change the demonstration while holding the robot scene fixed. Use examples with different feasible orders, targets, or relations, then measure whether behavior follows the supplied requirement. Next preserve the requirement while changing the demonstrator, view, object shape, or initial arrangement. The two tests distinguish interpreting the teaching from transferring its realization.
Record assistance as part of the method: depth calibration, object poses, contact annotations, action labels, and human corrections determine what information is available. Report demonstration acquisition and execution attempts as well as success. An accurate alignment can still fail through reachability, contact, or controller limitations.
A useful research opportunity is to determine when a new interaction needs a new reference. Independently moving objects and changing contact roles can invalidate a single global transform. Phase-specific correspondence and observation-driven correction make that boundary testable.
See context-conditioned policies, geometric demonstration transfer, the geometric comparison table, and teaching–execution training pairs.