Connect teaching to an action interface.
A vision-language-action model connects visual observations and language to robot actions. In-context learning adds a specific requirement: new task evidence must redirect behavior while the deployed neural parameters remain fixed. An instruction can specify a goal; a demonstration can additionally specify an ordering, a contact preference, or a procedure that the same scene does not determine.
The Robot ICL Survey connects these models to a broader account of contextual action inference: identify the information supplied, the representation that carries it, and the action generator that must realize it. This makes a useful distinction between acquiring robot competence during training and directing that competence with an example at deployment.
A pretrained policy needs an input route for each distinction the teaching is expected to express. Examples can enter as visual demonstrations, observation–action sequences, retrieved segments, or retained interaction history. Context interpretation and action generation then have separate responsibilities: preserve the taught requirement and produce a feasible motion.
Choose the evidence that resolves the uncertainty.
| Interface | What it contributes | A useful controlled comparison |
|---|---|---|
| Language instruction | A goal, constraint, or procedure expressed in words. | Change the feasible instruction while keeping the scene fixed. |
| Demonstration context | An example of the required behavior, with or without recorded actions. | Swap the taught order or relation under the same checkpoint. |
| Retrieved or retained history | Relevant examples, earlier cues, progress, or observed action effects. | Replace or remove the relevant evidence while preserving the current observation. |
| Parameter adaptation | A changed neural component, such as an action adapter. | Restore its pre-adaptation weights with inputs held constant. |
Training determines how an input is interpreted. Paired teaching and execution records can make similar observations require different actions. At deployment, retrieval determines which example reaches the policy; phase alignment determines which part is relevant now. These choices remain distinct from the token, diffusion, or flow-based action decoder.
Four studies expose different dependencies.
VIMA: multimodal task specification
VIMA uses multimodal prompts to condition an autoregressive action policy. Its separation of combinations and task templates helps ask which part of a task is unfamiliar. In the survey, demonstration-based specifications illustrate contextual evidence beyond a language goal alone.
ICRT: sensorimotor examples
ICRT places observations and recorded actions in a causal sequence for next-token prediction. It learns this interface without linguistic data, showing that contextual imitation and language conditioning are distinct design choices. The support example exposes actions as well as visual progression.
ICI-VLA: the right phase of an example
ICI-VLA retrieves phase-aligned micro-demonstrations. Its retriever learns from dynamic-time-warping supervision offline; deployment selects examples using observable query inputs and a fixed policy. This separates training-time alignment supervision from the evidence available during execution.
LIBERO-CF: instruction influence
When Vision Overrides Language tests feasible counterfactual instructions in familiar layouts. The survey uses these results to motivate behavioral checks of language dependence: task completion alone can leave unresolved whether the supplied instruction controlled the action.
Measure teaching dependence and physical realization.
Evaluate the same checkpoint with informative, absent, and meaning-swapped context. Measure adherence to the taught target or order alongside completion. Meaning-preserving changes in viewpoint, wording, or demonstrator then test robustness to how the evidence is presented.
Next vary objects, scenes, or embodiment while stating what motor skills and training remain shared. Track which neural modules are frozen, what history persists, and whether retrieval sees only information available to the robot. A context-length gain and a better-trained checkpoint address different questions.
A concrete research direction is to preserve task-defining evidence through retrieval, compression, and action decoding. Correctly representing a requested relation is only one stage; substitutions of context or intermediate representations can test whether that relation reaches physical control.
Continue with context-conditioned policies, connecting teaching data to pretrained policies, and the survey’s attribution controls.