Robot ICL taxonomy: four context-to-action interfaces

Follow the deployed path from evidence to control.

The survey classifies a method by the intermediate that execution consumes. A backbone name, input modality, or demonstration count does not determine this interface: transformers, diffusion models, and vision-language models can participate in several families.

Four method families and their transfer requirements. Adapted from the survey’s Table 2.
Method family	What execution consumes	What context changes	Requirement for transfer
Context-conditioned policies	An action distribution or retrieved actions	Demonstrations, instructions, or history condition action inference.	The inferred actions preserve the requirement supplied by the teaching.
Geometric demonstration transfer	A motion or contact reference	Correspondence aligns or retargets a demonstrated interaction for tracking or replay.	The matched interaction remains applicable to the current objects, geometry, and contact conditions.
World-model-based control	Predicted consequences	Context constrains future observations, states, or transitions used to decode, plan, or select actions.	The predicted evolution can be physically realized by the executor.
Skill- and agent-based execution	An executable specification	Context determines skill choices, programs, or robot-tool requests for a separate executor.	The available skills and tools preserve the task constraints during execution.

A packing demonstration, for example, can condition a policy’s next action, provide a trajectory to align, constrain an anticipated future, or specify a sequence of callable skills. The task evidence may be shared; the computational route to motion differs. Explore these routes through VLA models, demonstration learning, and robot world models.

Source: Section 2, Foundations, especially Section 2.3; Table 2 (PDF p.11). The method-level discussion continues in Section 3.

Classify the executed intermediate, then describe the supporting mechanisms.

A system can combine interfaces. Describe each deployed branch and the role it performs rather than forcing the whole system into one architecture label.

01Locate the new evidenceWhat requirement or uncertainty does it resolve?
02Trace the intermediateWhat object reaches the executor?
03Identify the updateWhich state, artifact, or parameter changes?
04State the boundaryWhat must remain true for transfer?

Geometry inside a policy

Matching points to condition a diffusion policy remains action inference. Transferring a recorded object-relative trajectory and tracking it is geometric demonstration transfer.

Prediction during training

An auxiliary prediction objective can improve a representation. It becomes a deployed world-model control interface when predicted futures guide action generation or selection.

A VLM choosing actions or tools

Motor-action tokens implement policy inference. Tool names, arguments, and returned feedback form a skill or agent execution interface.

Dynamics inference

Estimating current physical properties to condition a policy differs from predicting future consequences and using them to choose actions.

Classification is branch-specific.

A language model may write a program that calls a predictive planner, while a controller tracks the resulting reference. Record the interfaces and their dependencies explicitly; a single backbone label hides these distinctions.

Source: Section 2.3, “Four method families and shared design dimensions”; PDF pp.9–11.

Memory and correspondence work across the four families.

These mechanisms explain how evidence is selected, interpreted, and retained. They support control interfaces rather than defining additional method families. The guide to embodied-agent memory follows retained evidence into later decisions.

Shared mechanisms in Table 2 and the information they resolve.
Mechanism	Question resolved	Example
Geometric and functional correspondence	Where should interaction occur?	Locate a usable handle on a new object.
Semantic correspondence	Which roles and order matter?	Bind a cup to the source role and a bowl to the receiving role.
Temporal alignment	Which phase of the demonstration applies now?	Find the grasp stage before transferring the next action.
Retrieval	Which earlier experience is relevant?	Select a past demonstration under the present conditions.
Evidence retention	What must remain available?	Preserve an earlier observation or interaction outcome.
External knowledge reuse	What guides later attempts?	Retain repair guidance or a reusable skill.

Also distinguish the information’s role from its encoding. Task specification identifies a goal or procedure; correspondence maps it to executable counterparts; physical-response evidence describes action effects; execution state records hidden events and completed prerequisites. Images, language, and sensor histories can carry more than one of these roles.

Source: Section 2.3 and Table 2; shared context mechanisms are developed in Section 3.

“One-shot” describes a budget. “Fixed-weight” describes an update rule.

Robot ICL redirects existing competence through supplied teaching or interaction while deployed neural parameters remain fixed. A demonstration can also drive gradient updates; the presence of a single demonstration does not establish fixed-weight ICL.

Contextual state

Change the prompt, recurrent state, retrieved episodes, or retained observations. The neural model stays fixed; access and reset behavior still matter.

External artifacts

Revise a program, repair rule, skill library, or context-construction procedure. Identify the changed execution machinery separately from neural weights.

Parameter adaptation

Update a neural subset, fast weights, or persistent model parameters. A frozen backbone alone does not make the complete system fixed-weight.

Preparation and deployment

Training may teach a model how to use context. Evaluate that acquired capability separately from the effect of the new deployment example.

The corresponding controls differ: remove teaching, clear history, withhold retrieved artifacts, or restore pre-adaptation parameters. Our evaluation guide connects each intervention to an observable readout.

Source: Section 2.2, Contextual and parameter adaptation; PDF pp.6–9.

Use the taxonomy with its evidence and scope.

The 12 November author revision contains 616 references. Its four-family census covers 392 methods and comparison items: 160 policy, 29 geometric, 44 world-model, and 159 skill/agent entries. The census includes fixed-weight ICL systems, supporting control mechanisms, and parameter-adaptation comparisons.

The companion living catalogue contains 650 entries and has a broader scope. Supplement S1 records each manuscript reference’s source, classification, coding basis, and citation locations.

Read the foundations Download Supplement S1 Citation formats

Four ways context becomes action.