In-Context Learning for Robots: Methods and Applications

1 Introduction

A robot may know how to move yet still need evidence about what to do. Demonstrations specify a fold or assembly order; corrections revise the procedure. Interaction reveals friction or misalignment. In navigation, earlier visits reveal locations outside the current view [203]. In-context learning (ICL) for robots studies how such evidence changes deployed behavior without another task-specific update to neural parameters.

Broad pretraining supplies reusable perception and control [118–120] from multi-task and multi-embodiment collections [57, 60]. Yet the current scene can leave several procedures feasible, conceal an earlier event, or reveal little about an unfamiliar material’s response. These are information gaps that broader motor competence alone cannot resolve. Fine-tuning incorporates new evidence into weights; ICL makes it available at the decision. Cross-object transfer tests whether teaching remains useful after replacing either or both interacting objects [345]. The task relation must survive while its physical realization changes.

Two historical roots motivate a broad notion of context: one-shot imitation infers intended behavior, while meta-reinforcement learning infers tasks or dynamics from outcomes [1, 2]. Sequence models retain trajectories and learning histories [5, 6]; multimodal prompting combines task specifications and sensorimotor examples [13, 100]. With broader priors, teaching can direct generalist actions [98, 99], predicted futures [24], or executable procedures [31]. Physical experiments supply evidence for revising execution [88, 89]. Language participates throughout as instructions, examples, corrections, and retained summaries. Figure 1 situates these capabilities within complementary horizons of adaptation and longer-term learning.

Training establishes how a model uses context, as language-model studies illustrate [111, 121]. Robot learning must also determine what to teach and what to retain. RoboCoach directs supervision toward predicted skill failures [442]; RoboRSI turns diagnosed execution failures into reusable skill revisions [347]. Retained corrections can then inform later attempts [47, 49]. These mechanisms motivate active coaching that makes teaching informative and physical recursive self-improvement that improves subsequent learning. Their long-term value is transfer to new tasks.

Existing surveys organize learning from demonstration by teaching interfaces and learned policies, rewards, or plans [122, 123]. ICL and in-context RL reviews examine example- and interaction-based adaptation [124, 125], including environmental change [127]. Robotics reviews cover information transfer from human video [126], context and transfer in manipulation [371], VLA architectures [372], data and evaluation [129], embodiment [130], predictive control [128, 178], and executable skills [369]. At the system level, Towards Agentic Robots [548] organizes agents’ roles across Model, Data, Environment, and Harness, with detailed treatment of orchestration, memory, recovery, and skill evolution.

Our organizing question is how new evidence resolves what existing competence leaves undetermined, and how that resolution survives physical execution. This evidence-to-action perspective complements the system-level organization of agentic robotics: we trace which demonstration, history, or correction changes behavior, through which intermediate, and with what transfer limits. The four families distinguish these intermediates; correspondence, training, and memory explain how they preserve task information. Evaluation separates deployment-time context use from parameter updates and executor competence. Table C1 in Appendix C compares these perspectives, including recent world-action taxonomies [370].

Placing an object at the correct destination can still violate a required handle grasp. We therefore trace both the intended effect and prescribed order or contact through execution, developing three contributions:

• A taxonomy of context-conditioned policies, geometric demonstration transfer, world-model-based control, and skill- and agent-based execution, with shared correspondence and memory.
• An account of how training relationships establish context use and how object substitution, unfamiliar environments, and execution conditions limit transfer, separating broader motor competence from broader ability to learn through teaching.
• A synthesis of reported comparisons and evaluation controls that separates context dependence, transfer, and retained-experience benefits, motivating active coaching and transferable self-improvement.

Figure 2 follows this argument: why a robot needs context (Section 2), how context changes action (Section 3), and how that dependence is learned (Section 4). Recovery and applications expose its physical limits (Sections 5–6); evaluation tests its benefit (Section 7); the research agenda asks which limits must be overcome for broader transfer and improved learning (Section 8).

2 Foundations: Why Do Robots Need Context?

A packing robot may already grasp every part yet need a demonstration to establish their order, history to identify completed placements, and contact evidence to accommodate a tighter fit. The same visible scene can therefore require different actions. Useful context resolves this decision-relevant ambiguity through actions the robot can already execute. This distinction motivates the learning horizons, adaptation regimes, and four control interfaces below. Their history traces how robots acquired broader ways to interpret and act on new evidence.

2.1 From competence to contextual adaptation

We use six learning horizons to organize the relationship between prior competence, contextual adaptation, and learning from retained experience. S3 and S4 describe complementary sources of evidence; S5 and S6 express research objectives whose realization can combine several earlier capabilities. Together, the horizons connect distinct adaptation mechanisms to individual and collective learning objectives.

The six horizons distinguish where learning changes the system (Figure 3). Explicit control programming (S1) encodes task logic, control laws, and planning models [134–138]; neural policy learning (S2) acquires reusable behavior through imitation and reinforcement [139, 140]. History-based adaptation (S3) uses interaction memory to infer task or dynamics information. In-context task learning (S4) infers new task requirements from supplied teaching, including instructions, demonstrations, textual examples, and corrections. Both forms of ICL for robots redirect execution with deployed neural parameters fixed.

In assembly, contact outcomes can reveal a tighter fit (S3), while a demonstration or correction can specify a new part order (S4). A retained correction may guide both interpretation and execution; these roles can share a model and input modality.

Automated practice [114, 141], reward and experiment design [89, 115, 116], and retained guidance [47, 49] support a further objective: physical recursive self-improvement (S5), in which experience improves how the next task is learned. Collective knowledge evolution (S6) extends this learning cycle to a group of robots. Its premise is that an execution contains knowledge another body can use, even when the original motion cannot be copied. Shared progress and skill representations preserve what should be accomplished [29, 30, 79]; pooled policy learning supplies competence across sensing and action spaces [57, 132]; morphology-aware transfer changes the physical realization [131, 133]. These are complementary foundations for exchanging demonstrations and corrections. Whether such exchanges improve the group’s subsequent learning is the S6 research objective in Figure 1.

Active coaching chooses demonstrations, corrections, or practice for the learner’s current needs. Self-reflection diagnoses execution and turns verified lessons into reusable guidance, skills, or learning strategies. Together they connect teaching in S4 to improved subsequent learning in S5 (Figure 1). Retained context and programs can change with neural weights fixed; neural consolidation adds a separate training stage.

New evidence changes either policy parameters or inference-time inputs (Figure 4). Fine-tuning uses task data ${\mathcal{𝒟}}_{\mathrm{task}}$ to update pretrained parameters $𝜃$ to task-adapted parameters ${𝜃}_{\mathrm{task}}$. The illustrated ICL case supplies a support demonstration $D^s$ as context $C_t$, keeping $𝜃$ fixed. Both use existing perception and motor competence; the storage of task information determines acquisition cost, persistence, and reset behavior.

The updated object, rather than the optimizer’s name, identifies the adaptation regime. Closed-form instruction distillation writes the effect of a better instruction into a VLA’s MLP parameters [614]; language-model-directed robot debugging likewise edits selected weight rows without gradient descent [612]. Both change neural parameters. By contrast, optimizing an inferred physical latent or candidate action can leave the networks fixed. Reporting the modified object, its persistence, and its reset rule makes “training-free” and “gradient-free” claims interpretable.

What context enables a robot to learn. Fixed-parameter adaptation through supplied teaching or interaction is the review’s central subject. Parameter-adaptation methods provide comparisons; pretrained policies and control mechanisms supply the competence on which adaptation operates. New evidence can specify a command, identify a familiar routine, compose known relations, or teach an unfamiliar rule. These behaviors differ in the information acquired from context and the prior competence used to realize it. Section 8.1 develops these distinctions. The initial test is whether changing task-defining evidence changes behavior appropriately under matched execution conditions.

2.2 Contextual and parameter adaptation

Adaptation can reside in contextual state, a retained external artifact, or neural parameters. A common policy interface makes these alternatives comparable; their storage and reset behavior determine how an evaluation can identify the source of change. Table 1 collects the shared notation, separating supplied evidence, execution intermediates, and retained state.

At decision step $t$, the current attempt has history $h_t=(o_1,a_1,\dots ,a_{t-1},o_t)$. Observation $o_t$ includes the available images, proprioception, force, tactile or inertial measurements, and feedback messages. Action $a_t$ is the applied command; its measured physical effect appears in subsequent observations. Task-relevant context $C_t$ may contain instructions, goal references, multimodal support demonstrations $D^s$, textual input–output examples, corrections, retrieved episodes, or memory of interaction. Superscript $s$ denotes support evidence; the demonstration-only case in Figure 4 sets $C_t=D^s$. Context can summarize the current history or retain information from earlier attempts. The policy proposes an action block $A_t=({\widehat{a}}_{t\mid t},\dots ,{\widehat{a}}_{t+H-1\mid t})$ of horizon $H$, where ${\widehat{a}}_{t+\omega \mid t}$ is the action proposed at decision $t$ for offset $\omega$. With deployed neural parameters $𝜃$,

Only executed actions enter the subsequent history; new observations can replace the unexecuted part of a proposed block. When $C_t$ itself contains the state needed for future decisions, a memory update incorporates an action and its observed consequence:

Equation (2) updates a self-contained context state. For context retrieved or compressed from experience, Equation (17) distinguishes the stored memory from the evidence supplied to the policy.

The policy interface is compatible with several architectures. Throughout the survey, $𝜃$ denotes the collection of deployed neural parameters, including context interpretation, action generation, and any intervening learned model. A subscript $𝜃$ on a component means that it uses the relevant subset of this collection; different components need not share weights. During fixed-parameter adaptation,

The subscript $0$ denotes the start of the deployment adaptation phase. Observations, retrieved examples, recurrent state, or stored experience change instead. Equations (2) and (3) use one executed action per update; if several actions of $A_t$ are executed before replanning, their observed transitions are incorporated in order. A reward or other feedback signal can be included when available. Updating a neural component at deployment constitutes parameter adaptation, even when the rest of the system stays frozen.

Figure 5 traces a packing task. Prior competence supplies grasps and motions; teaching specifies part order; context construction binds its steps to the scene; execution produces placements; assessment checks them; and reuse retains verified corrections. The four families differ at contextual execution. The surrounding operations determine which evidence reaches it and whether its requirement is physically realized.

A corrected order changes the required actions; recorded placements identify the remaining work. Both can affect a fixed policy, but the retained information and its reset rules determine how long the change persists.

A fixed model can revise a goal, skill sequence, or program after feedback; an executor turns that specification into actions. For example, a language model can use prior errors to adjust a motion primitive’s settings [9]. Retaining the revised program lets later attempts reuse the correction without training neural weights. This stores task information in executable content, whereas fine-tuning stores it in model parameters.

The surrounding execution system, or harness, assembles context, exposes robot tools, executes requests, and returns feedback. Normally, this machinery remains fixed while the supplied evidence and generated requests change. A learning system can also revise its retrieval rules, context-construction code, or executable controller. We call this model-external adaptation: experience changes the machinery connecting a model to the robot. Identifying this revised component distinguishes changes in the execution system from changes in the neural model.

Figure 6 separates supplied evidence (a), recurrent state (b), and retrieved episodes (c) from fast-weight adaptation (d) and persistent weight updates (e). Storage and retention timescale are independent: external memory can persist without training, while test-time updates can be reset or retained. Continual learning examines acquisition and retention across changing experience [143].

These distinctions determine the evaluation controls. Removing supplied evidence tests (a); clearing recurrent state tests (b); suppressing episode retrieval tests (c). Restoring pre-adaptation weights tests the contribution of parameter updates in (d–e). Section 7 develops these interventions under matched task conditions, so gains can be attributed to the information or parameters that actually changed.

One-shot specifies the amount of teaching, not its storage mechanism. Visual meta-imitation uses one demonstration for task-specific gradient updates [7]. RoboTTT restricts updates to a neural subset during long-context execution [10]; WAM-TTT adapts video-side fast weights from target-domain human video before rollout [11]. Both follow the selective-update principle in (d), despite different update timing. Thus one demonstration can support either fixed-weight conditioning or parameter adaptation. The next taxonomy distinguishes how the resulting context reaches robot actions.

2.3 Four method families and shared design dimensions

The primary taxonomy follows the deployed path from context to action. The four families below are alternative computational interfaces, distinguished by the intermediate that execution consumes. We then separate this classification from the cross-cutting roles, modalities, and storage of evidence, before resolving boundaries between hybrid implementations.

• Context-conditioned policies infer actions from task evidence, including demonstrations, language, and interaction history as supported, through a learned generator or local action retrieval.
• Geometric demonstration transfer reconstructs a demonstrated motion or contact reference, aligns or retargets it to the current scene, and executes that reference through tracking or replay.
• World-model-based control predicts future observations, states, or transitions and uses those predictions to decode, plan, or select actions.
• Skill- and agent-based execution infers skill choices, programs, or robot-tool requests for a separate executor; VLM and LLM agents form one branch.

These interfaces extend the mappings, system models, and plans distinguished in learning from demonstration [122], independently of the representation, optimization, and training objective [144]. Transformers, diffusion models, and VLMs can each implement several interfaces; hybrids can combine predicted futures with skills or programs with reference tracking.

Four information roles explain what this context contributes. Task specification identifies the intended goal, relation, or procedure. Correspondence relates demonstrated objects, contacts, phases, or commands to their executable counterparts. Physical response describes how the current body and environment respond to action. Execution state records the scene, relevant hidden events, and completed prerequisites. A demonstration can specify a procedure and its contact geometry; an interaction can update both physical response and execution state. These roles guide evidence selection within every method family.

Language, images, and sensor records can carry the same information roles. Modality describes the encoding; the learning mechanism explains how its information changes a decision. Numerical states serialized as text retain their physical meaning.

Acquisition describes collection through teleoperation, UMI, egocentric video, or simulation (Section 4.1); sensorimotor channels describe recorded contents. Language can accompany any source. Collection choices and neural update rules remain independent of the control family and application demand.

The same demonstration can condition actions, define a motion reference, constrain a future, or specify a skill request (Figure 7). The intermediate required by execution distinguishes the families; acquisition, retrieval, and storage can support several interfaces.

Storage adds another independent choice. A context window, recurrent state $m_t$, or external archive $M_t$ can supply the same task information with different access costs and persistence. Static API documentation establishes the command interface; task-specific examples and interaction supply changing evidence. For later comparisons with test-time training, write ${𝜃}_t=(\overline{𝜃},{\varphi }_t)$: $\overline{𝜃}$ is the frozen backbone and ${\varphi }_t$ the adaptive neural subset. This follows the update-regime distinction in Section 2.2.

The boundary between policy conditioning and geometric transfer is the object being executed. Matching points to condition a diffusion policy remains context-conditioned policy inference; transferring a recorded object-relative trajectory and tracking it is geometric demonstration transfer. Similarly, estimating current dynamics for an action policy differs from predicting future consequences for control. RMA and the physical-context inference in ICWM illustrate the former [39, 41]; DOMINO and Zero-WAM illustrate the latter [19, 24]. An auxiliary future-prediction head contributes to training; a deployed future predictor contributes to world-model-based control when its output guides action generation or selection.

A shared VLM can support different interfaces. Motor-action tokens implement policy inference; robot-tool selection, arguments, and returned feedback implement agent-based execution. GPT-Policy’s Astra configuration follows the latter route [94], using demonstrations and interaction to determine its calls.

Memory connects these interfaces across time. Alongside its form, function, and dynamics [145], a retained fact carries physical conditions that determine when it remains applicable to the current robot and scene. Section 3.5 examines access and update mechanisms, and Section 5 examines how execution validates or revises retained evidence.

Intent adaptation changes the desired relation; physical adaptation changes its realization under new dynamics or embodiment. Joint adaptation requires both. Navigation additionally needs spatial knowledge even when the goal and dynamics remain unchanged. Section 6 organizes applications by these information needs and their implications for long procedures and task changeover. Any of the four computational families can serve several demands.

Table 2 connects each intermediate to its transfer assumption and shared mechanisms. A handle match can condition action inference or define a tracked motion reference. The execution interface determines which assumption must hold for the final motion.

2.4 Historical development

The historical account follows an expansion in what must be inferred from context. Task-family adaptation learned reusable responses to bounded variation; sequence interfaces made more of the preceding experience accessible; broader priors increased the behaviors that new teaching could direct. These developments overlap across the four families. The final publication census locates their recent growth within the literature reviewed here.

Task-family adaptation. Early contextual policies amortized adaptation across a task distribution: training learned how to interpret evidence so that deployment could change behavior without relearning the entire controller. In one-shot imitation, paired executions taught how one instance specifies actions in another [1]. In meta-RL, recurrent state carried adaptation across episodes [2]; probabilistic task inference separated evidence interpretation from control [4] and made task uncertainty available to action selection [333]. The resulting adaptation rule inherits the distinctions made available during training. Extending it to human demonstrations requires correspondence across bodies [8]; extending it to changed robot dynamics requires evidence about action response [39]. These extensions concern different unknowns, despite sharing rapid adaptation as their objective.

The four branches in Figure 8 respond to different obstacles in reusing experience. Context-conditioned policies learn which actions an example implies; geometric transfer identifies a relation under which a recorded interaction remains applicable. Predictive control makes consequences available before motion, while skill and program interfaces separate a new procedure from its motor realization. Their histories overlap because the obstacles coexist. Broader priors expand the available behaviors, but also make evidence selection and retention more consequential: a capable controller still needs to follow the current teaching rather than a familiar default.

Online system identification provided an explicit early version of the physical-adaptation idea: recent state–action history identified dynamics parameters for a universal policy [146]. Later methods could learn a latent instead of naming every parameter, but the central problem remained inferring how the present body would respond. The demonstration and interaction strands were already connected by Watch–Try–Learn: one policy attempted a demonstrated task and another used the resulting feedback [147]. A demonstration narrows the task hypothesis, while interaction can resolve information it omits.

Reusable sequence interfaces. A sequence interface addresses a tension between compressing experience and revisiting a decisive event. SNAIL combined temporal convolutions, which aggregate the preceding sequence, with attention, which retrieves relevant earlier information [3]. This made adaptation depend on both accumulated state and selective access. Transformers provide query-dependent access through attention [231]. Retaining more history increases the available evidence; attention and downstream computation determine which evidence informs the current decision.

Broader teaching made a fixed task summary less sufficient: two demonstrations can have the same goal yet differ in order, contact, or tool use. Multimodal task specifications placed goals and procedures in the input [100, 148], while trajectory models retained earlier observations and actions as evidence for later decisions [5, 6]. The benefit of a sequence interface was access to distinctions that a single task label could erase. Its cost was an alignment problem: the robot had to recover which part of that evidence mattered at its own execution stage. Temporal video correspondence and causal sensorimotor modeling addressed this pressure through learned representations [13, 14]; spatial representations imposed geometric correspondence before action prediction [16]. These complementary approaches preserve temporal evidence and geometric correspondence at different stages of action generation.

Adaptation over broad priors. Broad pretraining changes what adaptation must accomplish. When perception and a useful range of motion are already available, a new example can specialize an action model [17] or constrain an anticipated robot-domain future [24], rather than supply every ingredient of a new skill. Interaction histories similarly let reusable controllers account for a deployed body’s response [40, 88]. Broader pretraining therefore expands the behaviors available to teaching, but also strengthens familiar continuations that may conflict with a new example. Evaluation must establish whether the example can redirect this competence.

Geometric reuse and learned contextual inference allocate generalization differently. Visual nearest-neighbor methods rely on a representation in which nearby observations admit compatible actions [149]. Explicit alignment instead restores the conditions under which a recorded robot motion or human-demonstrated object trajectory remains usable [34, 35]. When those conditions are insufficient, a video can initialize further interaction learning, as in WHIRL [150]. The same one-example interface can therefore support replay, transformed reference execution, or new policy learning. What distinguishes these routes is the assumption that makes the demonstrated behavior executable in the new setting.

World-model-based control exposes the consequences of behavior before executing it. Contextual dynamics models first made recent transitions useful for evaluating candidate action sequences [19, 20]. Demonstration-conditioned models then added a second role for context: specifying which future should be realized [21, 24]. This development joins physical prediction with task interpretation. The controller must preserve the procedure inferred from the example while predicting how it can unfold in the robot’s current scene.

Task structure offers a different way to reuse competence: preserve the procedure while delegating its physical realization. Demonstration-derived hierarchies express a new task through existing routines [26]. Language-model programming expands the procedure to include perception calls, arithmetic, and feedback logic [27]; video-derived outlines let visual teaching specify that procedure for an existing planner and executor [31]. The gain is compositional reuse of capabilities. Its condition is that the intermediate preserve the task’s binding details, because the executor cannot recover a contact or ordering requirement that was omitted during interpretation. Section 3 develops this trade-off across the four interfaces.

These developments connect teaching, interaction, and procedural reuse. General-Level distinguishes breadth from synergy in multimodal generalists [151]; for robots, the corresponding learning question is whether combining these capabilities improves subsequent acquisition. LMPC studies responsiveness to later teaching [117], while reward and learning-program search change how new skills are acquired [89, 115]. Section 8.5 develops this question for individual and shared learning.

Review scope and source selection. This review uses a mechanism-oriented narrative synthesis with targeted updates through 14 November 2026. Searches and citation chaining span visual imitation, contextual meta-RL, navigation from video and experience, system identification, example-conditioned programming, interaction memory, and robot foundation models. Sources include arXiv, official proceedings and journal articles, and author or organization releases. Targeted updates also use Hugging Face Daily Papers, alphaXiv, and daily literature records for discovery, followed by primary-source verification and identity-based deduplication. Technical claims are traced to these primary sources; release reports are identified as such where their evidence differs from a research paper.

Included works explain a route from new evidence to behavior or supply relevant data, evaluation, training, or theory. Fixed-weight contextual methods are central; parameter adaptation and prior competence provide comparisons. Versions share one entry, preferably the official publication; separately cited papers and dataset releases remain distinct. Hybrid families follow the analyzed deployment branch, independently of modality, storage, and parameter updates.

The reference inventory in Appendix A records this coding and the first public year, using arXiv v1 or an official release. Supplement S1 provides the item-level records and counting rules. The resulting counts describe the composition and publication history of this reviewed corpus.

Publication patterns.

The 667 references include 546 first released since 2024 and 439 in the partial 2026 window. The concentration coincides with the recent expansion of generalist policies, multimodal teaching, and embodied agents. Figure 9 separates this complete reference set from the 415 method and comparison references; the latter includes supporting control mechanisms and parameter-adaptation comparisons.

Within the method subset, 163 references concern context-conditioned policies, 30 geometric transfer, 52 world-model-based control, and 170 skill- and agent-based execution. Branch-specific coding places ReCAP's Cosmos Policy branch under predictive world-action modeling [152], ENPIRE under program revision [89], and GPT-Policy under contextual tool use [94].

The recent concentration of publications and the distribution across control interfaces in Figure 9 motivate a closer comparison of how methods use context. The following chapter examines their execution intermediates and the assumptions that support transfer.

Takeaways

• Context resolves what the current observation and prior competence leave undetermined: the required task, relevant history, or physical response. Its value depends on whether that information changes a physically executable decision.

• Demonstration following and physical adaptation have distinct historical roots. Their convergence joins inference of a new task with inference of an unfamiliar body’s response.

3 Methods: How Does Context Change Action?

Context reaches action through an action distribution, motion reference, predicted consequence, or skill/tool specification. Each interface must preserve task-relevant distinctions until they change execution. We compare their representations, executor capabilities, and validity conditions (Table 2). Correspondence binds evidence, memory retains it, and external revision carries verified lessons into later attempts.

The factorizations use $C_t$, $h_t$, $A_t$, and fixed parameters $𝜃$ from Section 2, exposing the intermediate consumed by stochastic or deterministic execution. Table 1 collects the notation; additional symbols are defined locally.

A shared primitive vocabulary. Tables 3, 4, and 5 use one primitive taxonomy. Source codes demonstrations or examples (D), retained history (H), instructions or goals (I), explicit feedback (F), and scene or map evidence (E). Form codes visual observations or features (V), state/action records (S), geometry (G), and linguistic or symbolic records (L). Context operation separates nine operations: Retrieve selects stored records; Attend makes a query-dependent neural read; Align establishes spatial or temporal correspondence; Encode builds a learned representation; Estimate infers hidden physical response; Parse constructs symbolic bindings or procedures; Verify checks candidates against feedback or constraints; Simulate propagates candidate actions through a scene model; and Fit solves query-specific coefficients or geometry with network weights fixed. The tables code deployment evidence and its principal operations. Examples may be newly supplied or retained from training; the accompanying discussion specifies their role. Commas list evidence types; slashes combine operations or carriers without specifying their order.

The resulting carrier is an action output, motion reference, predicted future, learned skill, program/procedure, or tool/controller call. A carrier identifies what passes between components; the family identifies its role in the complete control path. Execute then distinguishes six operations: Replay returns recorded controls; Combine blends retrieved action continuations; Decode generates actions with a learned model; Track follows an explicit reference; Plan searches for an executable realization; and Invoke runs a program or calls a tool.

Evidence and execution roles. Language contributes according to the information it carries, independently of the control interface. In direct action inference, numerical states and action examples can be serialized for a language model, as in RoboPrompt [153]; ICRT learns its sensorimotor interface without linguistic data [13]. For a geometric archive, language can narrow the intended task before geometric matching resolves the motion, as in MT3 [36]. Command–program examples instead teach an executable mapping [27], while textual event memory preserves information missing from recent visual history [44]. These uses resolve task selection, action interpretation, and retention at different stages. The relevant comparison is what the text contributes to the decision, including whether it supplies a goal, a new convention, or a remembered outcome.

The separation between information and execution also applies to geometry. Matching can condition a learned action generator or supply a retargeted reference to a separate motion executor. Table B1 in Appendix B (p. 309) compares these roles. Geometry can instead supervise further learning: using hand–object motion to guide residual reinforcement learning trains a new policy [188]. The role of geometry becomes clear by identifying what it changes: a policy input, an executable reference, or the policy parameters.

Semantic abstraction preserves roles, prerequisites, and skill order while allowing grasps or paths to change [31, 165]. Execution must receive each binding constraint: plate-first teaching specifies order; a handle grasp additionally restricts contact. A generic command can erase either constraint; the analysis therefore follows how teaching reaches execution.

3.1 Context-conditioned policies

Context-conditioned policies map task evidence and robot history to actions. They either reuse recorded actions selected by correspondence or generate actions from an interpreted context representation. We compare these two forms before examining three complementary demands on the representation: execution phase, spatial correspondence, and physical response. The final comparison considers how pretrained competence affects the evidence each form needs.

For a context-conditioned policy, write $r_t$ for the representation of the evidence used at the current decision:

Here $E_{𝜃}^c$ interprets context and $q_{𝜃}^a$ generates actions from its representation $r_t$; both may belong to one network. The factorization requires $r_t$ to retain the context information used by action generation. Autoregressive, diffusion, and flow-based generators can implement this interface. If an example teaches red-then-blue packing, the representation must preserve that order while the generator adapts the reach-and-place motion to the current scene.

Reusing actions through correspondence. Let the nonempty set $J_t$ index the demonstration records selected for the current query, and let ${Ā}_j$ be the action block stored in record $j$. A nonparametric action-reuse policy has the form

The query-dependent weights ${\beta }_{\mathit{𝑡𝑗}}$ mix point masses ${\delta }_{{Ā}_j}$ at stored blocks. Nearest-neighbor replay gives one record weight one. Returning their mean, $A_t={\sum }_j{\beta }_{\mathit{𝑡𝑗}}{Ā}_j$, instead combines continuous controls deterministically, requiring compatible coordinates and separate treatment of discrete commands. All blocks must match the receiving robot’s interface. A learned generator in Equation (4) transforms retrieved evidence; replay directly returns its recorded actions.

Action reuse assumes that proximity in a representation implies compatible control. The workflow encodes the current scene, retrieves comparable demonstration states, and returns their recorded actions [149]. Its generalization rests on representation quality and archive coverage. A retrieved demonstration can instead condition a learned generator [17], which must infer how the example applies rather than return its action directly. This introduces a second source of generalization: transforming available experience beyond local replay.

Archive coverage and interpretation can fail separately. Under unseen MuJoCo embodiments, REGENT’s Retrieve-and-Play exceeds its frozen contextual transformer, while fine-tuning improves the learned agent [154]. A RICL retrieval study further distinguishes similarity from closed-loop utility [481]: on 25 tasks selected for prior RICL success, visual and VLA-based retrieval outperform random context, but their mutual differences are not statistically resolved. A useful retrieval metric must select evidence that remains actionable along the policy’s own rollout, beyond matching expert-trajectory states.

A retrieval baseline also tests how much competence resides in the representation. How Much of Your VLA Is kNN retrieves action chunks using embeddings from pretrained or fine-tuned VLAs and relates success to local agreement between neighboring observations and actions [582]. Strong retrieval performance shows that a learned embedding can expose reusable control neighborhoods. It does not establish that every generative policy behaves as a nearest-neighbor lookup: training the representation, covering the required actions, and selecting a compatible continuation remain separate contributions.

Local continuation bridges replay and learned generation. Behavior Predictive Control (BPC) reconstructs the live observation–action history from retrieved trajectory windows, then applies the same coefficients to their action continuations [389]. Unlike the nonnegative mixture in Equation (5), its regularized coefficients may be signed, permitting local extrapolation. The retrieval metric and residual correction are fitted offline; deployment solves the continuation coefficients from the current history while keeping those fitted components fixed. Reliable continuation depends on whether nearby records jointly explain the query with stable coefficients, connecting retrieval quality to local control compatibility.

The use of correspondence determines which component carries motion generalization. A retargeted human motion can serve as a proposal that a pretrained diffusion policy refines, as in DemoDiffusion [155]. The learned prior then contributes to the final motion. Alternatively, registration can directly transfer the next demonstrated end-effector pose, as in IMOP [156]. In that case, geometric applicability carries more of the transfer burden. Section 3.2 develops this second interface, where the matched evidence becomes an explicit reference for execution.

Retrieved motion can guide generation without becoming a prompt or a replay target. RAGu matches functional object roles, warps retrieved trajectories into the current scene, and applies their spatial constraints as an energy during action denoising [558]. Its online selection favors guidance compatible with the policy’s proposed action distribution. The base VLA remains fixed; geometric correspondence supplies a prior and the learned policy realizes the motion. This hybrid belongs to context-conditioned action generation because the warped trajectory steers, rather than directly replaces, the executor.

Figure 10a separates relevance, execution phase, and physical response. Panel b adds task specification: language identifies an operation and visual prompts bind its spatial target. These operations construct the evidence that enters the four control interfaces in Figure 12.

From task identity to execution correspondence. When retrieved actions do not transfer directly, the learner must interpret what the example specifies. Early demonstration-conditioned policies compressed examples into a task representation. Task-Embedded Control Networks jointly learned a demonstration metric and an embedding-conditioned policy [157]. Such a summary makes few-shot conditioning economical, but the policy must recover the relevant execution phase from its current observation. As variation extends from task identity to ordering, viewpoint, and object placement, identifying the task is no longer enough: the robot must relate what it sees now to a particular part of the example.

Table 3 traces context operations into action rules. Retrieval can return actions or condition a generator; alignment can condition inference or construct a tracked reference.

An attention-based interpreter implements a query-dependent read over demonstration elements. Let $D^s={(d_i^s)}_{i=1}^L$ be a support demonstration with $L$ elements. Its $i$th element $d_i^s$ may contain an image or a synchronized observation–action record; $k$ is a summation index over the same elements. One representative alignment is

The score $s_{𝜃}$ matches the robot history to an encoded demonstration element $e_{𝜃}(d_i^s)$. Normalized weights ${\alpha }_{\mathit{𝑡𝑖}}$ select evidence relevant at decision $t$, allowing the read to change with execution progress. This inference-time read supports image-only and action-bearing demonstrations alike; its training objective is a separate choice considered in Section 4.2. Richer attention or geometric representations can preserve several relationships without imposing a monotone frame match.

Execution-dependent attention addresses the loss of phase information in a single task summary: each action can consult the relevant spatial and temporal evidence [158]. Cross-domain contrastive supervision makes that read more reliable when demonstration and query differ visually [12]. Video cross-attention and causal sensorimotor sequences expose different evidence to the read, respectively visual progression and observations with recorded actions [13, 14]. Execution mismatch creates a further problem when frame similarity is misleading. RHyME constructs semantically compatible training videos through clip retrieval and composition, then uses learned alignment to condition its policy [80]. These designs address complementary parts of correspondence: when to read, what to compare, and how training makes the comparison meaningful.

Representation determines which taught distinctions action generation can recover. RGB leaves motion implicit; sensorimotor records expose actions, and keypoints or spatial graphs establish correspondence [15, 16]. Cross-domain latents couple human and robot movement, while hierarchical tokens separate persistent demonstration structure from finer decisions [159, 160]. Flow matching generates actions from encoded observations and commands [55]. TP-Flow uses support-demonstration prototypes to condition both its initial prior and velocity field [384]; deployment recomputes them with fixed weights. Richer generation helps only when its context preserves the required distinction.

Temporal selection addresses a different limit from action encoding. A complete trajectory may contain many locally irrelevant stages, so ICI-VLA retrieves phase-aligned micro-demonstrations using a retriever trained with dynamic-time-warping supervision [18]. The deployed policy reads examples selected from observable query inputs; target-action masking during training reduces reliance on exact action-prefix continuation. Phase selection determines which evidence reaches the generator, whereas token prediction or flow matching determines how that evidence becomes an action block.

Phase estimates can make this read explicit. VICTR trains a demonstration-conditioned progress estimator and uses it to select relevant demonstration chunks for an adapted VLA [560]. At deployment, the selected examples and current execution state change while the trained parameters stay fixed. Its progress supervision identifies where execution lies along a reference; the quantity is not a calibrated estimate of expected return. Comparing progress-based selection with visual retrieval tests whether the example is useful because it depicts the right stage, rather than merely a similar image.

Spatial correspondence as conditioning. Temporal phase identifies when an example applies; spatial correspondence identifies where its action should be realized. MatchingPolicy makes this second requirement explicit [161]. Foundation-model features establish demonstration–query point matches; tracking propagates them through time; RGB-D lifts them into 3D. A graph diffusion policy then generates gripper motion from these relationships. This workflow assumes that matched task geometry remains informative about the required action, without fixing a trajectory for the robot to replay. Section 3.2 considers the complementary case in which correspondence transforms an explicit executable reference.

Navigation extends this correspondence from object interactions to places. A policy can read successive visual subgoals [354] or attend to an action-bearing path memory [355] to follow new teaching. A scene preview supplies a different spatial reference: NOLO [350] combines a video with optical-flow-derived action labels, the current view, and a goal image to predict actions. Its offline reinforcement learning trains the interpreter; deployment reads a new scene video with weights fixed. The shared computational interface is contextual action inference, while the context can specify a route or reveal an environment.

Identifying physical context from interaction. Temporal and spatial correspondence can identify a desired motion while leaving its physical effect uncertain. Interaction history supplies the complementary evidence when the goal is known but the current body’s response is not. System identification makes this unknown explicit through physical parameters, whereas a learned adaptation latent can retain the response information needed by a policy [39, 146]. ICWM extends physical-context inference to changes in observation and embodiment [41]. These approaches require transitions that reveal how commands affect the present system; a task demonstration alone may leave that uncertainty unresolved. Their representation conditions an action policy directly. Predicting and evaluating candidate futures introduces the additional computational stage considered in Section 3.3.

Physical context must distinguish causes requiring different corrections. Self-Adaptive VLA uses command–response discrepancies for actuation bias, whereas joint-encoder offsets need visual evidence because reported and commanded poses can agree despite a physical error [391]. CAPABLE combines joint-wise command–response histories with the current Jacobian to condition a residual policy [492]. Its encoder and policy are trained offline, then fixed for held-out actuator and fault tests. History identifies bodily capability, rather than a new task.

Connecting pretrained competence to new evidence. Broader robot and language priors change what must be learned for demonstration following. An existing action generator can be conditioned on retrieved executions [17]; grounded spatial descriptions can supply scene information missing from the current visual input [162]. These inputs resolve different unknowns. A demonstration may change the requested procedure, while a scene description may clarify how to execute an already known procedure. The contribution of each channel depends on the decision its information resolves.

Observation-only teaching leaves action correspondence implicit. Behavioral Cloning from Observation estimates actions through inverse dynamics before training a policy [167]. Fixed-weight teaching uses an existing action interface learned from demonstrations or constructed through pretrained reasoning and geometry.

Textual action interfaces move part of action inference outside a visuomotor backbone. RoboPrompt supplies object poses and keyframe end-effector examples to a frozen language model [153]. BiCICLe conditions one arm’s keyposes on the other’s prediction, preserving coordination within a smaller inference problem [92]. Its v3 TWIN comparison uses ten demonstrations per task and reports 70.5% mean success over 13 simulated tasks, versus 64.4% for the strongest ICL comparator. Ground-truth segmentation supplies simulated object centroids; three physical tasks provide a separate execution test. These interfaces rely on perception and coordinate conventions that make reasoning outputs physically meaningful.

Teaching budget can change the preferred adaptation mechanism. KAT compares a text-pretrained model reading keypoint–action examples with KeyAct-DP, a diffusion policy trained on similar representations [15]. Contextual inference is competitive with few examples; policy training becomes stronger with more. The crossover differs against an image-based diffusion policy, indicating that both representation and adaptation mechanism determine the useful demonstration budget.

Generalist systems expose different teaching representations. BPP and GEN-1.5 accept sensorimotor prompts [54, 98]; S1 accepts video prompts [99]. The GLOW technical report [387] describes a Skill Demo Encoder that converts one human video into reusable skill context. A fixed-weight autoregressive model combines it with current observations, language, robot state, and history to generate end-effector actions. Its teaching pathway belongs to context-conditioned policies (Figures 5 and 7); Harness manages tools and feedback, while KnowinWorld supplies candidate rollouts. The primary family follows the teaching-to-action pathway, with prediction and tool management providing complementary support.

Retrieved actions need compatible coordinates and archive coverage; learned generation additionally depends on the interpreter transferring the taught distinction, as REGENT illustrates [154]. Geometric transfer exposes this dependency through an explicit reference that execution tracks.

Physical adaptation further separates inferred state from learned competence. Online latent adaptation fits an object’s hidden physical representation from transition prediction errors while retaining the learned dynamics and policy [563]. A deformable-object study compares explicit physical parameters with interaction-history representations: a better predictor does not necessarily provide better manipulation control [579]. The decisive test is whether the inferred context improves the action chosen under changed dynamics. GAP instead uses feedback from a completed piano execution to optimize corrections to the next trajectory; its simulated guidance prior can also be refined on hardware [576]. Action optimization and the additional prior-training stage therefore require separate accounting.

3.2 Geometric demonstration transfer

Geometric demonstration transfer preserves a motion or contact reference and adapts its realization to the query scene. Perception establishes correspondence; alignment or retargeting transforms the reference; tracking or replay executes it. This explicit reference is useful when contact geometry matters, such as preserving a handle grasp. We follow the mechanism from reference construction through spatial alignment and shape changes to reuse across a task repertoire.

IMOP makes the reference local to the next demonstrated action: learned invariant-region correspondences support analytic pose transfer, and a state router selects the relevant demonstration transition [156]. The reference can therefore be a target pose rather than a complete trajectory.

Reconstructing an executable reference. Let $D^s$ denote the supplied demonstration. A geometric transfer computation can be written as

The extractor $E_{𝜃}^m$ produces $L$ descriptors ${\zeta }_i$ of demonstrated hand poses, object-relative poses, or contacts. $\mathrm{Retarget}$ maps them to scene $o_t$ and robot kinematics/control constraints $\mathcal{B}$; executor $\kappa$ consumes the resulting motion reference $g$. Object-centric and hand-based transfer preserve different descriptors: object motion versus grasp and hand motion [35, 168]. Fitting these poses or trajectories can leave the extractor’s neural weights fixed; superscript $m$ denotes the motion interface.

For rigid object-relative transfer, this relationship has a particularly simple form:

Here ${}^W{T}_X\in \mathit{𝑆𝐸}(3)$ is frame $X$’s rigid pose in world $W$; $E$ is the end effector and $O_b$ the reference object for segment $b$. Superscripts $s,q$ denote source and query; $\sigma \in [0,1]$ is segment progress, and $-1$ denotes inversion. The source factors express object-relative motion; the query factor relocates it. A fixed reference and applicable relative motion are assumed. Shape change, moving references, or embodiment differences require Equation (7)’s richer correspondence, making reference selection part of task interpretation.

Table 4 traces four ways to preserve an interaction: align its visual reference, transform object-relative motion, match functional parts, or switch references across stages. A richer correspondence is useful when it preserves a constraint that a global pose transform would lose.

A transferred reference must preserve timing as well as geometry. MotionSkill represents named body points, root motion, and availability flags on a shared source clock before mapping them to each biped [633]. Its walking study trains a separate residual tracker for each body; a distinct timing study uses pretrained controllers and updates phase during execution. The comparison separates constructing a feasible reference from possessing a controller that can follow it. Retaining labels for imputed points also exposes where missing demonstration evidence enters the transferred motion.

From image alignment to object-relative transfer. Geometric reuse begins with an applicability assumption: restoring the relevant relative configuration makes the demonstrated interaction useful again. Image-based approaches enforce this through visual feedback. FlowControl aligns live RGB-D observations to successive masked reference frames [32]; DOME separates a learned visual servo to a bottleneck pose from subsequent velocity replay [33]. The difference is where feedback acts: throughout reference following or primarily before the interaction segment. Tracking active points and gripper events allows RoboTAP to change the visual reference as a task advances [169]. POIL closes the loop on the transferred object points themselves: functional-part correspondences establish a reference, and point-set feedback converts tracking errors into a rigid-body twist [403]. Using the same points for transfer and execution ties disturbance recovery to the demonstrated interaction.

An object-relative invariant makes applicability more explicit than image agreement. Pose-based transfer preserves a recorded end-effector trajectory [170]; category-level shape fitting extends correspondence to interaction keypoints [171]. DITTO reconstructs manipulated-object motion from an RGB-D human demonstration, transfers that trajectory to the detected objects in the robot scene, and realizes it through grasping and motion planning [35]. Hand-based retargeting instead reconstructs the demonstrator’s grasp and movement before converting them into robot configurations [168]. The embodiment gap thus appears in different places: realizing an object effect in DITTO, or translating a demonstrated hand configuration. In either case, successful transfer requires both a preserved interaction and a feasible realization.

Semantic anchors and functional geometry. Rigid pose transfer preserves a motion only when the source and target support the same local interaction. A different handle or articulated part requires correspondence within the object. SemAnCorr [38] selects semantically consistent surface anchors and propagates them through a functional map. Its manipulation pipeline transfers contact regions and local frames, selects a feasible grasp, and reuses motion relative to that grasp. Contact keyframes and relative motions form its execution specification. Semantic correspondence supplies the local geometry needed to extend the rigid transfer in Equation (8) across object shapes. Descriptor-based transfer makes the matched invariant explicit: NDF optimizes poses to reproduce demonstrated descriptors, L-NDF localizes the representation to transferable geometry, and IGA aligns object graphs conditioned on examples [420–422]. The optimized object is an alignment or pose, not the descriptor network's weights. A template library makes this correspondence explicit at retrieval time. RAGrasp combines semantic features with geometric filtering, then registers a selected RGB-D grasp template to the query and refines its contact pose [472]. Reuse avoids task-policy fitting but depends on calibrated depth and a transferable local grasp; physical tests cover calibrated planar overhead grasping.

Shape variation makes whole-object similarity too restrictive for some interactions. What must remain compatible may be a grasp role and a contact role, rather than the complete tool geometry [172]. A complementary approach starts from global shape and refines local curvature correspondences to preserve the contact used by a motion [173]. Surface interaction functions and screw interpolation connect such functional geometry to a transferred trajectory [174]. These routes differ in how they identify the invariant, but share a control assumption: preserving the relevant interaction structure makes the adapted motion useful. This shifts the transfer problem from matching objects as wholes to matching the parts of their geometry that the task actually uses.

When both objects change, correspondence must preserve their interaction. R-NDF aligns task-relevant local frames on a new object pair [345]; part decomposition separates local geometry from object arrangement to recover the demonstrated relationship [346]. Both reduce interference from irrelevant shape variation, extending transfer beyond relocating motion around one object (Section 3.5).

Deformable manipulation requires correspondence to remain meaningful as the object changes shape. Sparse Meets Dense extracts task-relevant contact pairs from a demonstration and uses an offline learned dense representation to track them during constrained execution [175]. Its final-state relative-pose annotation supplies interaction geometry; sparse constraints express the task, and dense correspondence maintains that geometry through deformation. This extends the reference from a rigid frame to a changing set of contact relations.

Affordance prediction can supply a related execution interface from a learned prior. VLAff predicts interaction heatmaps, grasp poses, and motion trajectories from an observation and language instruction [176]. Its human videos provide offline affordance supervision. These outputs can ground a contextual planner’s specification, while a demonstration-conditioned system additionally uses the newly supplied example to determine which interaction should transfer.

Cross-embodiment geometry can be learned before a new demonstration is supplied. CosmoH2G learns hand-to-gripper transfer from paired recordings, predicts endpoint poses and orientations, and refines the resulting trajectory through contact alignment, smoothness, and inverse kinematics [503]. Deployment optimizes a motion reference while keeping the learned mapping fixed. Its open-loop execution tests transfer of demonstrated geometry; correcting deviations would additionally require observation-driven control.

From one demonstration to a reusable repertoire. A geometric repertoire must select an applicable interaction and restore its starting configuration. DINOBot combines retrieval with wrist-view alignment before replay [34]. MT3 adds language filtering, geometric retrieval, and pose alignment [36]; its retrieval-versus-cloning comparison separates alignment from interaction. Its 1,000 single-interaction tasks require less than 24 hours of human demonstration; multi-stage tasks use one demonstration per stage. Adding references avoids task-policy retraining, while execution still depends on restoring each interaction’s preconditions.

Independent object motion and arm coordination expose the limit of one global alignment. A grasp and a later insertion must refer to different objects when the part and receptacle move independently; dual-arm interaction must additionally preserve relative coordination. Multi-stage visual servoing establishes a coordinated reference before ODIL’s replay [37]. Interaction segmentation with invariant-region matching, and VLM-based keyframe selection with repeated alignment, extend the reference-switching principle to longer demonstrations [93, 177]. Table 4 compares how individual systems bind and execute these references, while Table B1 in Appendix B contrasts the computational roles of correspondence. The key decision is when a change of interaction requires a different reference.

Geometric transfer exposes whether a matched interaction remains applicable after alignment. Correspondence and reference switching address shape variation and independent object motion; feasible tracking addresses physical realization. Predictive control instead represents the consequences execution should produce.

3.3 World-model-based control

World-model-based control predicts consequences to specify desired evolution or assess candidate actions. Learned predictors and physics simulators both support this interface when their predictions influence deployed control. The comparisons distinguish demonstration conditioning from prediction used only in training.

A representative factorization is

Here $v^{+}$ is a predicted observation or latent future. $p_{𝜃}^v$ predicts it and $q_{𝜃}^v$ decodes future-conditioned actions, with fixed neural parameters $𝜃$ in the ICL regime. Superscript $v$ denotes this predictive interface. Joint video–action generation may share a backbone [178]. For packing, the future can specify red-before-blue placement; actions must realize that sequence. The law of total probability marginalizes future-conditioned actions over predicted futures:

Discrete futures replace the integral with a sum. Sampling a future and then actions, or selecting one approximate future, implements this dependence. The future’s role in control defines the interface.

Generating a desired future or evaluating an action’s future. World-model-based control has two complementary forms. In Equations (9)–(10), context specifies a desired evolution and an action decoder realizes it. In model-based selection, candidate actions specify possible evolutions, and context determines which consequence is preferred. One planning form is

The set ${\mathcal{𝒜}}_t$ contains admissible candidate action blocks, ${\widehat{A}}_t$ denotes a candidate, and $A_t^{\star }$ is a selected maximizer. The action-conditioned dynamics distribution $p_{𝜃}^d$ estimates its future $v^{+}$; $d$ distinguishes forward dynamics from the context-conditioned future generator $p_{𝜃}^v$. The score $\mathcal{R}$ measures how that future satisfies the contextual task requirement. The expectation averages over predicted futures. This form makes the two roles explicit: the transition model predicts what an action may cause, whereas teaching determines which consequence is desirable. Optimizing ${\widehat{A}}_t$ leaves the neural parameters $𝜃$ unchanged. For explicit simulation, the predictive distribution can be induced by $s_{k+1}=F_{\varphi }(s_k,a_k)$ and $v_k=R(s_k)$, where $s_k$ is physical state, $F_{\varphi }$ a transition operator with scene parameters $\varphi$, and $R$ an observation renderer. Estimating $\varphi$ instantiates the scene without updating the planner's neural weights. SIMPACT constructs this simulator from RGB-D and returns rollout images and object states to a VLM for candidate revision [423]. Thus, simulated consequences become planning context; their value depends on whether reconstructed geometry and dynamics preserve the interactions relevant to the task.

Prediction and action support constrain different parts of search. LeWAM retrieves a future goal from training demonstrations and searches a frozen policy’s noise space with fixed latent dynamics [512]. Demonstrated progression grounds the objective; policy decoding restricts candidates in Equation (11). Its small-data simulation study evaluates this use of trained capabilities.

Candidate construction and goal scoring can be improved independently. LG-DKGP uses executed-action history to propose temporally coherent continuations before a frozen world model evaluates them [658]. Time-to-Goal instead learns a temporal-distance metric over frozen predictive features, so that visual similarity need not stand in for remaining effort [644]. Action-affine world models locally predict how corrections to a fixed policy’s proposal change task state, enabling quadratic-program refinement [628]. These interfaces alter the proposal distribution, the objective, or the local action search in Equation (11). Learned priors and metrics are established before deployment; optimizing a candidate action does not itself retrain the policy.

Unknown dynamics changes which candidate trajectories are credible. One response is to infer dynamics context from recent transitions and evaluate actions with a context-conditioned world model, as in DOMINO [19]. Another is to condition trajectory generation on those transitions and use reward and dynamics guidance during sampling, as in MetaDiffuser [20]. Both specialize prediction through new evidence while leaving learned parameters fixed. Context can also specify how actions are interpreted: SyncWorld uses paired action–video calibration segments to infer the current camera- and embodiment-dependent mapping from numerical commands to visual motion [506]. Coordinate augmentation and contextual training establish this capability offline; deployment supplies the new mapping and ranks candidate actions through imagined rollouts. Thus context can identify a control convention as well as a desired outcome.

Calibration records can become reusable control knowledge. Library of Hands infers how commands change images, stores the resulting action mapping, and reuses it when a setup recurs while keeping the world model fixed [589]. Transition examples can also condition a generic tabular foundation model to predict dynamics without task-specific fitting [569]. That study reports a substantial inference-time cost and degrading long-horizon rollouts. Thus identifying a local transition law from context and supplying a practical closed-loop planner are distinct achievements.

Prediction errors can determine which physical evidence remains useful. AFTER-WM combines experts with fast and slow correction memories, using observed transition errors to revise their contributions [624]. Fast updates respond to a changed condition; slower retention stabilizes recurring dynamics. Its dynamics benchmarks and flight-fault records support adaptation of prediction, with closed-loop benefit tested separately. This complements storing a reusable command mapping: recent evidence can change either the inferred interface or the correction applied to an otherwise learned predictor.

Inference and fitting change different parts of prediction. DeepJEPA recurrently refines imagined transitions using an offline-trained continuation head, with world-model weights and outer CEM settings fixed [448]. MetaPusher instead updates a pushing model and repairs its planning tree [367]; Sandwich-Residuals fits neural corrections around a frozen predictor [366]. These operations refine computation or change the predictor; contextual prediction specializes an unchanged model through new evidence.

Freezing the base predictor alone does not make the complete adapted model parameter-fixed. Few-shot residual adaptation learns correction modules around a pretrained latent predictor from a small number of target episodes [555]. The new modules change its effective dynamics. This supplies a useful comparison to transition-conditioned prediction: both exploit recent evidence, but they retain that evidence in different objects and incur different reset and fitting costs.

From goal-reaching to demonstration-conditioned futures. Goal-reaching and demonstration transfer place different demands on a predictor. Universal Planning Networks optimize actions toward a supplied goal with model weights unchanged [179]; ViGAR turns a goal image into subgoals for joint video–action execution [460]. The image specifies an outcome. PEWAM instead compiles a demonstration into object-keypoint trajectories, contact points, and a terminal state, then conditions a fixed world-action model on this effect program [459]. Removing the demonstrator’s body makes the desired object evolution an editable interface. Its four-task physical test reports 36/40 successes against 22/40 for goal-image conditioning after training on other real tasks. Yet terminal-state-only conditioning exceeds the full program on its small held-out LIBERO comparison (49/90 versus 40/90). Abstraction follows the taught constraint: placement may need only an endpoint, while ordered effects and contacts preserve additional requirements.

This inference can remain in a latent space. OSVI-WM decodes demonstration-conditioned latent states into waypoints [21]; Demo-JEPA supplies a predicted latent goal to a cross-entropy-method planner until an achievement threshold permits progression [22]. Prediction supplies a separate executor’s target. ModAR tests the predicted representation itself: sequential point-track, feature, and depth prediction supports action generation, while adding RGB offers no consistent gain [531]. Human video supplies training data, and deployment conditions on observations and a task label. This comparison separates useful predictive content from learning a procedure through a new example. In Demo-JEPA’s real-robot comparison, an action head performs better on familiar behaviors and planning on unseen tasks; the familiar-task advantage disappears in simulation. The value of search versus decoding therefore depends on task novelty and execution setting, even with a shared future representation.

Prediction can also supply a route or a discrete transition model. TADreamer reconstructs navigation waypoints from generated video and registers them to measured geometry before planning [334]; metric calibration connects imagined motion to feasible execution. GAVEL instead rolls plans through an explicit graph of action preconditions, effects, and location beliefs, repairing model-detectable violations before invoking semantic replanning [335]. GAVEL primarily tests modeled action consistency through a symbolic BEHAVIOR-1K executor; TADreamer connects imagined motion to metric execution. Prediction can therefore support different levels of action verification.

Human-video world-action systems connect demonstrated procedure to robot-domain evolution. Progress selects the relevant segment [23]; an example must also change the predicted future when alternatives remain feasible [24]. SimpleICL’s learned demonstration queries condition video and action diffusion; removing video conditioning or future prediction impairs held-out-task execution [517]. WATCH first pairs human demonstrations of the same task across scenes, then post-trains on robot data [536]. Reference video and actions condition joint action–future prediction, with reference-conditioned and context-free control evaluated separately. Cross-human pairing teaches transferable interactions; downstream improvement in both branches means pretraining gains alone do not establish deployment-time example dependence.

Procedure and physical history have different roles. WorldScape Policy 2.0 combines persistent demonstration context, event memory, and short-term dynamics [25]. Long-WAM instead studies the robot’s own observation history: causal video pretraining supplies a history-to-future representation, and partially denoised futures condition action generation [483]. Its context-window variants are trained separately. Their gains therefore concern learned history use, rather than adding teaching to one fixed checkpoint. Stable procedure tracking requires appropriate training and timely access to physical history.

Temporal factorization addresses the resulting imbalance. HiLo-WAM separates sparse long-horizon task evolution from dense local dynamics [599]; LaST0.5 couples interaction-centered latent reasoning to future and action generation, using consistency and progress signals to adjust execution and resampling [565]. These mechanisms concern how trained predictors use the robot’s unfolding history. A separate intervention on a supplied example is needed to measure acquisition of a newly taught procedure. In candidate-based control, Tiresias makes the selection path explicit: proposed actions generate imagined outcomes, whose ranking determines the command [587]. Prediction quality matters through this downstream choice.

Separating planning and control also determines what becomes memory. MaP-WAM generates a language subgoal and visual plan from sparse observed histories; its executor reuses that plan until observation-to-frame matching indicates progress and triggers replanning [399]. Actual execution frames then replace imagined ones in episodic memory, keeping predictions distinct from observed events. The executor’s auxiliary future-video head can be removed, while the upstream plan remains. RMBench uses a multitask planner with task-specific executors; constant executor latency excludes planning. Memory coverage, motor transfer, and response time consequently require separate measurements.

Retrieval can constrain prediction before detailed motion is generated. The retrieval-augmented policy ReCAP supplies trajectories to a frozen residual predictor [152]; its Cosmos Policy branch jointly generates future images and actions. The retrieved motion narrows the continuation, while prediction relates it to the evolving scene. Other ReCAP backbones route retrieved evidence through different action interfaces, so the predictive classification applies to its Cosmos Policy branch.

The predictive interface also localizes transfer. OpenWAM freezes a local-context inverse-dynamics decoder while training the target-task video producer with action supervision [496]. On four held-out LIBERO-90 tasks, both counterfactual dynamics coverage and restricting the decoder’s inputs contribute to transfer. The reusable component is the action decoder, while task adaptation remains in the producer. Across pixel, latent, and action-conditioned predictors [180], the central question is which component receives the new task information and whether its output preserves an executable interaction.

Prediction need not be invoked at every step. Bind-WAM learns a language–object binding signal that routes between action-only inference and joint future–action generation [629]. OSA instead evaluates a frozen ACT policy’s proposed chunk in a cloned PushT simulator and invokes beam search when predicted coverage deteriorates [647]. The former learns a selective predictive interface; the latter relies on simulator state and an explicit search budget. Their gains should be attributed to the information available to the gate, the selected extra computation, and the action ultimately executed. Binding uncertainty and a simulated coverage decrease are different decision signals, neither a universal estimate of failure probability.

Learning from futures without executing through them. Deployment can retain predictive competence after removing future generation. DreamFree distills future-model representations into an observation encoder and uses the resulting features for action generation without an online imagination rollout [585]. The prediction contributes training supervision, whereas the deployed action interface is reactive. This distinction complements the NowWAM and Simple-WAM comparisons below: a world-model training origin does not by itself imply predictive search during execution.

Predicted consequences can also improve the training signal directly. WM-PPO values candidate successors to construct auxiliary policy advantages and critic targets, weighting their influence by observed reliability [618]. Its main ALFWorld and ScienceWorld evaluations execute the learned policy without online lookahead. World-model-guided human-like navigation similarly converts social and semantic predictions into training preferences and distills them into a fast navigation policy [642]. These studies locate predictive benefit in how an executor is learned. Runtime contextual planning instead evaluates new consequences to select the current action.

Future supervision can improve an action representation without requiring a future to be generated online. Let ${\mathcal{L}}_a$ be an action-training loss and ${\mathcal{L}}_v$ an auxiliary future-prediction loss. The training and deployment interfaces can then be written as

Here $\lambda \ge 0$ weights the auxiliary loss. Optimization acts on the designated trainable components, with any frozen backbone held fixed. The retained components of ${𝜃}^{\star }$ initialize the deployed parameters ${𝜃}_0$; the auxiliary prediction head can be removed when action generation does not depend on its output. Equation (10), by comparison, retains a future-dependent action computation at deployment.

NowWAM learns actions while denoising the current observation, then acts on a clean observation in one pass [516]. On LIBERO-Plus, past and future auxiliary targets give similar results. Denoising across noise levels during training improves robustness over training only on clean inputs, even though the deployed policy never generates a future frame. Simple-WAM probes a different dependency. It reads pure-noise tokens at future positions in one pass, and removing them reduces transfer [524]. Videos from held-out tasks provide training supervision in that study. Testing deployment-time teaching would require varying the demonstration given to the same trained policy.

Predictive pretraining and contextual adaptation solve different problems. ZimaBlue aligns causal video priors with robot actions through video–action mid-training and target-robot post-training, then caches slow-model futures for fast execution [499]. Physical-demonstration prompting is a proposed next step in that work. Its trained generalization establishes a prior on which contextual teaching could operate.

Training intermediates need not remain execution dependencies. Gripper traces supervise action representations even when reasoning dropout or head removal permits execution without generating them [181, 182]. PonderPounce instead retains an asynchronous cognition token within action inference [183]. DeltaWAM uses future-delta supervision but deploys an action stream that reads cached observed changes [414]. The cache reduces scene-encoding cost, while periodic refreshes restore a recent scene anchor. The runtime interface is history-conditioned action inference, not control through a generated future.

Figure 11 contrasts these roles. Auxiliary future targets shape a policy during training; prediction-conditioned control retains a generated future in action computation. Panel (a) shows the trainable context and state–context transformers after a frozen visual backbone. Panel (b) shows trainable prediction branches, not input frames. The latter couples control quality directly to the generated future.

Predictive control makes an anticipated consequence available for inspection or planning. Forecast accuracy and action decoding determine whether it reaches execution. Skill- and agent-based methods instead specify operations and delegate their realization to available skills or tools.

3.4 Skill- and agent-based execution

Skill- and agent-based execution uses context to specify the selection, order, and arguments of available skills or robot tools. A red-before-blue placement video can become an ordered procedure realized by an existing executor. We compare learned skill composition with VLM/LLM-generated programs and calls, then examine when feedback can revise the specification and when additional motor learning is needed.

The specification can take the form of code, spatial targets, or an ordered sequence of learned skills. Its defining role is to govern an execution stage that can track or advance it, as described in Section 2. At that action interface, the deterministic form is

where $f_{𝜃}$ extracts the execution specification $g$ from context $C_t$ and robot history $h_t$, and $\kappa$ realizes it as the next action block $A_t$ using observation $o_t$. For stochastic specification generation and execution, the same interface becomes

Here $p_{𝜃}^g$ generates specifications and $k_{𝜃}$ realizes them. Discrete specifications give a sum; deterministic components recover Equation (13). Operations, arguments, and composition reside in $g$, while the executor’s neural parameters reside in $𝜃$. Code or argument edits can therefore change behavior with weights fixed. Programs can invoke geometric trackers, and predicted futures can supply specifications, composing the corresponding interfaces.

Skill composition and program execution instantiate this interface as

Here $z_{\ell }$ specifies a skill, $N_g$ counts skills, and ${\ell }_t$ selects the active one. $\mathrm{Run}$ executes program $g$ until it yields an action block, including permitted perception and control calls. A tool agent can generate one request at a time: its name and arguments form $g$, the robot adapter implements $\kappa$, and returned observations inform the next decision.

Composing reusable skills. Rearranging existing motor capabilities can express a new task. Neural Task Programming (2017) instantiated a hierarchy from a demonstration, with leaves invoking robot routines [26]. Task length, ordering, and object assignments could change while physical skills were reused.

Skill composition may itself be acquired through parameter updates. PRISM preserves a bank of primitive adapters while training the components that route and combine them for composite tasks [551]. Fixed primitive weights preserve a reusable basis; they do not imply that the entire composite controller learned the task only from deployment context. HITCH similarly teaches a vision–language model to operate grounded whole-body APIs before evaluating instruction-driven composition [571]. These systems locate two prerequisites for contextual composition: an adequate skill repertoire and an interpreter trained to invoke it.

Human-video teaching adds recognition across bodies and timing. XSkill learns shared human–robot skill prototypes [29]; UniSkill learns from unaligned cross-embodiment video [30]. Both translate human examples into guidance for a robot-side skill-conditioned policy.

Waypoints preserve motion targets, object-relation graphs preserve dependencies, and staged procedures preserve interaction changes [93, 164, 165]. Their role depends on the executor: they may specify a separate controller or condition action inference internally, as semantic subtasks do in StellaVLA [163].

Skill composition requires compatible handoffs. SkillWeave scores a frozen predecessor’s sampled chunks against starting states in the successor’s training demonstrations [511]. Demonstrations thus constrain transitions as well as individual behaviors. Completion checks determine when to advance: StageGuard’s distilled monitor continues, advances, or skips skills [336]; Navi-Agent checks subgoals against visual-place topology [337]; RoboMonitor learns phase, failure, and completion from instruction-conditioned visual history [413]. Recovery depends on how the controller uses these checks.

Execution also requires selecting a controller. RoboAware trains a coordinator on modular and end-to-end branches from identical restored simulator states [550]. At deployment, fixed coordinator and coding-agent weights map observations, history, and skill stage to a family and executable code. Privileged judgments and state restoration are training-only; testing varies seeds within the training task pool.

Navigation separates example interpretation from motion realization. Instruction–landmark pairs in LM-Nav [358] and instruction–subtask pairs in A${}^2$Nav [359] guide parsers over pretrained navigators. Select2Plan retrieves visual examples to select marked movements [351]; VLM-GroNav uses terrain feedback to revise their costs [352]. Section 6.2 compares what these contexts specify.

VLM and LLM agents: programs, tools, and feedback. The execution interface determines what a general model must infer. Geometric abstraction exposes typed object-part references, rebinds their metric values from RGB-D, and returns action residuals as compact checkpoints [600]. ActOR instead combines ambient scene observations with grounded clinical-workflow skills [584]. In each case, reusable task structure and updated physical state meet through an explicit interface. Computer-use embodied agents make the same separation visible by presenting robot observations and controls through a software interface usable by an existing agent [611]. Transfer then depends on the information and operations exposed to the agent, as well as on its prior reasoning ability.

An executable plan must preserve dependencies as well as name appropriate skills. BiPOP derives partial-order bimanual plans from a symbolic task description; resource-compatible parallel actions still require motion-feasibility checks [623]. Execution-aware bimanual planning grounds blocking, approach accessibility, and reachability in a scene graph, then uses hierarchical validation to repair failed stages on a physical robot [657]. NSDL assigns scene-dependent goal judgments to a trained VLM and dependency expansion and scheduling to a deterministic solver [666]. Its symbolic replay tests the planning interface. Together these cases distinguish perceiving the correct goal, constructing a valid dependency structure, and realizing that structure through collision-free physical motion.

Demonstrations can guide program acquisition before deployment. ENCORE supplies keyframes, gripper events, and contact evidence to a fixed coding agent, then freezes the program for sealed evaluation with perception APIs [534]. Matched model, API, and development-budget controls attribute benefits to demonstration-guided strategy discovery, especially early search and underspecified tasks. Teaching shapes the acquired controller; execution need not revisit the demonstration. Code as Policies (2022) uses command–program examples to recombine perception, control, arithmetic, and feedback logic [27]. Examples also specify available objects and interfaces [28]. Video teaching must preserve the constraints these interfaces can execute. Show-Harness passes textual video outlines to an unchanged planner and embodiment-specific action interpreter [31]. In FoldingAgent, manually selected video keyframes guide a fixed VLM to reconstruct and revise parametric folding programs through simulator feedback [501]. Its Pureland study evaluates geometric reconstruction; grasping, force control, and material response lie beyond its interface.

Contact-sensitive teaching requires a richer executable abstraction. RAPID derives trajectory-optimization primitives, relational constraints, and a verification environment from a human demonstration [388]. Predicates and simulated scene variants guide program revision before transfer; reconstruction fidelity determines what interactions these checks capture. Its nonprehensile experiments use demonstrations, whereas LIBERO-Pro constructs programs without them. Both require reconstruction and program search.

Context can specify commands, correct policy proposals, or revise programs. GPT-Policy supplies Cartesian targets and gripper commands through interpolation, inverse kinematics, and timing [94]; RoboDawn grounds discrete motions through examples, supplying a command primer even without demonstrations [364]. Galbot’s GPT-as-Policy compares direct commands with learned-policy corrections: both retain feedback and notes with fixed inference weights, while the hybrid also requires controller training [520]. AGP instead revises programs from physical outcomes and retains scripts and notes under fixed weights [513]. RoboICL’s observation–action–receipt records link teaching to executed effects [438].

Tool-based execution also needs embodiment context. iAm.md grounds a fixed coding agent in URDF, live ROS 2 interfaces, and deployment configuration [510]. Code and scene records change during adaptation. Interface knowledge grounds commands; physical observations establish their effects.

Tools supply the measurements and behaviors used by these programs. Robo-Harness K1 exposes calibrated depth, spatial measurements, persistent visual anchors, and grasp hypotheses to ground generic motion requests [377]. The agentic framework RACaP exposes typed Policy APIs: a ReAct agent combines working memory, retrieved experience, and visual feedback to choose calls, arguments, and recovery steps [378]. Both retain fixed neural weights during deployment; RACaP also freezes its generated source code. MOKA grounds motion through marked visual affordances, while RoCo coordinates waypoint plans through dialogue and collision feedback [424, 425]. Their prompting interfaces revise executable choices; MOKA's separate distillation branch instead trains a policy.

The memory–executor interface limits which requirements reach control. In 2AM, task memory supplies subtask language and optional 2D grasp, place, and move hints to an episodically stateless action model [401]. A grasp cue expires after acquisition, while a destination can persist during transport. Training prepares the executor for noisy or incomplete hints; memory must encode its requirements in these supported formats.

Previewing a candidate permits revision before execution. WAA renders proposed poses in interaction-centered views, checks kinematic feasibility, and converts in-view corrections into motion [386]. A separate Skill Agent supplies demonstration-derived procedures and visual checks. The main agent thus combines procedural guidance with geometric verification of its tool requests.

Feedback must update a request's supporting evidence. ETA observes after world-changing calls [348]; offline-trained EmbodiedSkills checks prerequisites and artifact freshness before execution, verifies effects afterward, and uses new observations to recover or revise plans [502]. Recovery restores operating conditions: RoboHarness retrieves entry states [349], while Causeway steers action features toward demonstration-derived re-entry poses [412]. Verification separates request defects from failures of the starting conditions.

Spatial context grounds these requests. VoxPoser builds affordance and constraint maps for planning [184]. HomeBody's digital twin grounds skill, target, and hand selection, with retries or replanning after failure [415]. Analytical manipulation and pretrained locomotion realize the chosen requests.

Applicability prediction, outcome observation, and probing supply complementary evidence. Skill-affordance estimates couple linguistic relevance to feasibility before execution [185]; observed effects revise the next step afterward [186]. When feasibility depends on unknown mass or stiffness, proprioceptive measurements can change the strategy [187]. Each operation resolves a different uncertainty about whether a skill will work.

Feedback-driven revision can change a procedure or its executor. ICPI adjusts primitive arguments [9]; CaP-X contrasts code revision with training the coding policy [166]. ENPIRE supports both program revision and neural policy learning [89], while VoxPoser’s dynamics-learning extension updates a predictive component [184]. PP-ProPS searches controller parameters from encoded candidate–reward histories [411]: its language model stays fixed, but each proposal changes the robot-side controller. The adaptation claim therefore depends on which component changes, not whether the reasoning model is frozen.

Explicit structure trades portability against physical detail: a procedure can transfer across bodies yet omit a required grasp or compliance condition. Executors may recompute only details left open by teaching. DualManip separates update rates accordingly: geometric correspondence revises contacts, while failed geometric validation triggers semantic replanning [408].

Table 5 connects context operations to execution: a video can supply a future for planning or decoding, a tracked motion, or an invoked program. Retrieval and verification augment these routes; their carriers determine which taught constraints reach control.

Missing evidence can itself become a planning obligation. VA-TAMP combines persistent object records with task-centric perception and inserts observation actions when the required relation is not currently visible [635]. Reobserving an occluded object refreshes the state used by task and motion planning, rather than merely enlarging a generic scene description. Its nominal-scene assumptions also define the search problem: restoring visibility of a represented object differs from discovering an unmodeled object or location. The interface should expose which missing fact prevents the next action.

Choosing an interface by the constraint it preserves. Interface choice follows the constraint most vulnerable to loss. Action inference relies on training relationships; geometric transfer exposes contact or motion; predictive control exposes consequences; skill interfaces expose order and preconditions. Hybrids can turn predicted futures into waypoints or invoke geometric trackers from programs. Classification follows the intermediate carrying the taught requirement into execution.

3.5 Shared mechanisms: correspondence and memory

Across all four interfaces, correspondence binds teaching to current objects, memory retains evidence, and the command interface determines which corrections can reach execution.

Figure 12 shows their control intermediates: demonstration features, contact frames, predicted futures, and skill sequences. Each depends on the evidence selected and interpreted in Figure 10.

Cross-object transfer of demonstrated tasks. Pouring can replace the jug, receiver, or both, within categories or through functional substitution (Table 6). Receiving roles include supports and contact surfaces. Transfer preserves the demonstrated relation under new geometry; teaching and training novelty differ.

The robot must still direct liquid into the receiver. A bottle changes grasp and tilt; a bowl changes pouring position and height. Replacing both couples the required geometric adjustments.

From object identity to task-relevant relations. Object replacement can change grasp, contact, and clearance while preserving role. R-NDF transfers relations to unseen object pairs using five to ten demonstrations, pretrained category representations, and a marked 3D keypoint [345]. Descriptor alignment keeps those representations fixed; its optional energy-based refinement trains another component. Part-based transfer uses one demonstration, learned part models, and segmentation assistance for new mug–rack and bowl–mug pairs [346]. Adapting by Analogy instead uses expert-selected correspondence to substitute an in-distribution observation into a policy [416]. Correspondence can therefore supply a motion reference or a policy input, with different prior geometry and human assistance.

From new instances to functional substitutes. Functional substitution preserves role across categories. FUNCTO derives grasp and use keypoints from actionless human RGB-D video and a task description, emphasizing tool changes over receiver changes [172]. SemAnCorr transfers kettle pouring to a toy cup and scissors cutting to a nail clipper using kinesthetic contacts and motion [38]. Their evidence formats differ, although both establish functional correspondence. Provided part equivalencies can instead isolate geometric alignment, as in the part-decomposition study’s watering-can pre-pouring test [346].

Permissible substitution depends on teaching: a content-transfer goal may allow another vessel, whereas a specified handle grasp restricts it. Preserving purpose may require a new procedure, as the Imitator Game’s affordance-adapted imitation illustrates [101]. Section 7.1 tests whether demonstrations determine the invariant: relation, contact, process, or effect.

Realizing a transferred requirement. Each interface gives the transferred requirement a different executable form: action features, a geometric reference, a desired future, or bound skill arguments. Geometry used during training [189, 190] differs from a deployment reference or demonstration-fitted motion system [191]. Spatial affordances and action representations supply priors [192, 193]; the new demonstration must select which requirement they realize.

Relational skill guidance provides an agent-based route to transfer. WAA’s frozen LIBERO-derived skills transfer to robosuite lifting, stacking, and restacking, where objects, scenes, and viewpoints change; grounding resolves poses from the current scene [386].

When interpretation produces a desired pose or feature configuration, a separate controller can track it. For a stationary target, a local visual-servo rule is

Here ${𝐬}_t$ and ${𝐬}_t^{\star }$ are current and desired visual features, ${𝐞}_t$ their error, ${𝐪}_t$ the joint configuration, and $J_{\mathrm{vis},t}$ the feature Jacobian. Its pseudoinverse and positive gain $k_{\mathrm{vs}}$ map feature error to joint velocity [194]. Context changes the reference while feedback corrects its realization. New contact or dynamics can still invalidate that realization, even with correct correspondence. The final part of this subsection examines this execution boundary. Transfer also requires the defining evidence to remain accessible when needed, motivating the memory mechanisms below.

Accessing and updating evidence across time. Alignment selects information within a demonstration, retrieval selects records from an archive, and memory preserves evidence across decisions. These operations can expose policy examples, motion references, predicted rollouts, or programs. The retrieved object’s use in control determines its method family; the access mechanism itself does not.

Memory must identify what matters, how to retrieve it, and when it expires. Task requirements, response estimates, and completion records need different update rules (Table 7). Moving a part can invalidate its pose while preserving the taught order. The read–update decomposition connects selection and reset timing to control; Sections 5 and 8.4 develop validity across execution and reuse.

Carrying evidence between decisions. A recurrent state $m_t$ and an external archive $M_t$ provide complementary ways of retaining experience. Let $x_t=(o_t,a_t,o_{t+1})$ be an executed transition, augmented with feedback when available. A common read–update decomposition is

The maps $u_{𝜃}$ and $w_{𝜃}$ update recurrent and external memory, ${\rho }_{𝜃}$ retrieves records $R_t$, and $b_{𝜃}$ constructs the policy input. Task evidence $C_t^{\mathrm{task}}$ includes instructions, goals, examples, and corrections; demonstration-only configurations use $D^s$. Retained state $(m_t,M_t)$ differs from selected input $C_t$. Either memory component may be omitted, and the maps can combine analytic and fixed neural operations.

Retaining the information that changes control. Early latent task inference compressed transitions to identify reward or dynamics [196, 197]. Selective retention matters because a demonstrator’s collection style can correlate with task identity in training logs without identifying the task reliably at deployment.

Physical adaptation depends on evidence age and validity. RMA uses recent transitions, ICWM collects task-agnostic probes, and LocoFormer carries interaction across trials [39–41]. RopeFormer retains action–response context across physical resets in dynamic rope tasks [363]. Zeva and Zeva-Ego encode action effects at two timescales; the latter combines recent traces with phase-matched memory across attempts on the same task instance [88, 362]. These fixed-weight interfaces trade acquisition cost against subsequent relevance.

Telemetry can reveal changed hardware response before action history does. TeAR gates bounded multiplicative and additive command corrections using robot state and motor measurements [458]. Offline training fixes both policy and adapter before deployment; context identifies hardware conditions. Controlled degradation tests response compensation, while physical Lift trials report thermal gains of two or three successes across 20 trials per condition.

Additional modalities resolve different hidden variables. BAT-WAM connects action and tactile codes to visual features and supplies the resulting representation to a trained dexterous policy [645]. Its conditional tactile forecasting uses target-time visual information, distinguishing cross-modal completion from predicting a wholly unobserved future. Be My Eye supplies a partner robot’s views, goal, and phase through learned context tokens [648]. The additional evidence concerns either contact unavailable to vision or a scene unavailable to one viewpoint. Both prepare an interpretation interface through training and vary the observations supplied during execution.

Event memory preserves facts no longer visible, such as whether a button was pressed. Perceptual retrieval and semantic records retain this evidence at different levels of detail [42, 43]; MEM combines recent views with a semantic history [44]. Recurrent updates [198, 199] and event-triggered keyframes [200] trade compression against direct access to earlier observations. MemBodied trains recurrent associative state alongside an initial-scene anchor, pairing each action with its ensuing observation [379], whereas ARMS records which arm acted and when [385]. HIRE’s ordered wrench traces distinguish visually aliased contact states; shuffling them tests the contribution of temporal order [402].

Environment facts and process facts need different write conditions. RoboCasa-Mem distinguishes remembered object attributes or relations from records of completed steps and pending operations [552]. Its accompanying memory hierarchy separates these roles before selecting a subgoal. Behavior-conditioned targets make the distinction operational: an earlier robot choice determines a later correct target. Updating a scene description alone cannot substitute for preserving which choice was actually made.

Retention determines what remains recoverable. Divide-and-Remember recursively selects $K$ visual tokens from merged $2K$-token blocks [444]; ECoMEM combines a fixed Writer of entity, state, and event records with a trained Reader [445]. Selection preserves visual detail selectively; records permit targeted updates within the Writer’s vocabulary. Both action interfaces are trained offline and fixed at deployment, with task-trained physical studies using different protocols.

Memory carriers expose different updates. ReCAT separates current and recurrent reads; DRAM learns associative state with a frozen backbone [417, 418]. RecastVLA updates shared fast weights between actions [419]. TaskAnchor’s CA-TTT rebuilds fast weights from history through a gradient-form update at each decision, then discards them after readout [507]. Its history persists within the episode and slow weights stay fixed, distinguishing transient computation from persistent adaptation.

Progress and historical detail resolve different ambiguities. TaskAnchor and its VLA undergo joint post-training on the target tasks [507]. At execution, a residual adds historical cues to the visual tokens and a milestone-supervised stage coordinate enters the language input. The VLA can thus act on an earlier human cue without a separate planner. PMTRM instead re-encodes a bounded state–action queue [508]. Mimir advances after verified postconditions, while OCC4M updates target location from support geometry [392, 393]. Dual-Stream Memory separates compressed progress from goal-selected historical frames [540]; its offline-trained interface resets between episodes. Stage and history jointly determine the next action.

A memory read can preserve spatial evidence and its computational representation separately. STVLA fuses spatial features with a bounded visual history and learns relevance-weighted retrieval for action prediction [621]. HeMeR instead tracks declarative, procedural, and working-memory chunks so that unchanged content can reuse compatible KV states when context is edited [619]. Its skill adapters are learned beforehand; online reconciliation updates the inference state. Selecting the right past observation determines what the controller knows, while cache compatibility determines the cost of processing that knowledge. These contributions require distinct accuracy and latency comparisons.

SimpleMemVLA routes timestamped history through subtask states to actions; MemoryVLA++ adds imagined states to retrieved events [45, 209]. TacitVLA separates prior subtask records from current-call reasoning: the action expert reads the historical ledger but cannot attend to newly generated reasoning, which is written asynchronously for later calls [537]. Reasoning supervision still trains its backbone. Thus removing current-call text generation need not remove historical context. $\mu$VLA transfers poorly when required memory structure changes [199]; retaining events and inferring unfamiliar rules remain different demands.

Workspace Models train latent workspace tokens using VLM-selected historical patches [395]. This amortizes online selection while inheriting the saliency target’s information choices. Comparing retrieval with learned compression requires both memory cost and recovery of newly relevant evidence.

Observation order alone cannot specify elapsed duration. TFP evolves recurrent memory over measured physical intervals before conditioning an action policy [539]. Scaling those intervals with weights and execution schedule fixed advances or delays actions. This intervention tests physically timed memory: the same visible state may require waiting or acting according to when an earlier event occurred.

History can also reconstruct an unobserved part of the present, rather than a past event or future consequence. LIFD combines recurrent evidence with geometry-anchored scene-token completion before action prediction [338]. This distinction matters for classification: generating missing current state supports context-conditioned inference, whereas world-model-based control requires predicted consequences on the execution path.

Selecting, compressing, and using history. Compression must preserve decisive events. Selection and ordering matter in language ICL [124]; robotics adds brief physical transitions that change a trajectory’s meaning. Robust Instant Policy’s downsampling study identifies lost grasp transitions [201]; its full system also fits a policy at deployment.

The useful horizon follows the unknown. AMAGO’s MetaWorld ML-1 analysis finds that short histories can identify hidden goals, while longer contexts can reduce sample efficiency [202]. ReLIC instead reuses experience across episodes to locate objects in an unfamiliar home, with deployed neural parameters fixed [203]. Section 6.2 relates this longer horizon to acquisition of spatial knowledge.

Retrieval targets the current uncertainty. HALO learns relevance through question answering; a history gate prioritizes sufficient current observations [105, 204]. RV-ICL exposes a read-only hierarchy of task, phase, moment, and clip views, letting a frozen coding agent revisit demonstration detail during execution [465]. Find, Fix, Reason trains a student with targeted spatiotemporal patches from a frozen teacher [453]. Both access resolution and evidence repair matter when they change the downstream decision.

Read-time abstraction can recover details omitted by fixed summaries. JitMem retrieves raw traces by task description and curates temporary guidance for a frozen executor [528]. Prompted and GRPO-trained curators share this interface; training fixes the bank, while deployment admits judge-filtered traces. ExpVoyager navigates trajectory and step views, tracking unanswered questions before synthesizing guidance [529]. These adjacent agent studies motivate a robotics comparison: whether revisiting raw interaction recovers a newly relevant event or condition lost during write-time summarization.

The basis of a remembered lesson can be retained alongside the lesson. WikiTrace links compact insights to original successful and failed trajectories, allowing an agent to inspect evidence when a summary conflicts with the present task or omits a needed detail [617]. Its read-only memory and insight-only controls in interactive agent benchmarks isolate the benefit of these links. For robotics, the corresponding design question is whether a retrieved repair preserves the observation, intervention, and outcome that made it applicable. This connects evidence recovery to the correction-reuse criteria in Section 5.3.

For an archive $M_t={\{{\xi }_j\}}_{j=1}^{N_t}$, ${\xi }_j$ is its $j$th record and $N_t$ is the number of records available at decision $t$. Let $J$ be a candidate set of record indices and $J_t$ the selected set. A budgeted retrieval rule is

Here $|J|$ is the selected-record count, $K$ its budget, and $\eta$ the relevance threshold. The score $s_{𝜃}$ follows Equation (6), now querying an archive record. Fixed ordering and tie rules determine $R_t$. A nonempty read supplies the normalized action mixture in Equation (5); an empty read requires a fallback policy or evidence acquisition. This additive rule selects records independently; sequence-aware retrieval instead scores combinations when events depend on one another.

Motion-sensitive features preserve executed change and temporal order [205]. EvoMem-VLA encodes ordered pairs of observed frames as source-conditioned change tokens, retaining initial, recent, and selected historical states [469]. These tokens summarize what changed before the current decision; predicted futures instead specify or assess what should happen next. Offline training determines how the policy uses the memory, while episode-time updates change its contents with neural weights fixed.

Stage-dependent retrieval and action refinement. Relevant experience must match execution progress as well as object appearance. RTCF aligns history to successful trajectories and transfers bounded low-frequency action components, leaving the remaining motion and gripper decisions to a frozen policy [195]. Its fixed evaluation bank isolates trajectory use from archive growth. PACE instead writes realized action–observation transitions into an episode-local fast-weight MLP, whose readout biases attention over the demonstration [475]. Training supplies stage-alignment supervision; deployment uses online gradients without stage labels. Both use history to resolve repeated-looking stages. RTCF selects an action reference; PACE updates the neural state guiding retrieval.

Failure traces add evidence about continuations to avoid. TraceFlow steers a frozen flow-matching expert with a bounded field derived from progress-aligned successful and failed traces [390]. Its ordering gains coexist with counting and occlusion deficits: stage identification and action correction impose separate demands. SALT uses another temporal signal, the unexecuted tail of the previous plan [476]. A calibrated gate activates updates to bounded projection scales using that detached, time-aligned target; updates accumulate within the trial and reset each episode. The tail carries prior intent rather than observed effects. Its usefulness under visual disruption therefore depends on whether that intent still matches the physical state.

Persistence and update timing. Persistence depends on what survives between decisions or attempts. Recurrent state and verified subgoal records preserve evidence [206]; task memory can organize repeated calls to fixed primitives [207]. Fast-weight methods instead update policy parameters during a sequence [10] or before rollout [11]; an online critic changes another component while leaving the actor fixed [208]. HumanoidTTT places learning in memory maintenance: qualified motions are reused through a frozen tracker when entry conditions hold, while a Double-DQN scorer learns which motions to retain [449]. Freezing the executor therefore leaves two independent questions: whether stored evidence changes and whether the rule maintaining it learns. Resetting records and adaptive parameters separately isolates their contributions. TTT-RM combines bounded explicit history with fast weights that encode the history decoder’s reconstruction residual [471]. Online gradients adapt those fast weights while slow weights remain fixed; resetting both carriers separates stored observations from learned residual memory.

Table 7 compares information roles and validity; Table B2 in Appendix B compares storage and updates. One memory design can serve several roles.

Capacity, access cost, and adaptation speed impose different constraints. KV caching saves prefix encoding while attention still grows with history; recurrence and fast weights compress it, and retrieval selects from an archive. Pre-rollout adaptation adds preparation time. Section 7 compares decisive-event retention and feedback-to-action delay.

Exposing changes the controller can execute. Memory makes evidence available; the command interface determines whether the inferred change can affect execution. ReSteer exposes the state dependence of this interface: policies that complete two instructions from their usual starting states can fail to switch between them midway [108]. Transition data and policy refinement improve switching. The implication for contextual control is that a new instruction or correction must be actionable from the state reached under the previous one.

Command granularity determines which correction can be expressed. Steerable Policies exposes subtask descriptions, motion instructions, and pixel targets [210]. The 2026 RoboPrompt system translates sparse points, trajectories, or directions into action drafts, then refines them through a frozen base policy [482]; its translator is trained beforehand. CIRRA instead revises an explicit skill queue through clarification, preemption, or fusion with the ongoing plan [486]. These interfaces modify an action proposal or its orchestration using available skills. The subsequent DAgger stage in RoboPrompt changes policy weights and constitutes a separate learning phase.

Correction requires available evidence, an expressive command interface, and execution from the current state. These dependencies span all four families. Agentic adaptation additionally revises the records, programs, or context-construction procedures supporting later decisions.

3.6 External knowledge reuse and revision

External reuse preserves guidance, callable routines, or context and verification code beyond the originating attempt. Guidance changes later inference; routines extend the execution repertoire; revised code changes how evidence is used. Table B3 in Appendix B distinguishes these update objects. Separating acquisition, reuse, and neural consolidation connects immediate adaptation to improved learning on subsequent tasks.

To separate within-attempt inference from persistent adaptation, let $B_n$ collect the external artifacts retained before attempt $n$: memory records, skill code, controller code and its numerical settings, or context-construction code. Let ${\mathcal{𝒯}}_n$ be the executed trace and feedback from that attempt. A proposal-and-validation loop has the form

The fixed model $F_{𝜃}$ proposes ${\tilde{B}}_{n+1}$; the indicator ${\chi }_n\in \{0,1\}$ accepts or rejects it after validation, which may require further rollouts. The effective policy ${\pi }_{𝜃}(A_t\mid h_t,C_t;B_n)$ exposes dependence on accepted artifacts for context construction or command execution.

Accepted code and settings can select executor ${\kappa }_{B_n}$ or determine program $g$; its learned neural parameters remain in $𝜃$. Systems without an acceptance gate set ${\chi }_n=1$. Attempt index $n$ differs from decision index $t$: acquisition can span multiple model calls and transitions. Section 7.2 includes acquisition and validation trials before measuring reuse.

Table 8 groups artifacts by execution role; Appendix Table B3 gives their mechanisms. Plans specify steps, guidance retains applicable rules, and repertoires expose callable routines. Programs or their settings determine the task controller. Execution oversight constructs inputs or supplies a separate verifier; checks within a task program belong to that program. Guidance and callable routines can coexist.

Two uses of the retained artifacts explain how this update reaches control:

The builder and memory inputs follow Equation (17). Guidance changes the inputs to otherwise unchanged control; executable reuse exposes library $B_n$ to specification generation and executor ${\kappa }_{B_n}$. Equations (19)–(20) thus connect acquisition to later behavior with neural weights fixed.

Grounding an interface and learning through it. Show-Harness separates semantic motion commands from their calibrated, body-specific interpreter [31]. Video can supply a missing task order. Reading explicit command conventions is easier than discovering arbitrary meanings through probing: one supplies correspondence, the other infers it from action effects. Both share the same low-level interpreter.

High-level commands delegate grasp synthesis and feedback control; lower-level commands require more model-side geometry and timing. Guava compares workflows and action abstractions [211], while compositions of frozen motor skills, planning, and verification expose their system-level effect [212]. Context interventions isolate what teaching adds to this existing competence.

SkipVLA makes this allocation explicit within a hybrid policy: a classical planner handles free-space transit, while the instruction-conditioned VLA handles contact-rich motion [339]. Its learned target predictor connects the two. The routing design improves execution efficiency by assigning transit and contact to interfaces suited to their respective demands.

Outcome-conditioned revision extends open-loop synthesis. Feedback revises plans [186] or Python controllers [213]; executable checks handle some responses without another model call [28]. Constraint violations narrow revisions to code and sampling bounds [214], while primitive errors guide control-setting changes [9]. The interface defines the available edits; Section 5 asks which edit a failure justifies.

Retaining guidance and executable routines. Reuse requires identifying what remains valid after repair. Constraints preserve scope [46]; causal annotations explain effects [215]; distilled rules support synthesis [48]. Attribution separates failure to follow valid guidance from a defective skill requiring revision [216]. Recursive Harness Distillation revises intervention playbooks using recipient feedback [428]. Its physical recipient is first fine-tuned on 50 demonstrations per task, then frozen during playbook evaluation; feedback tests improvement over the initial playbook. Robot-body assets and synchronized interaction records instead ground controller calls [429]. Diagnosis determines what to retain.

Visual evidence makes retained guidance more explicit. WAA turns expert demonstrations and human corrections into procedures with applicability conditions, reference images, and outcome checks [386]. In its one-demonstration stove study, review replaces knob motion as the success criterion with an observed activation signal. Fixed agents then interpret the acquired guidance. Real2Gym additionally reconstructs an interactive simulator from a demonstration, uses subtask programs with observable completion checks, and extracts object-relative rules and applicability conditions from execution [518]. The library changes while model weights stay fixed. Its second-run reuse tests and narrow Franka sim-to-real case ground different benefits: repeating a procedure more efficiently and recovering a physically unsuccessful task.

A request for teaching can target a particular program failure. ACE-CaP-0 lets a coding agent request robot demonstrations and simulation while developing a task program, then freezes the resulting code for repeated physical evaluation [625]. Demonstrations supply missing operational detail; execution exposes whether the repair works. Its development protocol counts assisted scene resets separately from demonstration resets, so the assistance budget should accompany the final success rate. This provides a concrete bridge from acquired guidance to executable reuse and motivates the active-coaching loop in Section 8.5.

Executable reuse makes recurring behavior callable. Function synthesis supplies missing operations; clustering and feedback refine candidates [27, 50, 51]. Video retrieval provides contacts and poses, and successful code can enter a library [166, 217]. EmbodiedEvo separates parameterized Skills from condition-linked Lessons, freezing both before fresh test agents receive the read-only artifacts [543]. Its raw-history comparison tests representation rather than accumulated access: similar observed success with lower token cost supports efficient reuse, without establishing equivalence. Transfer requires valid arguments and preconditions.

Demonstration-derived memory can also preserve a procedure without compiling every detail into code. RoboScript constructs semantic Task Memory from a demonstration, revises it during rehearsal, and freezes it for evaluation [554]. Privileged outcome diagnostics belong to its learning stage; the acting agent uses deployable observations, and evaluation still permits observation-driven recovery. SimplEvo separates a stable task description from working lessons revised across training episodes, then freezes both for fresh evaluation [562]. Their common mechanism is a controlled change in the memory supplied to a fixed executor. Rehearsal feedback, test-time observations, and optional later self-improvement must be reported separately.

Autonomous trials acquire different reusable objects. Controller rewriting improves an implementation [218]; task-agnostic play accumulates callable code [52]; repair distillation retains failure conditions and strategies [47]. ProactiveVLA changes which experience is collected: after the initial task, remaining turns probe affordances, state changes, and compositions [474]. Outcome checks support rules consolidated across adaptation scenes, then frozen for evaluation alongside the models. Exploring beyond a successful solution can thus supply guidance for changed conditions without expanding the learned motor repertoire.

Reusable libraries need failure attribution and regression checks. RoboRSI localizes failures to a task–skill branch, validates scoped revisions, and consolidates stable sequences into callable code [347]. Code reuse yields 174/600 versus 129/600 successes without code across 120 tasks and five layouts; a separate branch trains policies from execution traces. RPG diagnoses failures from simulation and videos, revises skills and prompts, and checks local and merged effects before frozen deployment [439]. DependEval tests dependencies, repository construction, and coordinated multi-file edits [454]. Physical regression and software consistency protect different aspects of reuse.

Selective revision becomes more interpretable when the reusable unit has an execution contract. ReSkill represents behavior as an object-centered skill graph with entry conditions, termination checks, and versioned implementations [553]. Repeated failures identify a local revision, which is screened on the failed conditions and then against broader tasks before deployment as fixed executable code. PyRUA-REAL acquires reusable code and lessons from physical exploration [586], while Flip Anything develops bimanual skill graphs through simulation before physical transfer [580]. Together these settings distinguish where a repair is acquired, what is retained, and where its transfer is actually tested.

Transfer also determines when knowledge may change. Skill2Real validates local guidance and higher-level compositions in simulation, then freezes both memories for physical deployment [462]; code sketches guide synthesis through existing APIs. RoboBridge instead revises guidance from target-side outcomes, comparing initial transfer with ten update episodes [463]. Its shared interface links different pretrained VLA checkpoints; action-gradient refinement changes actions with weights fixed. Procedures, neural weights, and executors thus have distinct transfer conditions.

Semantic effects can remain reusable across different geometries. Teach and Grow’s Skill Blocks specify effects, scope, executors, verification, and recovery [53]. Its simulation pilot expands a library with deterministic decomposition and a fixed executor; agent-based induction and Experience Memory updates remain proposed mechanisms for learning how to build that repertoire.

Reuse can encompass a complete closed-loop policy. Refined programs become examples for later synthesis [340]; generalized TAMP provides fully observed object states and simulator access during synthesis, then tests frozen programs on unseen instances [405]. RIVET grounds programs in object poses and relations [406], while HuGo revises code over a frozen whole-body controller [407]. InterEvolve revises staged reward programs and tunes constants with CMA-ES over a fixed controller [440]. Its unseen-composition comparison holds search rounds fixed while varying full, absent, and carry-only libraries. Carrying experience helps relocation but supplies less support for stacking or carrying–placing–kicking. Transfer depends on coverage of the required contacts and subsequent revision, rather than library size alone.

Adapting how evidence reaches control. Retained guidance improves how available routines are invoked. Parameterized call sequences preserve useful orderings, while global heuristics describe the operating conditions and failures of fixed primitives [207]. Hi-VLA’s tabletop study likewise finds greater benefits from prior-episode affordance summaries than longer within-episode history [219]. Such memory captures conditions of use, connecting agent reasoning to system identification of available capabilities.

A strong initial interface provides the baseline for this comparison. AMP1 directly controls manipulation through a general-model harness and separately evaluates augmentation with cross-episode memory [561]. Zero-shot competence establishes what the model and interface already provide; keeping or removing acquired records measures the additional value of reuse. This separation prevents improvements in the underlying action interface from being attributed to retained experience.

Persistent scene memory can support queries about an object’s present state or its earlier motion. Ledger maintains object records, and DynaMem updates spatial-semantic point clouds as objects move [426, 433]. Linguistic Trajectory Encoding (LTE) retains object timelines as motion captions linked to sparse 3D anchors and visual crops [547]. Language selects the relevant motion phase, geometry localizes it, and visual evidence resolves identity. This representation makes an earlier event recoverable after the scene changes; its retrieval interface supplies evidence for subsequent decisions.

Reusable guidance requires a further step from observations to action-relevant records. RobotUse returns compact scene, outcome, and unresolved-constraint summaries from local histories while retaining a cross-attempt playbook [470]. Models and the geometric backend stay fixed. MarvisNav renders exploration history onto candidate routes so a frozen VLM can select a route for a classical planner [466]. These interfaces expose retained evidence in the form the next decision consumes.

Repeated selection or verification failures motivate revising the procedure that supplies context. SHAPER changes procedural guidance and context construction [49]; Zetta changes critics and recovery tools [90]. RACaP evolves Policy APIs, its ReAct harness, and memory before deployment [378], while RoboFoundry promotes validated context and skill-system changes [430]. PhysEvo extends revision to the meta-agent’s diagnostic tools: a frozen model uses separate task and meta contexts to improve both execution artifacts and the means of inspecting failures [484]. Held-out evaluation then freezes the acquired artifacts. External revision changes what the controller can use; runtime context determines how those resources apply.

Compiling repairs moves computation from runtime to development. HarnessPAI revises programs using stage-indexed memory; 15 of 50 evaluation seeds also support evolution [396]. Embodied Turing Machines executes fixed perception and control code without VLM calls, using simulation asset and calibration priors [494]. ChunkAct instead compiles experience into callable chunks; explicit memory reads disappear, but a planner handles failures [544]. Against a matched harness with the same memory, planner, perception, and primitive, compilation improves success while reducing planner calls and tokens. These comparisons separate runtime retrieval from procedural reuse; acquisition cost remains distinct from execution cost.

Connecting contextual acquisition to neural consolidation. Neural consolidation changes the executor itself. Harness traces [211, 386], autonomous collection [220], and EmbodiedSWE's diversification of executable solutions into demonstrations [380] supply policy-training data; CAPEX amortizes collection-time reasoning [432]. RoboCoach makes supervision selection part of the learning loop: imagined failures select demonstrations and the skill adapters receiving LoRA updates, while the router, world model, judge, and backbone stay fixed [442]. Robo-COP additionally validates candidate policy updates and revises memory guidance after adoption or rollback [485]. A warning about an old policy’s weakness can otherwise suppress a newly acquired capability. Retained context must therefore track the executor version it describes. Training on successful teaching exchanges, as in LMPC [117], targets the successor’s responsiveness to later feedback.

The interaction record can supervise a successor that no longer calls the original agent. H2E converts complete navigation sessions into VLA training examples under the original task instruction, masks stalled actions using progress, and aligns waypoint and stop labels [636]. The resulting policy is evaluated independently on unseen REVERIE-CE scenes. This differs from retaining the session as runtime context: parameters now carry the distilled behavior. Comparing both routes at matched tasks reveals whether reasoning should remain an external interpreter or be amortized into the executor.

Takeaways

• Context determines an action distribution, a geometric reference, predicted consequences, or an execution specification. Equations (4), (7), (9), and (13) identify these four interfaces.

• Context interpretation and motor competence limit transfer separately. Retrieval and memory support transfer by supplying usable evidence to a controller that responds to later corrections.

• With neural weights fixed, context can revise task outlines, executable routines, and programs. Retaining these artifacts supports reuse, and successor comparisons measure their contribution to learning subsequent tasks.

4 Data and Training: How Is Context Use Learned?

An input channel exposes context; training determines whether it controls action. A scene that identifies every target permits fitting without consulting teaching. Useful training must make earlier evidence informative about a later decision. Acquisition determines what is measured, episode organization supplies these relationships, and construction at scale varies them.

Source taxonomies describe where embodied data originate and how closely they match execution [221]. Contextual learning additionally concerns what earlier evidence reveals about a later decision. Paired episodes, continuous streams, and linked attempts establish different dependencies; pretrained interpretation or geometric reconstruction can connect them without jointly training the whole path.

Cross-modal and cross-embodiment alignment determine what those records mean to the learner. SWAM uses egocentric video and UMI co-training to preserve language, video, and action alignment before robot adaptation [578]. CIALLO learns a shared intent representation, calibrates an embodiment-specific component from brief interaction, and then trains the task policy with the calibrated codec fixed [598]. These are training and calibration pathways. Their data efficiency does not by itself establish learning a new task solely by inserting a demonstration into a fixed deployed policy.

4.1 Input sources and context modalities

An acquisition source determines what is measured directly and what must be inferred. Teleoperation records robot commands, UMI estimates motion through a robot-like device, egocentric video records human activity, and simulation provides modeled transitions. The four sources are compared on this common basis.

Language can accompany every acquisition source. Instructions specify constraints, command–program examples establish conventions, corrections revise decisions, and summaries retain earlier events. Code as Policies, DROC, and MEM illustrate the latter three roles [27, 44, 46]. Records should preserve each statement’s source, time, and referent: a teacher’s requested change and a model’s interpretation of an event carry different evidence.

Figure 13 contrasts the directly observed interaction: robot execution in teleoperation, instrumented hand motion in UMI, human activity in egocentric video, and modeled execution in simulation. These sources leave different correspondence problems for the learner, as the following parallel comparisons explain.

Teleoperation: measurements and commanded interventions. A real robot provides synchronized scene observations, measured robot state, and issued commands. DROID and BridgeData V2 supply such robot demonstrations [56, 58]; RH20T additionally records force–torque measurements and, in one hardware configuration, fingertip tactile signals [59]. These records can connect an attempted intervention to the body’s response. A gripper-close command specifies an action, while measured aperture and contact help establish whether the object was grasped. Joint torque or motor current also requires interpretation through the robot’s mechanics before it can describe external contact.

UMI: tracked motion without the robot arm. UMI reduces collection dependence on a robot arm by preserving a gripper-like interface in the demonstrator’s hand. Images and inertial measurements support device-motion estimation, while visual tracking estimates gripper width [62]. The resulting trajectory can supply end-effector targets without measuring arm-joint motion. Sharing the end effector between collection and execution reduces a further source of correspondence error [66]; synchronized tracking improves the fidelity of the recovered motion [223]. Extensions address acquisition, whole-body realization, and richer interaction sensing [224–226]. Their common trade-off is deliberate: an instrumented interface restricts how a human demonstrates, but reduces the inference needed to turn that demonstration into robot-compatible evidence.

Egocentric video: visual procedure with optional human sensing. Egocentric recordings expose object interactions and their order without requiring a robot-compatible device. Ego4D and EPIC-KITCHENS-100 provide egocentric activity, and Ego-Exo4D adds synchronized external views [68–70]. Some collections also provide audio, gaze, inertial measurements, depth, or tracked hands. They measure human motion and the surrounding environment. Robot action targets must come from paired execution, reconstruction and retargeting, or a controller that grounds the observed procedure. An action-free video supplies visual observations without recorded robot commands.

Simulation: rendered observations and modeled responses. Simulation makes action consequences and privileged state available under a specified dynamics model. Demonstration environments such as RLBench, LIBERO, and ManiSkill2 support task execution with synchronized observations and controls [227–229]; trajectory or task-program construction produces additional executions [74, 230]. This control over the generating process is useful for contextual learning because a procedure can be held fixed while scene geometry changes, or varied while the scene remains comparable. Privileged object and contact states can verify those relationships during construction, even when the deployed policy receives cameras and proprioception. Section 4.5 examines how synthesis preserves the assigned teaching condition.

Channels and policy inputs. Table 9 distinguishes measured state, commands, and contact evidence. Collection channels need not enter the policy: UMI uses IMU data for pose estimation, while its policy observes images, end-effector pose, and gripper width [62]. Specify available fields separately for support, query, and supervision: human video may supply support while the robot queries joint and contact sensors; recorded support commands may condition inference, whereas future query commands supervise it. Measurements $o_t$ and issued actions $a_t$ form history $h_t$; selected records, language, and feedback enter context $C_t$.

Combining channels also requires preserving disagreement. TUNED combines feature and neighborhood evidence for multi-view classification [450]. For contextual control, the relevant question is whether conflicting observations remain distinguishable before fusion commits the controller to one interpretation: contact feedback can contradict an otherwise plausible visual estimate.

4.2 Training units and context dependence

Channels become useful training evidence through temporal and task relationships. Table 11 compares paired episodes, continuous streams, and linked attempts. We define the temporal units first, then use support–query supervision to distinguish input channels, action targets, and pairing criteria. Continuous experience and corrective records extend this dependency beyond explicit pairs.

Temporal units of contextual evidence. A contextual training unit must identify what was observed, what was commanded, and what the resulting feedback established. Language can specify the intended task or annotate the outcome. The sequence must preserve when each item became available, so that a later consequence can supervise learning without appearing as evidence for an earlier decision.

Temporal alignment preserves the relation between an intervention and its consequence through sensor timestamps, command application times, coordinate conventions, and episode boundaries. Measured motor state must remain distinguishable from its command or setpoint. Aligned windows or timestamp-aware encoding accommodate different sampling rates while retaining brief contact events. Missing channels and reconstructed motion must also remain identifiable, because they supply different evidence from direct measurements.

Table 10 groups records by what they contribute to learning. Support can specify behavior or reveal an environment; query history locates execution; transitions reveal action consequences. These units overlap: history contains transitions, and retrieved segments can come from demonstrations or earlier attempts. Their role is independent of modality. Images or synchronized robot records can supply a demonstration, while state–action history can support physical adaptation without one, as in RMA [39].

Supervising a query from a support episode. Following the paired-execution principle of one-shot imitation [1], let support demonstration $D^s$ and query observation–action trajectory ${\tau }^q$ share task specification $\psi$. The matched executions may differ in objects, timing, or embodiment. A common supervised objective predicts query actions from their history and support:

Here $P_{\mathrm{pair}}$ samples executions matched by specification $\psi$. Query history $h_t^q$ ends at the current observation, while $A_t^q=a_{t:t+H-1}^q$ is the recorded target block. The nonempty set $I^q$ contains positions with complete targets; averaging over it normalizes each episode’s contribution. The loss ${\ell }_{𝜃}$ may use likelihood, regression, denoising, or flow matching. These objectives differ in action modeling but share the same supervisory dependency: the support must explain which query behavior is appropriate.

Equation (21) specializes the general context interface to $C_t=D^s$. In instruction-bearing episodes, the same query-action loss conditions on $C_t$ containing both the demonstration and its associated text. Instructions specify goals and constraints; demonstrations can convey remaining choices of order or contact. A correction enters context when received, preserving what was available before each supervised action.

Figure 14 distinguishes motor supervision, training to use teaching, and deployment-time task specification. Uprighting a differently shaped object illustrates the third role: the operation must survive changed appearance. Paired supervision learns this dependency explicitly; continuous streams can expose it through the sequence of experience.

Support channels and action supervision. Equation (21) supervises query actions while leaving support channels open. Vid2Robot conditions on video without recorded support commands [14]; ICRT supplies support actions in a sensorimotor prefix [13]. Both supervise the query execution. Support content, the action loss, and the architecture that combines evidence are therefore separate choices.

Task identity and support–query pairing. The shared specification $\psi$ determines the granularity of teaching. Pairing all executions labeled “put the object away” can preserve a destination while discarding grasp or route preferences. BPP makes this dependence explicit in its task-grouped demonstrations [54]. The pairing rule must retain the distinction the deployment example is expected to convey. For cross-object transfer (Section 3.5), support and query can change the manipulated object, the receiver, or both while preserving the same requirement. Conversely, retaining comparable objects while varying the required relation makes the support informative. Together, these pairings separate task meaning from object identity and provide examples of which substitutions preserve the intended function.

Pairing requires both semantic agreement and compatible action meaning. Additional demonstrations can distinguish task interpretations left ambiguous by one example, as Instant Policy illustrates with differently shaped boxes [16]. The resulting query target must still express the same intervention in consistent coordinates, timing, and gripper semantics. WALL-OSS-0.5 addresses this physical side through synchronized video and actions, camera repair, and end-effector calibration [233]. These requirements are complementary: semantic pairing determines which behavior is intended, while calibration determines what command realizes it. With shared action coordinates, each body retains its own reachability and contact mechanics.

A shared environment supplies a different pairing relation from a shared procedure. Route-following training pairs a demonstrated path with a perturbed execution, as in RPF [355]. Scene-conditioned training instead pairs a preview with navigation to goals within that environment, as in NOLO [350]; ReLIC [203] links episodes in one home while changing the target and start. The shared variable is therefore the route in the first case and the environment in the latter two. Scene identity and memory-reset boundaries determine which spatial knowledge remains available across examples or episodes.

From paired episodes to continuous experience. Explicit pairing makes the teaching–execution relationship part of the training unit. S1 supplies an in-context demonstration in its pretraining episodes [99]. GEN-1.5 instead reports next-action pretraining on randomly sampled continuous physical spans, without dedicated demonstration packing or an ICL-specific objective, and inserts sensorimotor examples as prompts at deployment [98]. A stream exposes earlier observations and actions without assigning them separate support and query roles. Both systems later accept task examples, acquired through different organizations of training experience. Action generation is a further choice: ContextFlow uses explicit robot support–query supervision with continuous flow matching [55]. The action loss and the organization of experience play different roles: one fits the action model, while the other links preceding evidence to subsequent behavior. Cross-Episodic Curriculum orders trajectories by learning progress or demonstrator quality [435]; controlled MiniHack and Procgen studies link task diversity and trajectory burstiness to new-task ICL [434]. These training results explain why history can become useful, without establishing the same transfer on physical robots.

Behavioral diversity and contextual identifiability.

Training must make teaching informative about the continuation. If a scene predicts every action, the loss can be minimized without reading the demonstration. SimpleICL varies actions, object placements, orders, and grasp modes in paired executions, while varying demonstrator appearance and irrelevant scene content independently [517]. Its affordance ablation requires contrasting grasp modes for the same object; its composition test retains order discrimination without the corresponding explicit contrast. Thus different taught requirements need different data controls. Count distinguishable teaching problems alongside trajectories, and test each required invariance rather than assuming that one pairing recipe establishes them all.

Context can also support shortcuts: previous actions or a familiar scene may predict the training target without specifying the requested behavior [234]. CRAFT counters this dependence by varying the requested skill composition for the same observation and aligning separate skill and state representations [457]. Its counterfactual objectives train the VLA offline; deployment uses observation and instruction. Held-out compositions combine familiar constituent skills, making this evidence about compositional instruction use. For demonstration ICL, the corresponding test changes the taught relation while preserving the scene. Both designs ask whether task-defining evidence controls behavior beyond collection correlations.

The evidence distinguishes behavioral diversity from visual variety. ICRT’s selected DROID-only condition fails its test tasks; its multi-task scenes instead require different actions from similar observations [13]. At a fixed 10,000-demonstration budget, BPP’s mean drawing Chamfer error falls from 9.5 pixels for 500 tasks to 5.1 for 1,000 and 3.4 for 2,000 (Figure 15a) [54]. Its folding study can favor language conditioning. Its architectural comparison attributes much of BPP’s advantage over ICRT to the diffusion action decoder, separating decoder quality from prompting. Demo-JEPA varies task classes and trajectories per task independently [22]. Useful diversity adds distinctions that teaching must resolve and the action decoder must realize during execution.

S1 separates scaling gains on seen and unseen tasks under matched data, compute, and architecture apart from the prompt embedding [99]. Across 1k–100k training hours, both prompting methods improve on seen tasks. On unseen tasks, demonstration prompting rises from 1% to 66%, while language prompting rises from 0% to 9% (Figure 15b,c). Thus similar seen-task performance can conceal different returns to new-task teaching. The cumulative per-step metric includes human recovery, mainly for the language baseline. The S1 comparison concerns combined data scaling and prompting, while BPP directly isolates task diversity at a fixed demonstration budget. Matching the procedural information conveyed by text and video would further isolate modality from teaching detail (Section 7.1).

Linked attempts and corrective supervision. Learning from a failed action or a diagnostic probe extends context dependence across attempts. The training unit must link what was tried to what happened next, preserving reset boundaries and any change in the task or physical condition. Algorithm Distillation models such learning histories [6]; LocoFormer retains experience over varied morphologies and dynamics [40]. Reward-based meta-training can value an informative action through its benefit to later attempts [2]. A short probe gains learning value when its outcome informs the next decision. Linked attempts expose this dependency during training.

In recovery-conditioned supervision, the failed attempt and its feedback enter context while a later continuation supplies corrective targets. RoboTTT’s DAgger Distillation uses the full rollout as context but applies action loss only to human corrections, also improving its GDN comparator [10]. The failed actions record what was tried; the selected corrections specify what to imitate. This training organization can support different memory mechanisms.

Equation (21) extends to this case by replacing support-only conditioning with $C_t$. For block-level losses, $I^q$ includes only complete blocks composed entirely of actions selected for imitation. An objective supporting action-wise masking can instead suppress unselected positions within a block. Other executed actions remain in history. The context mask controls available evidence, whereas the target mask controls desired behavior. Runtime context contains only evidence available before the decision; future outcomes may supply supervision. Attempt boundaries must separately record physical resets, memory resets, and changes of tool or dynamics.

Long-context control requires representations that preserve decisive events. Past-Token Prediction adds reconstruction of earlier actions to future-action prediction, first learning short-context visual features and then caching them for longer-context training [235]. Caching reduces repeated computation; the cached visual features determine which task details remain available to the controller.

The supervisory unit determines what evidence can explain a later action: pairs link teaching to realization, streams link earlier events to current control, and linked attempts connect outcomes to revised decisions. Their common requirement is a decision that earlier evidence helps disambiguate. The remaining analysis examines acquisition sources that preserve this dependency with usable action correspondence.

4.3 Data coverage and action correspondence

Data coverage has two distinct meanings: the range of behavior recorded and the range of teaching–execution relationships recoverable from it. Robot records, motion-bearing human demonstrations, and ordinary video trade direct action correspondence against collection breadth. The comparison below follows that trade-off, asking which annotations make each source usable for contextual training. Simulation supplies controlled counterparts in Section 4.5; Table B4 lists resource-level details.

From reusable experience to annotated procedures. Motor coverage supplies reusable behavior before teaching can specialize it. Autonomous multi-task experience and shared demonstration collections addressed this need [236, 237]; Bridge Data included 7,200 demonstrations across 71 tasks and ten environments. Later collections broadened scenes and embodiments [56–58]. Contextual training additionally needs relationships between these executions that preserve what a new example should teach.

Procedure annotations identify why a continuation changes. AgiBot World’s March 2025 release spans 1,001,552 trajectories, 2,976.4 hours, 217 tasks, and 106 scenes, including subtask and recovery records [60]. Task intervals, step instructions, and later corrections connect demonstrations, autonomous attempts, and human interventions [238, 239]. Linking an attempt to its correction preserves the evidence that makes a different continuation appropriate.

Embodiment diversity separates task correspondence from action correspondence. Galaxea shares a platform and subtask timing [61]; RoboMIND 2.0 and RoboCOIN span more platforms [240–242]. Procedural metadata establishes task equivalence; kinematics describes the differing realizations. Pairing needs both.

Robot records supply executable supervision. Robot records become contextual supervision when a query execution is associated with an example of its intended procedure [56–58, 60]. Cross-scene retrieval can locate candidate executions [243], but matching must distinguish alternatives that begin in similar scenes.

Human–robot pairing exposes an asymmetry between teaching and supervision. A human video can specify the intended interaction, while robot measurements reveal the commands and contact response needed to realize it. RH20T provides corresponding human videos together with robot images, joints, gripper state, force–torque signals, audio, and tactile sensing in one configuration [59]. These richer query records can supervise a policy whose eventual human support contains fewer channels. Task-level human–robot organization helps define the transfer setting [101], but a shared label such as fold can still merge different procedures. Matching ordered events and terminal outcomes, as in WorldScape’s robot and UMI pairs, preserves a finer teaching relation [25]. Pairing granularity therefore determines whether the model learns an activity category, a procedure, or an interaction detail.

Mixed capture can reduce pairing cost. XR-2 combines 531.7 hours of robot teleoperation with roughly 1,000 hours of dual-UMI demonstrations [244]. Robot views, state, and actions supply executable targets; egocentric views and gripper motion supply candidate support. Shared laundry tasks narrow the search, while procedure-level matching identifies compatible executions.

Motion-bearing demonstrations reduce correspondence uncertainty. Instrumented teaching trades restrictions on collection for less uncertainty in transfer. Handheld grippers make the demonstrator’s observations and motion closer to a robot-compatible interface [62]. Preserving the end effector across collection and execution further reduces what must be inferred: YUBI reports 8,434 hours over 119 tasks and transfers the same gripper between arms through per-robot inverse kinematics [66]. Sharing the gripper reduces hand-to-robot correspondence uncertainty, leaving each arm’s reachability and control limits to be handled during execution.

Bimanual teaching makes relative motion fidelity more consequential than the visual plausibility of either hand alone. Stereo-inertial tracking, synchronized views, and inter-gripper reconstruction preserve this coordination [223]. HiFi-UMI’s reported 2K release also includes subtask boundaries, and its policy studies use UMI post-training without target-task teleoperation. Calibration preserves the physical relation between hands; boundaries identify which procedure that relation instantiates. Modular tracking can simplify acquisition [224], while a simulation-trained whole-body controller can supply realization beyond the demonstrated gripper motion [225]. More accessible capture and more accurate coordination are consequently distinct improvements to the teaching interface.

Contact sensing extends what the teaching record can express. OmniUMI synchronizes vision and motion with tactile signals, internal grasping force, and external interaction wrench [226]. Such measurements can distinguish a visually similar motion performed with different contact conditions. Their transfer still depends on compatible sensing, calibration, and execution interfaces; a measured wrench and a commanded gripper setting carry different information.

Collection can be conditioned on the current policy’s weaknesses. HIL-UMI compares handheld human trajectories with policy predictions on the same observation stream and targets additional data to disagreement and low-advantage segments [341]. The resulting demonstrations support further policy training. Thus policy-aware collection improves which experience is acquired, while remaining distinct from using that experience as fixed-weight deployment context.

Removing the collection device shifts this burden from measurement to inference. Tracked hands and segmented point clouds can be converted into gripper poses [16]; motion inferred from human play can supply pseudo-actions, retrieved context–query relations, and retargeted humanoid supervision [245]. These approaches broaden the usable recording pool, but the action evidence inherits reconstruction and retargeting error. The trade-off is most consequential near contact, where a small positional error can change whether a grasp or insertion is feasible. A common action format enables joint use of these records while their reconstruction reliability determines their supervisory value.

Mixed-source training can preserve this distinction instead of forcing all trajectories to carry the same supervisory weight. Aligning camera-space actions, embodiment, and chunk duration provides a common temporal interface, while reliability-weighted human pseudo-actions supplement a robot-supervised primary objective [246]. In ACE-Ego-0, the auxiliary objective emphasizes more dependable position channels. This weighting lets human records expand coverage while measured robot actions anchor execution. Carrying reconstruction confidence into contextual pairs likewise avoids treating uncertain hand tracking as an equally reliable robot target.

Zeva-Ego separates this acquisition of a physical prior from deployment adaptation [362]. Its Action-Centric Encoder converts egocentric visual transitions into continuous action-token supervision for mid-training. A later interaction-conditioned stage trains the action model to use observed action effects; deployment freezes the weights and updates memory. Human experience expands prior competence, while robot interaction supplies evidence about the current physical conditions.

Motion annotation reduces the uncertainty left by ordinary video while retaining a human-centered collection process. Hand–object geometry [63, 64] and contemporaneous hand and finger tracking, as in EgoDex’s 829 hours [65], expose motion that otherwise must be reconstructed from RGB. Robot learning still requires conversion into feasible contacts and controls. EgoScale’s 20,854 hours of action-labeled human data are connected to robot behavior through aligned mid-training and downstream post-training [67]; its one-demonstration experiment uses policy updates with one robot demonstration per task and additional human trajectories. These stages show why improved action evidence and fixed-weight task acquisition are separate contributions: the former can support either contextual inference or further policy fitting.

The transition from human pretraining to robot deployment also changes the meaning of language supervision. Xiaomi-Robotics-1 combines over 100K hours of UMI and egocentric records with cross-embodiment post-training [247]. State-transition descriptions during pretraining and imperative instructions during post-training serve different prediction targets. For demonstration-conditioned learning, the corresponding design choice is which task details remain unspecified by the instruction and must be recovered from the example. This connects source breadth to the pairing criterion in Section 4.2.

Ordinary video broadens procedural coverage. Ordinary video reverses the trade-off of instrumented capture: activity coverage is broad, while robot-compatible action information is sparse. Egocentric activity labels help locate candidate procedures [68, 69]; paired viewpoints and short human–object changes offer cues about correspondence and outcomes [70, 71]. These sources can identify what happened without fully specifying how a different body should reproduce it. For contextual teaching, selection must preserve the choice that distinguishes this example from another clip of the same activity.

Annotation is useful when it removes uncertainty about the relation teaching must preserve. Common capture protocols reduce incompatible recording conventions [248]; atomic segments and reconstructed hand and camera motion expose where an interaction begins and ends [249]. Object-state and contact annotations can resolve physical events hidden in RGB [250]. These additions are not interchangeable: reliable boundaries help preserve order, whereas contact-sensitive transfer needs relative motion and object response. The appropriate annotation follows the task distinction the support example must convey.

Procedural and conversational datasets contribute another missing dimension. Assembly101 records natural variations, mistakes, and corrections during assembly and disassembly; HoloAssist pairs a performer’s egocentric activity with an instructor’s spoken guidance and intervention annotations [72, 73]. Their contribution is the relationship between a partial procedure and information that changes its continuation. Transferring that relationship to robot ICL requires robot-compatible continuation targets or an executor that can realize the corrected instruction. Preserving mistakes, instructions, and corrections as linked events retains this distinctive supervision.

Human-video priors and demonstration following use this evidence differently. Learning latent plans from human play can reduce the robot experience needed for control [251], even when human and robot executions are not paired. Following a newly supplied video additionally requires preserving the demonstrated choice in a compatible robot execution. WAM-TTT learns that connection through paired human–robot meta-training, while its deployment-time fast-weight update needs only action-free human video [11]. This is parameter adaptation under the distinction in Section 2. Robot supervision establishes the connection during training; the later teaching input can omit recorded robot commands.

Cross-embodiment training requires a relation stronger than shared appearance. H2RB aligns hand-grounded visual features and learns shared latent actions with embodiment-specific decoders [651]. AIT uses action intent as the transferable intermediate, while SALA aligns unpaired human and robot clips through language-defined action neighborhoods [652, 667]. Each approach establishes which human transition should supervise which robot behavior. Their transfer experiments update models using human and robot data; following a newly supplied demonstration with those parameters fixed remains a separate deployment test. Preserving this distinction locates human-video scaling within the acquisition of contextual competence.

Preserving a teaching relation requires more structure than a shared task name. A mobile-manipulation episode contains identifiable substeps and transitions that can anchor retrieval or a corrected continuation (Figure 16). Human support and robot execution can traverse those stages at different speeds, so task correspondence matters more than frame equality. Constructing this usable teaching signal incurs a cost beyond recording the raw activity, motivating the acquisition comparison below.

4.4 Acquisition cost and contextual capability

Data value depends on the cost of producing usable teaching, not recording alone. We first examine correspondence costs, then compare explicit contextual training with broad priors and robot grounding.

Collection cost and correspondence. Acquisition cost includes filtering, synchronization, reconstruction, pairing, and validation. Egocentric video records hand–object interaction; UMI also measures device motion. Packing may require event boundaries and contrasting orders, whereas insertion needs calibrated relative motion and contact evidence. More views or longer recordings help when they preserve these distinctions.

Recording design determines which contextual relation can later be tested. HEIR couples egocentric views, gaze, speech, and robot trajectories across interactions in which later requests refer to earlier objects [630]. ToolDex records the hand–tool–target relation, separating movement of the tool from its effect on the manipulated object [655]. Ropedia Xperience 2 links human activity to reconstructed three-dimensional environments [637]. These resources preserve intent history, mediated contact, and scene correspondence, respectively. Their value for contextual learning comes from constructing examples and queries that require those relations, with released data distinguished from the larger collection inventory.