Follow the prediction into execution.
A world model contributes to robot control when its predictions influence which actions are generated or selected. It may predict a desired visual or latent future for an action decoder, or predict the consequences of candidate actions for a planner. Learned predictors and explicit physics simulators can both participate in this interface.
The Robot ICL Survey connects world-model control to the role of context. A demonstration can specify the intended procedure; recent transitions can identify physical response; paired actions and video can specify how commands map to motion. These sources resolve different uncertainties in the same control loop.
The deployed computation determines the classification. An auxiliary prediction objective may improve a policy during training even when no future is generated during execution. Identifying what the action computation actually consumes makes comparisons between world models, policies, and their training objectives clearer.
Choose what the future is used for.
| Role | Control path | Central question |
|---|---|---|
| Future-conditioned action | Context specifies a future; a decoder generates actions conditioned on it. | Does the future preserve the teaching, and can the decoder realize it? |
| Action-conditioned planning | A predictor rolls out candidate actions; a task-dependent score guides selection. | Do predicted consequences rank actions reliably under the current dynamics? |
| Reconstructed simulation | A scene model simulates candidate actions and returns outcomes for revision. | Does reconstructed geometry and physical response preserve the relevant interaction? |
| Auxiliary prediction during training | A prediction loss shapes representations; deployment may use only an action head. | Which benefit comes from training, and which computation remains online? |
Optimizing candidate actions with a fixed predictor changes the plan, not the predictor’s neural weights. Fitting a neural residual or updating the dynamics model changes another part of the system. Report these operations separately from supplying new contextual evidence.
Four studies clarify the interface.
Zero-WAM: demonstrations shape robot futures
Zero-WAM connects human demonstration context to future robot observations and actions. Its deployed video and action branches remain fixed, and generated future video replaces recorded future video. A central test is whether alternative teaching changes the predicted and executed procedure.
Demo-JEPA: latent goals for a planner
Demo-JEPA supplies a demonstration-derived latent goal to a cross-entropy-method executor until an achievement threshold allows progression. The survey discusses its action-head comparison to show that the value of planning depends on task familiarity and execution setting.
SyncWorld: contextual action calibration
SyncWorld uses paired action–video calibration segments to establish a camera- and embodiment-dependent mapping from numerical commands to visual motion. The fixed predictor then supports ranking candidates through imagined rollouts. Here, context describes action semantics as well as the task.
SynthICL: prediction as supervision
SynthICL provides a contrasting design: its auxiliary subgoal-image head is disabled at deployment, while a demonstration conditions its action policy. This makes prediction a training contribution rather than an online intermediate consumed by control.
Evaluate imagined and realized outcomes together.
Test whether changing the teaching changes the required aspect of the future: target, ordering, contact, or another relation. Then measure whether the robot’s actions realize that change. Visual plausibility alone leaves both questions open. A goal image may be sufficient for an endpoint task, while a demonstrated procedure can impose additional intermediate constraints.
To locate failures, substitute a task-consistent future or correct intermediate while preserving the executor and interface format. For planning, compare action budgets, rollout horizons, candidate support, and available calibration evidence. For transfer, state which predictor, decoder, or scene model changes and which remains reusable.
A research opportunity is to connect prediction errors to decisions that matter physically: an incorrect contact transition or action convention can alter the chosen command. Measure the resulting behavior and feedback-to-action delay, including planning, instead of evaluating visual prediction quality or decoder latency alone.
Continue with world-model-based control, prediction in training and deployment, predictive and skill interfaces, and response delay and reuse cost.