Test dependence, realization, and reuse separately.
Under the same checkpoint and executor, compare informative, absent, and meaning-swapped teaching. Measure whether the robot’s behavior follows the changed requirement as well as whether the task completes. Then vary physical conditions and retained state to establish the scope of the benefit.
Does teaching change behavior?
Hold the current scene fixed and change the taught count, order, target, or relation. A meaning-changing intervention should change behavior appropriately.
Does the requirement transfer?
Preserve the taught requirement while varying objects, scenes, bodies, or dynamics. Identify which factor changes and what prior preparation remains available.
Does retained experience help later?
Compare retained and reset memory or artifacts under equal physical initialization and retry budgets. Test applicability beyond the practiced instance.
Can the executor realize it?
A fully specified target or corrected intermediate can diagnose motor competence. Correct interpretation and successful physical execution are distinct readouts.
Source: Section 7, Evaluation: When Does Context Help?; PDF p.65 onward.
Specify the target, intervention, and readout.
“Adaptation improved performance” leaves the changed component unresolved. Table 17 separates checkpoint quality, new evidence, neural updates, and reusable artifacts. Match tasks, physical resets, base execution resources, and attempt budgets.
| Target | Intervention | Readout |
|---|---|---|
| Training checkpoint | Cross checkpoints with the same context conditions. | Acquisition of the ability to use context. |
| Task teaching | Original versus replacement teaching; include absent context. | Adherence to the supplied requirement. |
| Retained history | Retain versus clear at a matched physical state. | Use of evidence from earlier observations or interactions. |
| Control intermediate | Original versus corrected correspondence, future, or executable specification. | Intended action change and task success. |
| Neural parameters | Adapted versus restored pre-adaptation weights, with matched inputs. | The contribution of parameter adaptation. |
| Reusable guidance | Retain versus withhold verified lessons. | Transfer of acquired guidance. |
| Skill library | Expanded versus base library. | Dependence on newly acquired routines. |
| Active observation | Enable versus disable evidence acquisition under matched budgets. | The value of acquiring new information. |
Within-checkpoint differences test evidence use. Comparing checkpoints under the same interventions tests how training acquired that capability. Restoring neural weights is a separate intervention from clearing memory; otherwise the two sources of adaptation remain mixed.
Source: Section 7.2, Sources of improvement and comparable evidence; Table 17 (PDF p.71).
Build a comparison that can reject the easy explanation.
- State the taught distinction. Define the order, target, relation, or procedure that the context supplies, and define an observable adherence measure.
- Hold execution conditions fixed. Use the same checkpoint, executor, scene initialization, available tools, and attempt budget for the context comparison.
- Vary information deliberately. Include informative and absent context, a feasible meaning-changing replacement, and meaning-preserving variants such as a different view or demonstrator.
- Record behavior and outcome. A changed internal representation is insufficient if action does not change; task completion is insufficient if the taught requirement is ignored.
- Test a separate transfer axis. Change an object, scene, relation, body, or physical condition while stating what training and preparation remain shared.
Illustrative design: an ordered packing task
In the same scene, supply teaching for A-before-B and B-before-A. Measure order adherence and completion, then repeat on substituted objects. Keep the motor repertoire and retry budget fixed.
Unknown pretraining limits claims of task novelty. Scene variation, instruction variation, and learning an unfamiliar rule support different conclusions; report the tested distinction explicitly.
Source: Sections 7.1–7.2; the transfer distinction follows the survey’s tests of meaning-changing and meaning-preserving context.
A later success can come from memory, progress, or another chance.
Reset the physical task independently from retained information. Otherwise a later attempt may benefit from objects already moved, additional trials, or an expanded skill library rather than from the memory being tested.
Δnreuse = Snkeep − Snreset
Here, both rates measure the same success or adherence outcome at attempt n. Match the task distribution, physical initialization, executor, other artifacts, preceding attempts, and execution budget. Per-attempt differences isolate retention benefit; cumulative success also grows when independent retries provide more chances.
Then test distance from the practiced task: a new instance, a new composition, or changed physical conditions. Clearing an external archive, withholding repair guidance, restoring a base library, and resetting neural parameters answer different questions about what was retained. The robot self-improvement guide connects these interventions to learning across tasks.
Source: Sections 7.2–7.3, including matched retention tests and physical realization.
Read each result with its experimental contrast.
The survey’s Table 16 assembles 14 reported comparisons. Each comparison uses its own tasks, information, control interfaces, and preparation.
| Study and source | Reported contrast | What to preserve when interpreting it |
|---|---|---|
| Demo-JEPA [22] | Action-head and planning interfaces favor different task groups: the head performs better on known real tasks; planning performs better on unseen tasks. | Keep task familiarity, simulation versus real execution, and the action interface explicit. |
| TraceFlow [390] | Ordered successes change from 21/50 to 39/50 on one real packing task; aggregate simulated performance is unchanged. | An ordering benefit in one setting does not establish a general aggregate success gain. |
| LMPC [117] | A trained successor improves reported success from 39.4% to 66.3% on held-out teaching tasks. | Successor training occurs between sessions. This contrast is distinct from changing context within one fixed checkpoint. |
Supplement S1 records protocols and source locations for the reported comparisons.
Source: Table 16 (PDF p.70); Section 7.2; Supplement S1.
Report the information, effort, and limits behind the score.
- Information and preparation: what the demonstration exposes, available observations and action labels, retrieval pools, prior training, controller access, and any correspondence assistance.
- Controls and transfer: changed and fixed factors, physical and memory reset rules, task split, attempt budget, randomness control, and uncertainty.
- Behavior and physical outcome: adherence to the taught requirement alongside completion, failure modes, and diagnostic intermediate changes.
- Total effort: demonstrations, exploration, failed attempts, human corrections, validation, and preparation costs. Distinguish inference latency from feedback-to-action delay and end-to-end completion time.
- Reuse scope: performance on repeated and new tasks, the conditions under which retained information remains applicable, and the cost of maintaining it.
Source: Section 7, including Section 7.4 on teaching effort, response delay, and reuse cost.