Cross-Embodiment Policy Transfer

Cross-painting

A rendering copy of the Franka keeps its colliders, so it stays parked 20 m above the cell whenever physics advances. Before each policy request it is moved to the Franka’s base position and posed at the virtual joints. The Kinova’s visuals are hidden, the exterior camera renders, and the scene is restored. This is ground-truth-assisted simulator rendering, not image inpainting, so the policy also sees whatever the Kinova would hide. A real robot would need Mirage’s mask-and-inpaint step [1], which this project does not test.

Timing

In these simulator assets, the Franka’s gripper reaches 95% closure in 2 control steps and the Kinova’s in 23. Holding each policy action for 12 steps gives the slower gripper time to close. The cost is pace: a Kinova episode may run 600 s of simulation where the Franka gets 50 s for the same 750 policy actions.

Lift

Once the policy has held a close command for more than 24 control steps, on a grasp that began with the banana touching neither the bowl nor inside it, the adapter raises the commanded hand 15 mm over 24 steps and lowers it again when the policy opens. That eligibility check uses privileged simulator information: banana–bowl contact and the simulator’s geometric in-bowl test, both from the previous control step. A real Kinova would need its own sensing for both. The lift is a fixed rule, not learned; it was tuned on one development scene and tested once before the freeze.

A matched check held robot state fixed and swapped one input at a time. Its predeclared tilt measure was negative. An exploratory analysis instead found that changing the exterior image moved the 8-step target by 19–116 mm, compared with 4–12 mm repeat noise; changing the wrist image or mapped state stayed within that range. With painting, the target discrepancy was 7.11 mm, against 6.38 mm repeat noise and 74.38 mm unpainted. These checks motivated the intervention; they do not establish that appearance explains every failure.

P1 is P0 fine-tuned on 27 Franka wins approved by the reviewer (it kept 5 other scorer wins out of training), 6,367 frames from 32 practice layouts over two policy seeds. A layout won under both seeds appears twice, and a winning episode keeps its failed sub-attempts. Fine-tuning ran 2,000 steps at batch 32 and learning rate 1e-4 on the projector and diffusion head. The locked selection rule ran P0 and the checkpoints saved at 500, 1,000, 1,500 and 2,000 steps on 8 validation layouts (3000–3007). After the reviewer’s adjudication they won 6, 0, 1, 3 and 3 of 8. No fine-tuned checkpoint beat P0. The rule takes the most wins, ties to the earlier checkpoint, so P1 is checkpoint 1,500, labelled “P1, no A-validation improvement”: the selected fine-tuned checkpoint, not an improved policy. Its 3 of 8 is selection-biased. The after test completed; all 24 P1 episodes and two P0 canaries were copied and verified locally.

Two-sided exact McNemar over the paired layouts. P0 won 7 of 12 on each robot, so P1 can gain at most 5 layouts; 5 to 0 gives p = 0.0625, and only a decline can reach p < 0.05. The after test ran in a separate session from the before test; checks included full pins and a rerun of Franka P0 on layouts 4000–4001, which gave 0/2 v5.1 wins, matching the before half layout by layout. All 12 pairs per robot were scored. The canary trajectories differed, so matching failure outcomes do not rule out drift. A later CPU replay of the locked scorer over all 48 saved before and after episodes reproduced every verdict, with matching scorer and trace hashes, so the automated grading was the same before and after.

How it ran. The Kinova ran P1 through the unchanged painted adapter on layout 4003, policy seed 0, for 3,600 controls (240 simulated seconds). While the gripper was closed, Nemotron looked at the Kinova’s real exterior camera and its wrist camera; the simulation paused for each call, and P1 kept acting between checks. Counting the runner’s ticks from 0, it said WAIT 27 times, every 12 ticks from the first gripper close at tick 672 to tick 984. At tick 996 it said READY and the controller held the arm. At tick 1005 it confirmed READY from fresh views, and the open command executed at control 1006; the controller kept the gripper open from then on. The 29 calls cost $0.0223 from returned token counts, with 56.5 s of model wait.

The unchanged v5.1 scorer, the rule behind every result above, scores a PASS at control 1071, with the banana still on the bowl at the end. At the open, the banana’s root was 4.98 cm from the bowl centre, measured horizontally, and its centre 10.5 cm above the bowl top; Nemotron never saw these evaluation-only numbers. The runner’s older live v4 checker reported no success: it ruled the open ineligible. v5.1 stays the primary rule because the before and after results were scored with it.

Why a vision check first. Instead of the intent demo’s robot run (see “Before ABBA”, below), I proposed DROP: a Nemotron vision model watching P1 carry the banana on the Kinova and deciding when to let go. The vision model, Nemotron 3.5 Super VL, had named the banana and the bowl in 12 of 12 start frames, an easy test with only two objects on the table. On saved frames from the P1 Kinova runs, it passed a small screen of held, approaching and already-resting cases after two failed prompt rounds, but near the rim it still said “ready to release” in 4 of 4 cases with the banana 5–8 cm off the bowl’s centre and 2 of 4 at 8–12 cm (69 calls, $0.0522). That was not selective enough to trust with the gripper on its own.

• One assisted run on one selected layout. It is not a success rate, not a controlled or paired comparison, and not a repair of training; the results above stand.
• Picked, not drawn. 4003 was the only P1 Kinova layout with a long, centred carry over the bowl in the after test. That earlier run is not a matched pair: the intervention changes everything after the hold.
• “Central” is not a measurement. Nemotron called the banana central; it was about 5 cm off.
• The controller did part of the work. The hold and the latched open are controller behaviour around the model’s decisions.
• What P1 alone would have done is unknown. After the forced open, P1’s own gripper command also turned to open from control 1096, after 90 controls of the latched open; why it switched isn’t known.

Before launch, the package had 43 CPU tests and an implementer review, but no independent review of the final runner. Afterward, an independent AI review re-hashed all 298 copied evidence files, reproduced the frozen score and checked the cost ledger. The RGB-D capture and intent-control robot experiments never ran. Across all Nemotron experiments, 439 calls totalled $0.1441 at prices applied to returned token counts; this is not a provider billing receipt.

Nemotron’s job here is to turn a plain request into a structured task spec a program can check. NVIDIA Nemotron 3.5 Lightning (nvidia/Nemotron-3_5-Lightning) on Nebius Token Factory takes a request such as “put the fruit away” and returns the object and the target, named from the scene’s own objects, where the object should end up, and the conditions a success has to meet, such as whether the gripper must be opened on purpose and whether the bowl’s rim counts.

The parsing ran; the robot run did not. The spec also carries the one sentence the robot’s policy should get and whether this robot should try at all, and a fixed local rule has to agree before anything moves. Each request was asked three times, and the robot would only get an answer given every time. With my first prompt, Nemotron declined “put the fruit away” as naming an object not in the scene, refused to move the bowl as expected, and its three repeats split on “put the banana beside the bowl, on its left”. A second prompt, written after seeing those answers and disclosed as such, gave the same answer on all three repeats of all three requests. The planned run of the unchanged policy (P0) on the Franka with those specs was never performed; 87 offline tests are all the checking it got.

An earlier version of the compile step, the intent contract, was tested on its own: 11 instructions, three runs each, from RoboLab’s own wording and paraphrases to five adversarial ones. On the adversarial five, all 42 field checks matched a faithful reading: “the apple” came back as not in the scene, “don’t let go of it” turned off the release requirement, and “the edge of the bowl” became the rim. It also exposed a real gap: it read “in the bowl” literally, so the rim didn’t count in any of the 18 runs on the six ordinary instructions, while the project’s win rule counts it.

Is a bigger model better here? Not on this test. I sent the same 11 instructions, five of them traps like “put the banana in the bowl but don’t let go of it”, three times each to four Nemotron models on Token Factory, with the rule for picking a winner written down before the first call. Lightning, Super (120B) and Ultra (550B) got every trap right and gave the same answer on every repeat; Nano ignored “don’t let go” all three times. On a tie the cheapest wins, so Lightning stays: 33 calls for a fifth of a cent. Five traps can’t tell the top three apart, so this doesn’t show the bigger models are no better on harder requests.

I also tried using Nemotron to grade runs from their text logs; on 24 runs it did no better than always guessing “success”, so the locked scorer and an independent reviewer do the grading. The 178 Nemotron calls before the comparison cost $0.02, per the Token Factory usage page.

Nemotron’s second job, deciding when the Kinova lets go, did reach the robot: that’s ABBA, above.