A robot-world video can look convincing while leaving a more basic question unanswered: did the generated motion follow the action sequence that conditioned it? A smooth arm movement and a recognizable grasp are evidence about the image. They do not, by themselves, establish that the model predicted the consequence of a particular command, especially when the command differs from familiar demonstrations or should lead to failure. That distinction matters when developers consider generated rollouts for policy evaluation or training.
NVIDIA’s Nov. 5 article describes four open-weight Cosmos 3 Nano checkpoints. Its publication date is not the release date: the cards mark V1.0 as Oct. 20, 2026. Transfer-Auto supports multi-view driving-data generation; Sim-Auto targets closed-loop driving simulation; Sim-Transfer generates visual observations from simulator control sequences; and Sim-Bimanual generates video conditioned on robot and camera actions.
The Sim-Bimanual card lists robotic policy evaluation and verification, reinforcement-learning training, and synthetic demonstrations among its uses. Its reported video/reference measures include PSNR, SSIM and LPIPS, which the card describes as proxies for action following. Those measures can say something about resemblance between generated and reference frames; they do not answer every question about whether the action-conditioned outcome is right. This is a limit of what the measures establish, not evidence that the checkpoint fails them.
Testing the action, not just the image
The WorldEcho v2 preprint, revised Oct. 23, 2026, makes the comparison more direct. It evaluates 50 RoboTwin tasks using 1,750 queries. For a query, the researchers compare the generated rollout with a simulator replay of the same action from the corresponding initial state. They check visual integrity and align the robot’s end-effector trajectory, rather than treating a plausible-looking video as sufficient. The query set includes five distinct categories: Demonstrated Action, Cross-State Replay, Local Perturbation, Policy Rollout, and Feasible-Space Sampling.
These off-expert query types are designed to stress-test the model’s ability to generalize beyond standard training data. By introducing variations like cross-state replays or local perturbations, researchers can determine if the model maintains physical consistency when faced with novel or slightly altered inputs. WorldEcho reports that these off-expert queries increased integrity-gated error by 0.022 to 0.048 meter-equivalent units specifically across the six expert-trained models evaluated in that study. In its Cosmos3 row, the expert-demonstration condition recorded 0.0989 integrity-gated error, 0.0572 raw NDTW and an 82.51% visual pass rate. With expanded action coverage, those values were 0.0799, 0.0419 and 85.09%. Lower error is better; the pass percentage is a separate visual measure. The paper describes the comparison as study-specific: its baselines received 40,000 fine-tuning updates, and its Cosmos3 initialization was Cosmos3-Nano. These numbers are not results for the identical interactive checkpoint released by NVIDIA.
The MiraBench preprint, submitted May 28, 2026, separates three questions: physical adherence, action-following fidelity, and whether a model preserves failure when an action should fail. The authors argue visual fidelity is a poor proxy for action fidelity and test failure-inducing actions, such as insufficient grip force, premature release, grip carry slip, contact oscillation, wrist tilt, and approach overshoot. Its 12 evaluated configurations include Cosmos-GR1 2B and 14B, not the four Cosmos 3 Nano checkpoints. MiraBench therefore offers a different evaluation framework, not a direct assessment of the October releases.
Together, the papers clarify why a video-quality score, a command-following test and a failure-preservation test should not be collapsed into one claim. A model could produce a visually coherent outcome while missing the commanded trajectory; nominal action following also does not, by itself, show how a model responds to an input that should produce failure. Each test catches a different kind of gap. By isolating these variables, researchers can examine distinct limits in model evaluations without treating any one score as complete.
Neither preprint establishes that the released Cosmos 3 Nano checkpoints fail these tests, and both remain preprints. The practical implication is narrower: teams using generated rollouts should ask what their metric actually measures. Visual resemblance, action-specific outcomes and failure preservation are separate evaluation questions. A model card’s proxy metrics can characterize parts of the system, while simulator-replay or failure-inducing tests can probe different properties. Those results should be reported as complementary evidence, not treated as interchangeable proof of reliability or safety.