The world of robotics is buzzing with innovation, and LeRobot v0.6.0 is here to revolutionize how robots learn. This exciting new release focuses on completing the robot learning loop, introducing powerful features that enable policies to imagine future actions, reward models to gauge success, and a streamlined deployment process that transforms failures into valuable training data.
From depth sensing and VLM-powered dataset annotation to custom video encoding and cloud training, this update brings a suite of enhancements designed to make robot learning more intuitive and efficient. Plus, with a leaner install and six new simulation benchmarks, getting started and measuring progress has never been easier.
World Models: Policies That Imagine the Future
A fundamental question in robotics is whether world models truly enhance robot policies. LeRobot v0.6.0 introduces three pioneering policies to explore this very idea, each designed to help robots “imagine” the future as part of their training process.
These models each employ unique, cost-effective approaches to integrate future imagination without hindering performance:
- VLA-JEPA: This model trains a compact VLA (built on Qwen3-VL-2B) to predict future latent space frames based on its actions. Crucially, the world model is discarded during inference, providing world-model supervision at zero additional inference cost.
- LingBot-VA: Taking imagination a step further, LingBot-VA is an autoregressive video-action model that predicts future video and actions collaboratively, chunk by chunk. It continuously incorporates real observations to keep its predictions grounded, and you can even save and compare imagined vs. actual outcomes.
- FastWAM: This model explores whether world action models truly need test-time future imagination. It combines a powerful ~5B video-generation expert with a compact action expert, learning to “dream” its own rollouts but skipping the dreaming entirely at inference for direct action denoising.
The Growing VLA Model Zoo
LeRobot’s collection of Vision-Language-Action (VLA) models continues to expand, offering more robust and versatile options for various robotic tasks. Each new addition brings unique capabilities and optimizations:
- GR00T N1.7: We’ve updated our NVIDIA GR00T integration to the newest open generation, N1.7, which features Cosmos-Reason2-2B (built on Qwen3-VL) and a flow-matching action head. This integration is parity-tested against NVIDIA’s original implementation, ensuring consistent inputs and outputs.
- MolmoAct2: The Allen Institute for AI’s MolmoAct2 is now fully ported into LeRobot, supporting fine-tuning (full or LoRA), evaluation, and real-robot deployment. With ready-made checkpoints and calibration correction, it can run zero-shot on SO-100/101 robots.
- EO-1: Contributed by one of its paper’s authors, EO-1 is a VLA pretrained on interleaved vision-text-action data. It boasts a Qwen2.5-VL-3B backbone with a flow-matching action head, integrating seamlessly with the standard LeRobot training workflow.
- Multitask DiT: This policy brings the TRI Large Behavior Models recipe to LeRobot, employing a ~450M-parameter diffusion transformer conditioned on CLIP vision and language embeddings. It enables a single model to learn numerous tasks specified via natural language.
- EVO1: Demonstrating that VLAs don’t have to be massive, EVO1 packs its policy into a compact 0.77B parameters. Featuring an InternVL3-1B backbone and a flow-matching action head, it’s efficient enough for real-time fine-tuning and execution on modest GPUs.
Reward Models: Knowing When Your Robot Succeeds
Detecting success and estimating progress are crucial, yet often overlooked, elements in the robot learning loop. LeRobot v0.6.0 now provides a unified reward models API (lerobot.rewards), mirroring the policies API, with four powerful reward models accessible through a single interface. Alongside the existing HIL-SERL reward classifier and SARM, we introduce two exciting new additions:
- Robometer: This pretrained, general-purpose reward model can be pointed at any LeRobot dataset to score task progress and success directly from raw video and a language instruction. Built on Qwen3-VL-4B, it requires no task-specific training, having been trained via trajectory comparisons over a dataset of more than one million robot trajectories.
- TOPReward: Taking a fully zero-shot approach, TOPReward operates without any reward weights. It leverages an off-the-shelf VLM (Qwen3-VL) to read the log-probability of the token “True” given the trajectory video and task instruction, effectively turning any capable VLM into a reward function.
Both Robometer and TOPReward come with labeling scripts that generate per-frame progress curves within your datasets. These curves are invaluable for reward-aware behavior cloning (RA-BC), dataset quality inspection, and creating progress-overlay videos, significantly enriching your data for more effective robot learning.
Datasets: Faster Loading, Richer Data
This update significantly enhances dataset management, offering faster loading, greater flexibility in video encoding, and richer annotation capabilities. Now, your dataset operations are more efficient and adaptable than ever before, supporting complex robotic tasks.
Your Codec, Your Rules
Recording is no longer limited to a single hard-coded codec. The new --dataset.rgb_encoder.* options provide full control over encoding parameters like codec, quality, pixel format, GOP, and presets. The vcodec=auto option intelligently probes for hardware encoders (NVENC, VideoToolbox, VAAPI, QSV) before defaulting to the efficient software AV1 encoder. For existing datasets, a single command can re-encode all videos to your preferred format, such as H.264, ensuring optimal storage and playback.
Depth Support, End to End
Integrating depth sensing is now seamless. Simply plug in an Intel RealSense camera and set use_depth: true, and LeRobot will record depth maps end-to-end. Captured in millimeters, these are compressed into compact 12-bit depth video streams alongside your RGB cameras, then decoded back to physical units during training. Depth renders live during recording and in lerobot-dataset-viz, and it’s compatible across a wide range of robots including SO-100/101, Koch, OpenArm, reBot, and Unitree G1.
Language Annotations at Scale
LeRobot datasets now natively support rich language annotations, moving beyond simple task strings per episode. This includes timestamped subtasks, plans, memory, corrections, speech, and per-camera VQA pairs. The new lerobot-annotate CLI can automatically populate these annotations using a VLM that watches your episodes. A YAML recipe layer then renders these annotations into chat-style training messages at sample time, providing the exact data needed for tomorrow’s long-horizon, talking robot policies.
Up to 2x Faster Data Loading
Training on video datasets is now significantly faster, with improvements of up to ~2x out of the box. Multi-camera frames decode in parallel, and dataloader workers now ship compact uint8 frames, reducing inter-process memory by 4x. Persistent workers keep decoder caches alive across epochs, further boosting performance. Loading a subset of a large dataset has been optimized from minutes to milliseconds, and sampling is now deterministic and resumable, allowing interrupted trainings to restart sample-exact.
Benchmarks: One CLI to Evaluate Them All
LeRobot v0.5.0 set the stage as an evaluation hub, and v0.6.0 makes it the definitive platform with six new simulation benchmarks. All are conveniently runnable through the unified lerobot-eval CLI, each featuring comprehensive documentation, a dedicated Docker image, and a SmolVLA baseline checkpoint smoke-tested in CI.
Simulator backends may require specific system dependencies, but each docs page provides the exact recipe, and ready-made Docker images simplify setup. This expansion, alongside LIBERO, Meta-World, and NVIDIA IsaacLab-Arena, brings the total to nine benchmark families under one roof. A new “Adding a New Benchmark” guide details how to integrate your own, while parallel evaluation is now up to 2x faster with async vectorized environments.
Training & Inference: Streamlined Workflows
Deployment and data collection are now more structured and efficient, with dedicated tools and advanced strategies for training and inference. These improvements ensure that your policies can be deployed reliably and that valuable data is captured effectively.
lerobot-rollout: Deployment Gets Its Own CLI
Policy deployment, once a workaround, now has its own dedicated workflow with the new lerobot-rollout CLI. This tool offers pluggable strategies and inference backends, including Real-Time Chunking for compatible VLAs. The base strategy simply runs the policy, while sentry records continuously, rotating episodes and uploading to the Hub.
The highlight strategy keeps a ring buffer, saving the last N seconds when you hit a key, ensuring no interesting moment is lost. episodic mirrors the classic episode/reset recording workflow. And dagger transforms deployment into an interactive data collection process: you observe your policy, intervene with a leader arm to correct errors, and then hand control back. Actuated leaders are driven to the follower’s pose before handover, ensuring a smooth, jerk-free transition, and every correction frame is automatically tagged for improved training data.
Source: Hugging Face Blog
