Vision-Language-Action models, end to end

A vision-language-action model (VLA) is a robot policy built on a vision-language model. It looks at camera images, reads an instruction like “put the cube in the bin”, knows where its own joints are, and outputs motor commands. The idea that made these work is that you don't start from zero. You take a model that has already learned from the internet what objects, words and scenes are, attach an action output to it, and teach it to move with robot demonstrations.

RT-2 named the category in 2023. Since then OpenVLA, Octo, π0 and π0.5, GR00T N1, Helix, Gemini Robotics and SmolVLA have settled into roughly the same recipe, with differences in the details. This post is how I think about that recipe, as the eight stages in the graph below, from internet pretraining to a policy running in a control loop on the robot.

In words: given the latest images, the instruction and the joint state, predict the next H actions (an action chunk). Everything below is about getting that one function right, then making it run fast enough on the robot.

Scroll, and the 3D view follows along. Point at a stage in the article or a block in this map to load its scene, and click to jump to it. Drag the view to orbit.

Everything starts with a vision-language model (VLM) trained on web-scale image–text data. A vision encoder cuts each image into patches and turns them into tokens. A language model then reads those tokens alongside the text and learns to predict what comes next. π0 starts from PaliGemma (3B). OpenVLA uses the Prismatic VLM, which puts Llama 2 7B on top of fused DINOv2 and SigLIP features. GR00T N1 uses NVIDIA's Eagle-2, and Gemini Robotics builds on Gemini 2.0.

Why bother, when the goal is a robot? Because robot data is tiny and expensive, while the web is huge. A web-trained VLM already knows what a mug is, which drawer is “the top one”, and that a banana goes in a bowl. RT-2 showed that this transfers: co-trained with web data, its robot could follow instructions about objects and ideas that never appeared in its robot data.

The catch is that most VLMs were trained for captions and question answering on low-resolution images. They are weak at precise spatial work such as pointing, depth and small objects, which is exactly what manipulation needs. That's why newer recipes add embodied-reasoning data (points, boxes, trajectories, subtask labels) before or during robot training. Gemini Robotics-ER and π0.5 are good examples.

input

image patches + text tokens

objective

next-token prediction (plus contrastive pretraining for the encoder)

scale

billions of image–text pairs

you get

a backbone that understands scenes and language

This is the expensive part, and I think it's the part that decides who wins. The standard method is teleoperation: a person drives the robot, and every timestep is logged with the camera images, the joint state and the commanded action, plus a language label for the episode.

The scene shows an ALOHA-style leader–follower rig. The operator moves two small leader arms, and the follower arms copy their joint angles at 50 Hz. It's cheap, it feels natural for two-handed work, and the actions come out in exactly the robot's own joint space. Other options are VR teleop (common for humanoids), space mice, GELLO-style puppets, handheld grippers like UMI that record demos without a robot, and plain human video.

A dataset is a pile of these episodes. Open X-Embodiment pooled more than 1M trajectories from 22 robot types. DROID collected 76k Franka trajectories (about 350 hours) across 564 scenes. π0's own mixture had over 10,000 hours from 7 robot configurations.

Quality matters more than count. You want consistent operators, demos without hesitation, wide variety in scenes and starting positions, and some recovery behaviour (start from a mistake and show the fix). Mixed-quality data is fine for pretraining but hurts during post-training.

obs

3–4 RGB cameras (top + 2 wrist), joint positions

action

14 target joint positions (2 × 6 joints + gripper)

rate

50 Hz, episodes of roughly 10–60 s

format

LeRobotDataset (Parquet + MP4) or RLDS

Now the VLM learns to act. There are two main ways to get actions out of it.

• Discrete tokens. Split each action dimension into 256 bins and have the language model predict them as tokens (RT-2, OpenVLA). It's simple and reuses the language head. But it needs one forward pass per token, which is slow, and the bins are coarse for fast, dexterous control. FAST (π0-FAST) fixes much of this by compressing whole action chunks with a DCT and BPE before tokenizing.
• A continuous action head. A separate network outputs continuous actions with diffusion or flow matching (Octo, π0, GR00T N1, SmolVLA). In π0, a ~300M-parameter action expert sits next to PaliGemma, attends to its tokens layer by layer, and turns noise into a 50-step action chunk in about 10 integration steps.

The 3D view follows the architecture figure from the GR00T N1 paper, which I find the easiest way to see the whole thing at once. It plays as a guided tour: the camera moves to each part as it runs and a subtitle explains it, and you can click a step number to jump. In short, left to right:

1. Vision-language side (top row). The camera image goes through a vision encoder and the instruction through a text tokenizer. The pretrained VLM (Eagle-2) turns both into tokens. GR00T N1 keeps its language model frozen (❄) and reads features from a middle layer (layer 12) rather than the last one.
2. Robot side (bottom row). The joint state q_t and a noised action chunk go through small embodiment-specific encoders (green). These are the only parts that differ between robots, so one network can serve arms, bimanual rigs and humanoids.
3. DiT blocks. A stack of transformer blocks alternates cross-attention, which reads the VLM tokens, with self-attention over the state and action tokens.
4. Decoder and K iterations. An embodiment-specific decoder turns the result into H = 16 actions. The output goes back in as the next, less noisy input, K = 4 times (flow matching), and the final chunk drives the robot.

Action chunking itself (predicting H steps at once, from ACT) is the other key idea. It smooths motion, copes with demos that solve a task in different ways, and lets a slow model drive a fast control loop.

Pretraining mixes everything: single arms, bimanual rigs, mobile manipulators and humanoids, each with a different action space, padded to a common size. Some web data stays in the mix so the VLM doesn't forget what it knew (co-training). π0.5 also predicts the next subtask in text before acting, and “knowledge insulation” stops the action expert's gradients from degrading the VLM.

in

images + instruction + joint state (+ noised actions)

out

a chunk of H actions: 16 in GR00T N1, 50 in π0

steps

K = 4 (GR00T N1) to ~10 (π0) flow-matching passes

loss

flow matching or diffusion (action head); cross-entropy (tokens)

size

GR00T-N1-2B: 2.2B, 1.34B of it in the VLM · π0: ~3.3B

The pretrained VLA is a generalist that does many things reasonably well. Post-training specializes it to one robot and one family of tasks with a smaller, cleaner dataset. That might be tens to hundreds of hours for hard, dexterous tasks like folding laundry, or a few hundred demos for simple pick-and-place.

Full fine-tuning gives the best results if you have the compute. LoRA on the backbone (OpenVLA), or keeping the VLM frozen and training only the robot-specific encoders, DiT and decoder (GR00T N1.5, the mini figure behind the table in the 3D view), gets most of the way there on a single GPU. OpenVLA-OFT showed that the fine-tuning recipe matters as much as the base model: parallel decoding, action chunks and continuous actions with an L1 loss made it much faster and more accurate.

The things that tend to move the success rate:

• Match the action space and control rate to the robot (joint vs. end-effector, absolute vs. delta).
• Normalize each action dimension with quantiles, not min/max. One outlier can ruin min/max.
• Curate: drop failed or hesitant demos, and balance the task variants.
• Hold out whole scenes or objects for evaluation, not random frames.

data

50 to 1,000+ demos per task family

method

full fine-tune, LoRA, or frozen VLM + trained expert

watch

overfitting to one scene, forgetting language

Imitation learning tops out around the quality of the demonstrations, and it never learns from its own mistakes. Small errors compound into states the demos never covered. RL closes that gap. The graph marks it “optional, rising” because 2025 is when it started working for VLAs at scale.

• In simulation: many parallel environments and a sparse success reward. SimpleVLA-RL applies GRPO-style RL with a binary success reward to an OpenVLA-OFT policy on simulation benchmarks. That's the 3D view: nine environments, failures flash, and the success rate climbs each iteration.
• On real robots: learn from the robot's own attempts plus human corrections. π*0.6's Recap trains a value function on that experience and conditions the policy on the advantage. On some of the hardest tasks (espresso, laundry, box assembly) it more than doubled throughput and roughly halved the failure rate.
• Steer instead of fine-tune: DSRL runs RL in the latent noise space of a frozen diffusion or flow policy, so it changes behaviour without touching the weights.

The hard parts are reward design for real tasks, resets, and safety while exploring. That's why most real-world RL today starts from a strong imitation policy and keeps a human in the loop.

Evaluating a robot policy is harder than training one. Offline metrics such as action error on held-out demos barely correlate with real success, so you need rollouts.

• Simulation benchmarks: LIBERO, CALVIN, SimplerEnv (built to track real Google Robot and WidowX results) and RoboCasa. They're cheap and reproducible, but many are near saturation, and the sim-to-real gap never fully goes away.
• Real trials: a fixed task suite with randomized starting positions, run A/B against a baseline and scored by success (often with partial credit). RoboArena spreads evaluation across labs with pairwise comparisons.

The statistics matter more than people think. 16 successes in 20 trials is “80%”, but the 95% Wilson interval is 58–92%. Two policies 10 points apart on 20 trials are usually indistinguishable. The tally boards in the 3D view show this interval live.

Every failure gets triaged (perception, grasp, recovery, language grounding) and fed back as targeted data or an RL signal. That's the dashed loop in the graph. Evaluation also gates the optimized model: if a quantized build regresses, it goes back.

A 3B-parameter model in bf16 is about 6.6 GB of weights. It has to run on the robot, because cloud inference adds network jitter a control loop can't absorb. It also has to be fast enough to keep the action queue full on something like a Jetson AGX Orin (up to 275 sparse INT8 TOPS) or the newer Jetson Thor.

• Quantize: bf16 to FP8 or INT8, and INT4 (AWQ, GPTQ) for the language-model weights. Check layer by layer; the action head is usually more sensitive than the backbone.
• Do less work: fewer flow steps (10 to 5), fewer vision tokens (lower resolution, token pruning), and cache the image + text prefix so only the action expert reruns during denoising.
• Compile: export to TensorRT (or use torch.compile with CUDA graphs), use fused attention kernels, and fuse small ops such as the Q, K and V projections.
• Split the model: run a slow “System 2” VLM at roughly 7–10 Hz and a fast “System 1” policy at 120–200 Hz (GR00T N1, Helix).

Then re-evaluate. Compression that looks free on benchmarks can quietly cost a few points of success on the real robot.

On the robot, the policy runs a receding-horizon loop: read the images and joint state, run the VLA, get a chunk of 50 actions, and execute them at 50 Hz while the next chunk is computed. In the 3D view, the dotted path ahead of each gripper is the current chunk. It refreshes every 0.5 s, so the robot only executes the first half before replanning.

• Latency vs. chunk boundaries. If inference takes 100 ms and the robot waits for it, the arm stalls. If it switches chunks blindly, the arm jerks. Temporal ensembling (ACT) blends overlapping chunks. Real-time chunking (RTC) inpaints the next chunk so it agrees with the actions already committed.
• Safety lives below the policy. Joint limits, velocity and torque clamps, workspace bounds and an e-stop sit in the controller, never inside the model.
• Log everything. Deployment failures are the best training data you'll ever get, so the loop closes back to stage 02.

loop

observe → infer → act, then repeat

rates

a new chunk every 0.5 s · joint commands at 50 Hz

chunk

H = 50 actions (1 s); execute 25, then replan