How it works
AI Foundation Models in Self-Driving
A foundation model for driving is a single large neural network, pre-trained on vast amounts of driving video, maps and language, that a company then adapts to many tasks: driving the car, simulating the world around it, and judging its decisions. Since 2024 the leading developers have rebuilt their software around such models, borrowing the recipe of large language models.
What is a foundation model for driving?
The recipe comes from language AI. Train one very large model on broad data, then adapt it to specific jobs, instead of building a separate small network for each job. Applied to driving, the broad data is years of fleet video, map data and human text, and the jobs are perception, prediction, planning, simulation and self-evaluation.
In December 2025 Waymo described its Waymo Foundation Model, trained with Google's Gemini so that it carries Gemini's knowledge of the world. The same model sits behind three systems: the Driver that steers the car, the Simulator that tests it, and a Critic that grades its decisions. At CVPR in June 2026 Tesla described its own approach: a single vision encoder consuming all eight cameras and emitting driving actions alongside 3D occupancy, object detections and segmentation.
This is a step beyond the first end-to-end networks, which learned one narrow mapping from pixels to steering. A foundation model is general, reused across tasks, and knows things no driving log teaches, such as what an ambulance is or what a hand-written detour sign means.
What is a world model, and why does driving need one?
A world model predicts what happens next. Given the scene so far and a candidate action, it generates the next few seconds of sensor data, the way a video generator continues a clip. Driving companies use this in two ways: as a generator of rare scenarios for training and testing, and as a planner's imagination, letting the car try out outcomes before committing.
The pressure behind world models is rare events. A fleet can drive a hundred million miles and still see a tornado or a flooded underpass only a handful of times. A world model can produce thousands of variants of that scene from a prompt. And because the car's own decision changes what the model generates next, the simulation is closed loop, which replaying a recorded log can never be.
Wayve released its first driving world model, GAIA-1, in 2023 and has shipped a new generation roughly every year since; GAIA-4, announced in August 2026, puts its AI Driver inside the loop and generates radar as well as video. NVIDIA's Cosmos family, launched in January 2025, was trained on about 20 million hours of video. In February 2026 Waymo introduced the Waymo World Model, built on Google DeepMind's Genie 3, which generates matching camera images and lidar point clouds. Chinese developers split into camps: Huawei and NIO built their driving stacks around world models, while XPeng published its X-World model in April 2026 as the simulator behind its production driver.
| World model | Developer | First shown | What it generates | Main use |
|---|---|---|---|---|
| GAIA-1 to GAIA-4 | Wayve | 2023, latest Aug 2026 | Video, now radar | Closed-loop evaluation of the AI Driver |
| Cosmos | NVIDIA | Jan 2025 | Physics-aware video | Synthetic data for any developer |
| NIO World Model | NIO | 2024 | Trajectories on the car | On-vehicle planning |
| WEWA | Huawei | 2025 | Cloud scenarios plus on-car actions | Training the ADS 4 driver |
| Waymo World Model | Waymo | Feb 2026 | Camera and lidar | Simulating rare conditions |
| X-World | XPeng | Apr 2026 | Multi-view video | Closed-loop testing and reinforcement learning |
Read: Sensing: the cameras, lidar and radar these models imitate →
What is a vision-language-action model?
A vision-language-action model, or VLA, takes in camera images, reasons about them in language, and outputs a driving action. The language step is the novelty. Because the model was first trained on text, it can name what it sees, think through an unusual situation step by step, and explain its choice afterwards, in words a person can read.
Waymo's EMMA paper in October 2024 showed the idea: it recast driving tasks as questions put to a Gemini-class model, which answered with trajectories and detections. NVIDIA turned it into a product at CES in January 2026 with Alpamayo 1, an open 10-billion-parameter reasoning model, followed in June 2026 by the 34-billion-parameter Alpamayo 2 Super aimed at Level 4 robotaxis. Mercedes-Benz's CLA was announced as the first production car on the platform.
China moved fastest to put VLAs in consumer cars. Li Auto rolled a VLA driver out to its AD Max fleet in September 2025 and showed a successor, MindVLA-o1, in March 2026. XPeng's VLA 2.0 entered mass production in 2026 and took over half of the assisted-driving miles across its fleet within a month, while XPeng began public robotaxi tests on the same software.
How is reinforcement learning changing how cars are trained?
Most driving networks learn by imitation: watch human drivers and copy them. Imitation has a ceiling, since the model can be no better than the people it copies, and it is weak exactly where data is thinnest, in rare situations where a small error compounds.
Reinforcement learning adds practice. The model drives in simulation, is rewarded for good outcomes and penalized for bad ones, and improves beyond its teachers. That only works with a good simulator, which is why world models and reinforcement learning arrived together. Tesla's FSD release notes in 2026 describe an upgraded reinforcement learning stage trained on hard cases pulled from its fleet. XPeng uses X-World for online reinforcement learning. Waabi trains its truck driver as a student inside Waabi World, a generative simulator it calls the teacher.
Waymo adds a third model to the loop, a Critic, which judges the Driver's proposed behavior against safety rules. Grading by a model rather than only by rewards is one way companies try to keep a learned driver inside bounds a person can specify.
Read: Robotaxis: where these systems carry passengers today →
Who is building what?
Every major developer now claims a foundation model, but what is shipped on cars and what is research differ widely. The table separates the two.
| Developer | On the road today | Foundation model work | Approach |
|---|---|---|---|
| Waymo | Driverless robotaxis in US cities | Waymo Foundation Model (Gemini), Waymo World Model, EMMA research | One model behind Driver, Simulator and Critic, with lidar, radar and cameras |
| Tesla | FSD (Supervised) and early Cybercab rides | Video foundation model, eight-camera encoder, reinforcement learning stage | Cameras only, end-to-end |
| NVIDIA | Supplies chips and models, not cars | Cosmos, Alpamayo 1 and 2 Super, AlpaSim | Open models any automaker can adopt; Mercedes CLA first |
| Wayve | Supervised robotaxi service with Uber in London since September 2026 (safety driver aboard); AI Driver licensed to Nissan for 2027 cars | GAIA-1 to GAIA-4, AI Driver | End-to-end driver validated inside a world model |
| Mobileye | Driver assistance in tens of millions of cars | Compound AI | Several parallel systems; end-to-end is one path, not the only one |
| Li Auto, XPeng | Assisted driving on consumer cars in China | VLA drivers, X-World | VLA on the car, world model in the cloud |
| Huawei, NIO | Assisted driving on partner and own cars in China | WEWA, NIO World Model | World model camp |
| Waabi | Driverless trucks in Texas | Waabi World and Waabi Driver | Teacher simulator trains a student driver |
What are the limits?
Size is the first. A 34-billion-parameter reasoning model does not run on today's in-car computers at the speed driving demands, so what reaches the vehicle is usually a smaller model distilled from the large one, which gives back some of the capability.
Trust is the second. A language model can produce a confident, fluent explanation that is not the actual reason for its action, and generative simulators can produce scenes that look right but behave wrongly. Regulators who could once inspect a perception stage or a rule now face a single learned system, and the safety case for it is still being invented. Waymo's answer is a critic model and public safety data; Mobileye's is to refuse to rely on one model at all.
Cost and data are the third. Training these models takes thousands of accelerators and billions of miles of logged driving, so the technique concentrates power among a few companies and the suppliers who sell open models to everyone else.
Watch
Frequently asked
- What is a foundation model in self-driving cars?
- A single large neural network pre-trained on huge amounts of driving video, maps and text, then adapted to several jobs at once: driving, simulating the road and evaluating its own behavior. Waymo, Tesla, NVIDIA and the leading Chinese developers all now build around one.
- What is a world model in autonomous driving?
- A model that predicts what the road scene will look like a few seconds from now given a candidate action. Companies use it to generate rare scenarios for training and testing, and increasingly to let the car imagine outcomes before it acts. Examples are Wayve's GAIA, NVIDIA Cosmos and the Waymo World Model.
- What is a vision-language-action model?
- A model that takes camera input, reasons about it in language, and outputs a driving action. The language step lets it handle unusual scenes and explain its choice. NVIDIA's Alpamayo, Li Auto's VLA driver and XPeng's VLA 2.0 are shipping examples.
- What is the difference between a world model and a VLA model?
- A world model predicts the environment: what happens next around the car. A VLA model decides the action: what the car should do. Many stacks use both, with the world model as a simulator in the cloud and the VLA as the driver on the car.
- Does Waymo use Gemini to drive its cars?
- Not directly. Waymo says its Waymo Foundation Model is trained with Gemini so it inherits Gemini's world knowledge, and that this model underlies its Driver, Simulator and Critic. The car runs a driving model, not the chatbot.
- Are foundation models already in cars people can buy?
- Yes, mostly as assisted driving. Li Auto and XPeng ship vision-language-action drivers on consumer cars in China, Tesla's FSD (Supervised) runs on an end-to-end foundation model, and Mercedes-Benz's CLA was announced as the first car on NVIDIA's Alpamayo platform.
Keep learning, once a week
The weekly digest: the stories worth your time from the autonomous vehicle industry, hand-picked by people who work in it. No marketing gloss.