How it works
Sensing: How Self-Driving Cars See the World
Sensing is how a self-driving car perceives its surroundings. It combines several sensor types, cameras, radar, lidar and ultrasonic sensors, with satellite positioning and prior maps. Each has different strengths and blind spots, so most systems fuse them for redundancy, while a few rely mainly on cameras. Together they answer where the car is and what is around it.
Why use more than one sensor?
No single sensor works well in every situation. Cameras see color and detail but struggle in glare and darkness. Radar sees through rain and fog but gives a coarse picture. Lidar measures precise shape but is costly and degrades in heavy weather.
Combining their outputs, a process called sensor fusion, gives both redundancy, so one failure is covered by another, and complementary information, so the strengths of one cover the weaknesses of another.
What are the main sensors?
The four sensors below build the picture of what is around the car. The table summarizes what each measures and where each falls short.
| Sensor | Measures | Strengths | Limitations |
|---|---|---|---|
| Camera | Color and texture in 2D | Cheap, high detail, reads signs and lights | Poor in low light and glare, must infer distance |
| Radar | Range and speed by radio | Works in rain, fog and dark, long range | Low resolution, weak at classifying objects |
| Lidar | Precise 3D shape by laser | Accurate distance, day or night | Costly, degraded by heavy rain and snow |
| Ultrasonic | Very short distances by sound | Cheap, good for parking | Only a few meters, low speed only |
How does the car know where it is?
Satellite positioning, the family of systems often called GPS, fixes the car on the globe to within a few meters. With correction signals, the technique known as RTK can sharpen this to centimeters, but it is blocked in tunnels and among tall buildings.
To stay located when signals drop, the car blends positioning with an inertial measurement unit and wheel sensors, a method called dead reckoning. Many systems also match what they see against a detailed prior map to pin down the car and the road layout, though some teams aim to drive with little or no map.
Cameras only, or cameras plus lidar?
There is an open debate over whether cameras alone are enough. Tesla argues that vision, like a human driver, can suffice, and it removed radar in 2021 and ultrasonic sensors in 2022 in favor of a camera-only stack.
Waymo and most robotaxi operators take the opposite view, pairing cameras with radar and lidar so the sensors can check one another. The trade is cost and complexity against redundancy and direct distance measurement.
Frequently asked
- What sensors do self-driving cars use?
- Most use a mix of cameras, radar, lidar and ultrasonic sensors to perceive their surroundings, plus satellite positioning and often a detailed map to know where they are. Some systems rely mainly on cameras.
- What is the difference between radar and lidar?
- Radar uses radio waves to measure range and speed and works well in rain, fog and darkness, but its picture is coarse. Lidar uses laser pulses to build a precise 3D shape of the scene, but it costs more and is degraded by heavy weather.
- Why does Tesla not use lidar?
- Tesla argues that cameras, paired with strong software, can drive the way a human does with eyes alone, and that lidar adds cost without enough benefit. Most other developers disagree and keep lidar for redundancy and precise distance.
- What is sensor fusion?
- Sensor fusion is the process of combining the outputs of different sensors into one consistent picture of the world, so their strengths add up and a weakness or failure in one is covered by another.
- How accurate is GPS in a self-driving car?
- Plain satellite positioning is accurate to a few meters. With correction signals, RTK can reach a few centimeters. Because signals drop in tunnels and cities, cars combine positioning with inertial sensors and maps.