When you take a photo with your smartphone, a small chip inside the camera — the image signal processor — quietly polishes the raw light captured by the sensor. It adjusts brightness, corrects color, pulls dark areas up into visibility. The pretty picture you post on Instagram is the result of that polishing. Tesla, however, hands its self-driving AI not the pretty picture but the unprocessed data — something that, to a human eye, looks off in color and unnaturally lit. On the driver’s screen, the image still looks normal. Behind it, the AI is reading a completely different world.
The reason is simple. The very process that makes a picture look good is the process that hurts the AI. Picture the exit of a tunnel. Polish the scene for human eyes and the bright parts wash out to white while the dark parts get artificially brightened. Maybe there was a car ahead in the washed-out region; in the artificially brightened region, the line between a real object and noise becomes blurry. “Making it look good” is, functionally, “erasing the cues that matter for judgment.” Elon Musk said as much on the Lex Fridman podcast in 2021: “We’re bypassing the image processing chip entirely and using only the raw photon counts. That saves us 13 milliseconds of latency.” In autonomous driving, 13 milliseconds at 100 km/h is about 36 centimeters of distance traveled. A small number on paper. In an emergency-braking situation, it’s the gap between stopping and crashing.
Trace Tesla’s last eight patents and the full strategy comes into view. The footage from the eight onboard cameras is first mathematically corrected for lens distortion and exposure differences. Then, the hard-to-detect distance bands — like the area near the horizon — are processed in high resolution, while the rest is downscaled to save computation. The principle is the same one your eyes use, sharpening only the central field of view. That’s preprocessing. The core is what comes next. The AI converts these images into a 2D top-down map and a 3D Lego-block world, dividing the space around the car into hundreds of thousands of small cubes — voxels — and marking each one as occupied or empty. The strength of this approach is clear. Even an object the AI cannot name — a sofa fallen onto the highway — can be detected by its shape and avoided. In 2024, more than 300,000 lines of rule-based code were replaced by a single, enormous neural network. The AI taught itself, by training on millions of driving clips, how to behave at a roundabout.
Competitors are climbing the same mountain by different trails. Google’s Waymo augments its cameras with four LiDAR — light detection and ranging — units and six radar units, for 23 sensors in total. It costs more, but the logic is that different sensors compensate for each other’s blind spots. A dust storm can blind the cameras and the LiDAR will still see the pedestrian. Waymo currently operates fully driverless taxis — no human in the driver’s seat — in six U.S. cities, including Phoenix and San Francisco, and logged 15 million paid rides last year. China’s Huawei sticks with up to four LiDAR units per vehicle, while DJI has rolled out an ultra-low-cost urban driving package using seven cameras and a chip that costs about $1,000.
The argument all converges on a single question. With a sufficiently smart AI and sufficiently large amounts of data, can cameras alone do everything LiDAR does? Tesla’s roughly 9 million vehicles worldwide accumulate driving data on the order of 100 billion miles per year — a scale no competitor can mimic. But in a 2024 industry-wide evaluation, Waymo finished first and Tesla finished last, a reminder that volume of data doesn’t equal completeness of technology. The Tesla Robotaxi pilot in Austin has reported 14 collisions, putting safety squarely back in the conversation. The next-generation AI chip due late this year will reportedly multiply onboard compute by more than five. How intelligently that chip reads these “ugly” pictures will be the first real test of whether camera-only autonomous driving can hold its own.