The Illusion of Software Automation

There is a widely held belief that AI is automating software engineering.

That belief misunderstands what software engineering actually is.

AI is genuinely very good at generating code. In many cases it can produce correct, idiomatic implementations far faster than a human typing from scratch.

But writing code was never the entire job.

Most engineering time is spent elsewhere: understanding the problem, choosing an approach, tracing consequences through complex systems, and judging long-term trade-offs. Much of this work depends on context that is implicit, undocumented, spread across people, or even lost to history.

AI helps with these activities to some extent. It can summarize files, explain unfamiliar code, and suggest approaches. But the gains are smaller than in code generation because these tasks require deep situational knowledge: why a system evolved the way it did, which constraints are real versus accidental, and how a change interacts with business needs, team processes, and long term maintenance.

This creates an asymmetry. The part of the job that AI accelerates the most, producing new code, is not the part that typically limits delivery speed.

I use AI to generate most of the code I write. It clearly saves time at the keyboard. But my overall productivity has increased modestly, not dramatically, because the surrounding work still dominates: understanding the system, validating assumptions, coordinating with others, testing, and safely integrating changes.

Many claims of “10x productivity” focus on a narrow metric: how quickly code can be produced. That is not the same as how quickly correct, maintainable, production ready software can be delivered. The latter includes discovery, decision making, integration, and verification. These activities remain much less automated.

People outside the profession often imagine programming solely as an act of creation. In practice, much of the work involves archaeology. Engineers spend much of their time digging through layers of past decisions, reconstructing context, and carefully modifying systems that already exist.

AI has made the creation phase faster. But it has not fundamentally changed how much archaeology there is, and that remains the larger share of the work.

We are mesmerized by how quickly AI can produce code and do agent-like tasks. Searching docs, moving files, scaffolding projects. It feels like a massive productivity boost. But we are overestimating the gains because we underestimate the cost of reviewing, validating, and integrating all that output. We act like we are removing work for free. In reality, we are shifting most of it.

The main tradeoff with AI tools is replacing write time with review time.

AI lets us dramatically increase code production, but human review and understanding don’t scale at the same pace. Engineers either review more code and miss more mistakes, or reviews simply take longer.

With how fast AI generates (mostly) correct code, some people reach for a third option. Let AI review the AI and remove humans from the loop.

But that does not remove the bottleneck.

When AI writes and reviews the code, no one necessarily understands why the system behaves as it does. You are no longer shipping software you understand. You are shipping software you hope is correct.

That might be acceptable for low-stakes systems. It is not acceptable for serious ones.

Would you ride in a Waymo if you know its software was written and reviewed entirely by probabilistic models that no human fully understood? Probably not. Not because it couldn’t work, but because safe systems require predictable failure modes and clear boundaries. Those are hard to establish without humans who understand the world in which the system exists.

This leads to a point that matters and gets discussed too little.

You are responsible for what you ship. Not the model.

If you cannot blame the AI when things go wrong, then the AI has not taken any of your responsibility. It just made it easier to create things you are now accountable for.

For a side project or internal tool, the productivity gains are real. For customer-facing, revenue-critical systems, the gains shrink because the review burden stays heavy and the stakes are real.

The productivity story everyone is telling is about speed of code generation. The story that matters is about ownership.

Are you faster? Probably.

Do you understand what you are building? That is the only question that counts.

When the system fails, the person who understands it will fix it. The person who does not will wish they had.

AI will not save you in that moment. Only understanding will.