Are We Controlling Software in the Wrong Place?

Are We Controlling Software in the Wrong Place?

AI agents are now producing more changes than people can realistically review. Maybe we don't need better reviews—we need to rethink where responsibility and control actually emerge.

Software development is changing faster than its tools. Not because GitHub has become worse. Not because CI is failing. But because people are no longer writing most of the code. Agents are producing changes at a pace that would have seemed impossible just a few years ago.

That creates a new problem. Not writing the code. Responsibility for it.

Today, we review every change before it is integrated. Tests run. Security scanners run. Linters complain. People read pull requests, leave comments, request changes, and eventually give their approval. This process has proven itself over decades because it matched the pace at which people wrote code.

But the faster agents work, the more difficult this strategy becomes. A single agent can produce more pull requests in one afternoon than a team used to write in a week. And it doesn’t stop with one agent. Several work in parallel, on different parts of the same system. Some interact with one another. Some contradict one another.

At some point, nobody can truly understand every change anymore. Not because people have become worse. But because the volume exceeds the capacity a person can realistically devote to reading code carefully.

The obvious reaction is better tools. Faster scanners, smarter linters, reviewers that are themselves AI-assisted. More control to keep up with more volume.

We No Longer Review Everything Ourselves

Before taking this question any further, it’s worth taking an honest look at what is already happening today. We have long since started trusting the results of our tools instead of verifying everything ourselves. The test suite is green, so we assume the feature works as intended. The scanner finds no known vulnerability, so we assume there isn’t one. The review is brief but positive, so we merge.

Nobody reads every line of every dependency they add to a project anymore. Nobody understands every detail of the operating system their code runs on. Trust in tools and processes has always been part of software development. What changes with agents is only the proportion. The share that a human actually read used to be large. Now it is becoming small.

And that leads to a different question than how to build better tools: Maybe the problem isn’t just the amount of code. Maybe it lies in our model of control itself—in the assumption that trust has to emerge at the point of integration by reviewing as much as possible beforehand.

A Look at Another Domain

A human body consists of trillions of cells. Every one of them divides, produces proteins, responds to its surroundings, eventually dies, and is replaced. No central authority reviews each of those actions before they happen. Yet the organism functions for decades under constantly changing conditions.

This is not a claim that software and living organisms are the same. It is the observation that a highly complex system can apparently organize control differently from the way we are accustomed to in software development.

Responsibility emerges locally. Each cell is responsible for its own area, knows its task, and responds to signals from its immediate surroundings. It does not need to know what is happening at the other end of the body to do its job correctly.

The organism as a whole does not observe every individual action. It observes the current state—and responds when that state deviates from the norm. So control does exist. But not as prior review of every individual action. Instead, it exists as continuous observation and reaction to anomalies.

Human beings themselves do not control their individual cells one by one. They cannot decide that one particular cell should respond differently at this moment. What they can influence are the surrounding conditions: what they eat, how they move, and what kinds of stress they expose themselves to. And they rely on regulatory feedback loops and an immune system that works continuously in the background without being directed in detail.

Where Control Actually Takes Place

Applied to software, this would not be a recipe but a shift in the question: What, exactly, should control focus on?

If developers can no longer read every line of code that enters a system anyway, then their real ability to influence things may no longer lie in the internal implementation of an individual component. Instead, it may lie where they can actually shape the system: the surrounding conditions, the rules, the boundaries of responsibility, the interfaces, the mechanisms for testing and observation—and the definition of what counts as an anomaly or damage during operation.

If a component fulfills its responsibility and honors its interface, then how much code it required internally—or exactly what that code looks like—becomes secondary. What matters is no longer primarily how much code enters the system, but whether a responsibility continues to be fulfilled, whether relationships and boundaries remain stable, whether the system detects anomalies during operation, and whether a failure remains local or spreads.

That would also make something else possible: Multiple modules could assume the same responsibility, satisfy the same interface, and be observed during live operation. Which implementation actually fulfills its task better under real-world conditions would no longer have to be decided in advance through code review. It could reveal itself at runtime. The more suitable implementation would retain—or receive—more work. One that proves conspicuous or performs less well would lose it.

That might also allow a different approach to change itself. A new implementation would not have to replace the existing one immediately. It could initially take on the same responsibility in parallel and prove itself under real operating conditions. Only once it fulfills that responsibility more reliably would it take on more of it—not because of a single decision, but because of continuous observation.

That would not be uncontrolled arbitrariness. The selection rules, the safety boundaries, the fallback mechanisms, the observation itself—all of that would still need to be designed by people. Just no longer as a review of every individual line of code before integration, but as the design of the framework within which implementations either prove themselves—or they don’t.

Maybe we don’t just need tools that can review ever larger amounts of code before integration. Maybe we need to redefine what developers should actually control—and where that control makes the most sense.

The real bottleneck may not be GitHub, the CI pipeline, or the shortage of reviewers.

It may be the way we organize responsibility.

Michael of the Dragons

develops books, software, and open frameworks around technical systems, digital independence, and durable software architectures.
More about Michael →
Are We Controlling Software in the Wrong Place?
← Next Article Codeberg, AI Code, and the Health of Software Systems