Let’s imagine that AI fully commoditizes labor.

In software companies, automated bots would be coding literally nonstop. Great. OKRs have never looked better, and the C-Suite is giving themselves a standing ovation after laying off all human employees.

The amount of work getting done has increased exponentially. But that raises a pressing question: Can people even evaluate the work?

Automated bots can pump out infinite software. But the people in charge need to (1) understand and evaluate what the bots are producing, so they can (2) guide the bots towards their human goals.

The most powerful AGI imaginable is misaligned with human ends, and thus functionally useless, if we can’t even understand what it’s doing. An army of AI software engineers is equally misaligned and useless if the human managers can’t comprehend whatever’s getting produced.

At minimum there are two potential problems:

• There’s so much code getting written + deployed that people can’t physically keep up in reading it/testing it
• The code is optimized to the point of being indecipherable

There are endless quibbles here. Software evaluation doesn’t literally mean reading the source code, or we could just have post-deployment setups which run automated tests on system behavior, or maybe the C-Suite only wants bottom-line profits so who cares if we don’t know what these robots are doing, et cetera, et cetera.

The main point is: if we want to steer AI towards accomplishing our goals, we need to be able to evaluate its outputs. Whenever the AI does something, it needs to be human-legibile: people digesting whatever the AIs are doing at a rapid-enough cadence to retain directional control.

What does this mean for software engineering? In the future where AI fully commoditizes labor, I imagine that code readability/legibility/style will become the north star.