I want AI to make people more capable. It has already done that for me, helping me build things I wouldn’t have been able to without a team. But getting more work done is only part of what I mean by capable. I also mean understanding what you’re doing well enough to question it.
Some of the effort AI removes from our work, is the very process through which we develop that understanding. Working through an idea exposes what we haven’t figured out. It can even change our minds about the problem itself. I believe that getting more output in exchange for giving up our ability to understand the work is a bad trade.
Ontrack started as a Google Sheet. I had a trail running training plan and wanted to compare it against the workouts I was actually doing. I started working with AI to pull the sheet into a webpage where I could see it more clearly. But it’s suggestions kept leaning toward a tighter Google integration and an increasingly elaborate system for keeping everything in sync.
Eventually I stepped back, because comparing this week’s workouts was only part of the solution. I needed to see how future training blocks would fit together and look back at how my training had been trending. I was a runner trying to use what I was building, and it still wasn’t giving me the view of my training that I needed.
That changed the direction of the product. AI helped me build it, but my understanding of what to build developed through being involved in the work. My original request was incomplete in ways I hadn’t yet recognized. A better spreadsheet integration could have been a perfectly good answer to that request and still left the underlying insight undiscovered.
This is why I’m wary of treating the user’s first instruction as a complete account of what matters to them. Sometimes working through the problem is how they find out. An AI product that makes it easy to keep producing things can also make it easy to keep pursuing an approach that needs questioning.
But there’s another way I could have used AI to write this essay. I could have asked it what to think.
That isn’t entirely new. Search engines gave us access to other people’s answers, and social media made it easy to absorb opinions from people who sound convincing. We’ve always been capable of adopting a conclusion without doing much of the thinking that produced it.
But I assert that AI makes that much easier. I can ask a question specific to my work or my life and get a personalized, thoughtful, persuasive answer in seconds. It can connect things I hadn’t connected myself and explain them better than I could. That’s incredibly useful, but it also makes it easy to mistake having a good-sounding answer in front of me for having understood it.
I want my opinions to have earned their place in my head. If I accept a convincing answer without understanding why I believe it, or what would make me change my mind, I’ve skipped something important. Being able to explain the opinion fluently doesn’t mean I’ve actually thought it through.
That gap between something looking thought through and actually being thought through becomes more consequential when other people are expected to act on it.
When the output looks finished
Give an AI a rough product idea and it can produce a detailed requirements document. Imagine that document specifies an automatic refund in a particular situation. Did anyone decide the business could afford that? Is it consistent with what support tells customers? A paragraph that reads like a requirement may hide a decision nobody has made.
Hand it to a team without working through it and they inherit those decisions. Feed it to a coding agent and the unresolved assumptions can become perfectly functional code. Now the thing nobody thought through is part of the product.
You’ve outsourced the thinking to the model, then handed the unfinished part to your colleagues. You get to feel productive because the document exists, but they inherit the burden of making sense of it. Niklas Gruhn describes a similar transfer of work to code reviewers in “Don’t be a meat proxy”[1].
If my name is on a recommendation, I should understand the important choices in it. When I can’t explain one, that’s a reason to keep working. Generating more detail won’t resolve a faulty premise or a decision that I haven’t faced.
Where does the judgement come from?
There’s also the question of what people learn while producing the work. In a small Anthropic experiment with 52 developers[2], participants learned an unfamiliar programming library with or without AI assistance. On an immediate comprehension quiz, the AI group averaged 50%, compared with 67% for the group without it. It’s a limited study of learning a particular tool, but it shows why completed work and acquired understanding need to be assessed separately.
That matters when we expect humans to supervise more AI work. At Orbit, we reached a point where we didn’t see a need for junior engineers on our team. I understand the calculation. I also think it leaves a question we have to answer.
Junior work gave someone a way to contribute while learning the codebase and gradually taking on harder problems. The senior engineering staff worked on more complicated projects while a junior engineer gained the experience needed to take on that type of work in the future. If more companies decide to hire seniors and use AI for the rest, who develops the seniors they all want to hire?
Automating the work doesn’t automatically replace the learning. We can develop people differently, including with AI, but we have to actually do it. A team that depends on experienced judgment needs a credible way to help people acquire it. Otherwise, we’re relying on expertise while neglecting how it gets renewed.
The strongest objection to all of this is that people want results. If a machine can do something well, why insist that a person learn to do it? I don’t need to understand every implementation detail of Ontrack. A shop owner shouldn’t need to become a copywriter to get a useful website. Delegating those things can give us more control over what we spend our lives doing.
The understanding I want to protect is what lets us decide whether the result serves our purpose, recognize a mistake that matters, and change direction. That understanding develops through engagement with the problem. We shouldn’t assume it will remain intact however much of that engagement we remove.
People may also value doing the work itself. Some engineers in Anthropic’s interviews with its own staff[4] missed the flow of coding; others preferred what AI let them accomplish. More output doesn’t settle how someone feels about the work they now spend their day doing. That choice matters, too.
Nothing loads from YouTube until you press play.
For the tools I want to build, this has practical consequences. An incomplete thought should be a valid starting point. The AI tool should help people discover what they haven’t considered, even when that means asking a question instead of immediately producing an answer. In the refund example, it could ask for the current policy and help the owner understand the cost of changing it before quietly treating the change as a requirement.
That interaction may take longer than generating a plausible policy. I think that can be the better product. The time to an answer is a poor measure when the answer conceals work that still needs doing.
The same applies to learning. If a workflow depends on a person’s judgment, helping them develop that judgment is part of making the workflow successful. Useful practice deserves time, even when immediate output would be higher if we skipped it.
I want AI to take a lot of work off people’s hands. I also want the AI products that I build to leave them better able to understand and direct the work they care about. I’m willing to keep some effort in the process to achieve that. If all we’ve done is make it easier to approve decisions people can’t question, I don’t think we’ve built something very good.