For more than a month now, I have been trying to build AI agents that do actual work and produce meaningful results. The distinction between those two things has become increasingly important to me, because getting an agent to do something is surprisingly easy today. Getting it to do that thing reliably enough that I can stop worrying about it is a completely different problem.

You can get a basic agent running in a couple of days. Give it a capable model, access to a few tools, some instructions and enough context, and you can quickly build something that looks impressive. It can browse the web, research a company, write something, interact with an API or execute a sequence of actions. If you record the right run and put it on YouTube, it can easily look like the future has already arrived.

This is something I have also noticed while building normal AI products. AI has dramatically compressed the amount of time required to go from an idea to something that works. What used to require weeks of development can sometimes be put together in days, and what used to take days can sometimes be done in hours. But I have increasingly started to feel that we are confusing this improvement in the speed of building with an equivalent improvement in the speed of building something useful.

Those are not the same thing.

The 24-hour product is usually a 24-hour prototype

There is a certain category of content that has become common over the last couple of years. Someone builds an application in a few hours and the title eventually becomes something along the lines of, "I built a million-dollar SaaS in 24 hours." I understand why these videos exist, and some of the things people are building genuinely are impressive. But they also create a strange impression of what product development actually involves.

If I had to put an arbitrary number on it, many of these demonstrations probably get you to around 40% of a real product. You have the interface, the main functionality works and there is enough there to demonstrate the idea. AI is exceptionally good at helping us reach this point quickly.

Getting to around 60% gives you something more functional. The happy path works reasonably well, the major pieces are connected and you can probably put deterministic rules around the places where you need predictable behaviour. Someone other than the person who built it might even be able to start using it.

You can push further towards 80%, and AI continues to help enormously. But somewhere around this point, the nature of the work starts changing. You are no longer asking whether you can build the feature. You start asking what happens when the feature doesn't behave the way you expected.

That is where product development begins to become painfully normal again.

The first part of the curve has been compressed dramatically by AI. The final part hasn't been compressed by anything close to the same amount. A prototype can take hours, a functional product might take days, but getting something reliable enough for real users can still take weeks or months.

This is especially important for people building AI products today. Generating software is getting cheap. Judgement isn't.

The last 20% contains most of the uncomfortable questions

Once real people start using a product, you discover all the things that were conveniently absent from the demonstration. A user enters something you never expected. An API goes down halfway through a workflow. The model returns perfectly formatted nonsense. A permission isn't available. Context disappears. Something that worked 50 times suddenly fails on the 51st attempt.

Then you have to decide what the product should actually do.

Should it retry? How many times? Should it use another model? Should it ask the user? Should it silently recover? What information needs to persist? What happens if two things happen at the same time? What gets logged? What should the user see when something fails? How much should all of this cost?

These are not primarily coding questions. They are product questions, architecture questions and judgement calls. AI can certainly help answer them, but somebody still has to understand the problem well enough to decide whether the answer makes sense.

This is why I don't see AI removing the value of good product builders. It might actually make their judgement more important. When everyone can build twenty features, deciding which two actually matter becomes more valuable. When anyone can generate a decent interface, understanding how people behave becomes more important. When connecting an AI model takes minutes, knowing where AI should and shouldn't be used becomes a differentiator.

The first 80% is becoming cheaper. A lot of the value is moving into the remaining 20%.

Building agents makes this problem much more obvious

The same problem becomes more extreme when you move from conventional AI products to agents. In traditional software, much of the behaviour is deterministic. The user performs an action and the software follows a path that the developer has defined.

With an agent, you deliberately give away some of that control. The system has to decide what to do next based on the situation it finds itself in. That is the entire point of having an agent, but it also means that every additional bit of autonomy introduces another place where judgement can go wrong.

I experienced this while building an outreach agent. Getting the basic system running wasn't particularly difficult. It could find companies, research them, identify relevant people, collect information and prepare outreach. Looking at the workflow, it felt like most of the problem had already been solved.

Getting meaningful results from it was much harder.

Better research meant more searches. Better judgement sometimes meant using better models. Better personalization meant providing more context. Verification meant additional steps. When something failed, the agent needed a mechanism to retry or take another route. I started juggling different models, platforms, agents and efficiency techniques just to figure out the right balance between quality and cost.

The agent was capable of doing the work. That wasn't really the problem anymore. The question became whether I could trust it to do enough of the work independently, at sufficient quality, for a cost that made the whole system worthwhile.

That distinction has changed how I think about AI agents.

The benchmark shouldn't be “Can AI do this?”

We spend a lot of time looking at benchmarks that measure whether an AI can answer a question, solve a coding problem or complete a specific computer task. These benchmarks matter because they tell us something about the underlying capabilities of the models, but they don't completely capture what matters when you are trying to replace a piece of real work.

The benchmark I care about increasingly is much simpler: Can I give the system an outcome and walk away?

If I ask an agent to research 50 prospects, I don't want to specify every search it should perform. I don't want to inspect every source it opens. I don't want to tell it when its research is insufficient or remind it that the company it found doesn't match the criteria I gave it twenty minutes ago.

I want to tell it what a good outcome looks like and have it work towards that outcome. It should be able to notice when its original approach isn't working, change the approach, verify its own output and continue until the work is completed or it encounters something that genuinely requires human judgement.

That is a very different standard from completing an isolated task successfully.

And there is now an interesting way to measure our progress towards it.

How long can AI actually work on its own?

METR has been measuring something called the task-completion time horizon, which I find much more useful when thinking about agents than many traditional AI benchmarks.

Instead of simply asking whether an AI can solve a particular benchmark question, METR looks at the length of tasks AI agents can complete successfully. The task length is measured by how long the same task would take a skilled human. In its original analysis, METR found that the 50% task-completion time horizon for frontier models had been doubling roughly every seven months.

You can explore METR's methodology, current results and updated time-horizon measurements here:

METR, Measuring AI Ability to Complete Long Tasks:
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

METR, Task-Completion Time Horizons:
https://metr.org/time-horizons/

I find this much more useful when thinking about agents because time acts as a rough proxy for the amount of complexity an agent can survive. A five-minute task usually has fewer opportunities for something to go wrong than a five-hour task. A task lasting several days requires the system to maintain context, execute many steps, respond to failures and continue making sensible decisions.

There are important limitations here. METR explicitly warns that its current measurements become less reliable for very long tasks, so I would not take the historical growth rate and simply draw a line into 2028 or 2030 as if that future is guaranteed. The trend is what interests me, rather than a prediction of the exact date when a particular threshold will be crossed.

This gives us a much more practical way of thinking about progress. An AI that can reliably handle a few minutes of work is an assistant. When it can reliably handle multi-hour workflows, it starts looking more like a worker. When it can independently pursue outcomes that would take a human days, we enter very different territory.

And there is already evidence of the strange gap between being good at bounded tasks and being genuinely autonomous.

Stanford's 2026 AI Index documents the rapid improvement in agentic and computer-use benchmarks. On OSWorld, for example, benchmark performance has moved dramatically closer to human performance. But these results still represent relatively bounded environments where success can be clearly evaluated.

Stanford HAI, AI Index 2026, Technical Performance:
https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance

The important threshold may therefore have very little to do with the point at which we officially declare something to be AGI. There could be an economically transformative threshold well before that.

AI doesn't need to become human to change how companies work

Consider the outreach agent again. It doesn't need to possess everything we would associate with human intelligence. It doesn't need to understand every subject I understand, learn any arbitrary new skill or function independently in every environment imaginable. It needs to be extremely good at outreach.

It needs to understand the objective, find the right companies, judge whether they fit, research them properly, identify relevant people, understand what matters about each prospect, prepare sensible communication, observe the results and adjust what it does next. If it can reliably own that outcome, the fact that it cannot perform hundreds of unrelated human activities becomes commercially irrelevant.

This distinction also shows up when we test AI in more open-ended environments.

ARC-AGI-3 is particularly interesting here because it tests whether an agent can explore unfamiliar environments, discover how they work and adapt its behavior. Humans can solve these environments, while frontier AI systems performed extremely poorly in the initial evaluations. This is a very different problem from answering a question where the model already understands the domain.

ARC Prize, ARC-AGI-3 Technical Report:
https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf

GAIA2 approaches the problem from another direction by testing agents in dynamic environments where the world can change independently of the agent. The agent has to deal with time, ambiguity, changing information and other actors rather than simply following a fixed workflow.

GAIA2, Dynamic Agent Benchmark:
https://arxiv.org/abs/2602.11964

These benchmarks are useful because they show why today's AI can simultaneously feel extraordinarily intelligent and surprisingly incapable of owning relatively ordinary work.

The same principle applies across industries. A coding agent doesn't need to become a universally intelligent machine before it changes software development. A finance agent doesn't need to understand every aspect of the world before it changes accounting workflows. The same applies to marketing, operations, support, research, analytics and many other functions.

We don't necessarily need AGI to create massive economic change. We need AI systems that are sufficiently capable within useful domains to own meaningful units of work. That threshold feels much closer.

Capability alone will not get us there

There is another problem that becomes obvious once you actually build these systems: autonomy costs money.

A simple AI interaction might involve one model call. An autonomous workflow can involve dozens or hundreds. The agent has to understand the objective, retrieve information, call tools, inspect responses, reason about what happened, decide what to do next, retry failed actions and verify the final result.

This behavior is exactly what makes the agent useful. It is also what makes the agent expensive.

This is why I don't believe model intelligence alone determines when agents become transformative. Two things need to happen together. Capability and reliability need to improve while the cost of achieving a successful outcome falls.

The second part is particularly important because the number we usually discuss, price per token, isn't necessarily the right economic metric.

What matters is cost per successful outcome.

A cheap model that requires ten attempts and constant human supervision can ultimately cost more than an expensive model that completes the task correctly on its first attempt. Similarly, an agent that costs very little in API calls but requires someone to inspect everything it does isn't really cheap. The human supervision is part of the cost.

Once you start measuring the complete outcome rather than individual inference calls, agent economics starts looking quite different.

The strange gap between intelligence and autonomy

This creates one of the strangest things about using AI today. I can have a remarkably sophisticated conversation with a frontier model. It can understand complicated ideas, write software, analyse information, research topics and use tools. There are moments when interacting with these models where the level of intelligence feels extraordinary.

Then you put the same intelligence inside an agent, give it a relatively ordinary real-world objective and watch it make a ridiculous decision halfway through.

The problem isn't necessarily that the model lacks intelligence in the traditional sense. Real-world autonomy requires something else. It has to preserve the right context, understand what matters, detect when something has gone wrong, decide whether to retry, change strategies when necessary and continue doing all of that over a long sequence of actions.

This is the gap that interests me most right now. AI can write. It can research. It can use tools. It can reason. What it still struggles to do consistently is take responsibility for the complete outcome.

Humans remain the glue between many of those capabilities. We tell the agent that its research isn't good enough. We notice that it misunderstood the objective. We decide that the third retry isn't going anywhere. We change the strategy. We connect information that the system failed to connect itself.

The moment the AI starts reliably doing more of that glue work itself, the usefulness of agents changes dramatically.

Autonomy also has a cost curve

There is a frustrating relationship here. The closer I try to push an agent towards genuine autonomy, the more work the system often needs to perform internally.

A basic system might make one attempt and return whatever it gets. A better system verifies its answer. A better one might evaluate the verification, identify weaknesses, gather additional information and try again. A more capable agent might maintain memory, compare different approaches, use multiple tools and escalate to a stronger model when required.

Every one of those mechanisms can improve reliability. Every one can also add cost.

The ideal future isn't simply that models become cheaper. The important change is that the cost curve for successful autonomous work becomes flatter.

If an agent can attempt a problem twenty times, evaluate those attempts and arrive at the correct outcome for a negligible cost, its behaviour can become very different. We can allow it to explore. We can let it make mistakes. We can let it test alternatives instead of trying to engineer every workflow around getting the correct answer immediately.

Cheap intelligence makes experimentation cheap. Cheap experimentation makes self-correction practical, and self-correction is one of the foundations required for useful autonomy.

There is a threshold where all of this changes

This is why I increasingly imagine AI progress as several lines moving towards the same region rather than one intelligence curve moving towards AGI.

Capability is increasing. The amount of work agents can handle is getting longer. Tool use is improving. Reliability is gradually improving. Inference costs are falling, and developers are getting better at routing tasks between different models instead of using the most expensive model for everything.

None of these individually creates the transformation.

The interesting moment comes when enough of them converge.

Imagine an agent that can independently pursue a multi-day objective with high reliability. It knows when it has made a mistake, can change its strategy and can use different tools or models depending on what the situation requires. Human intervention happens when there is genuinely an important judgement call rather than every time something slightly unexpected occurs.

Now make running that agent inexpensive.

At that point, we have crossed a very different threshold.

The instruction is no longer, "Help me do this."

It becomes, "Take care of this."

From answering, to acting, to owning

Looking at the last few years, I see three broad stages in how AI is becoming useful.

The first was AI can answer. ChatGPT made this obvious to everyone. We could ask questions, generate text, analyse information and use AI as an extremely flexible thinking and creation tool. It saved time, made knowledge more accessible and changed how many of us work, but the human remained responsible for taking those answers and doing something with them.

The stage we are in now is AI can act. Models can browse websites, use computers, execute code, manipulate files, call APIs and interact with external systems. This is what has made agents possible. AI is no longer restricted to producing an answer inside a chat window.

The next stage is AI can own. This is the point where we can specify the outcome rather than the steps. The system figures out how to get there, executes the work, evaluates what happened, corrects itself and continues.

That is the threshold I am waiting for.

When that happens, AI stops being something that merely makes an existing worker more productive. It starts becoming another unit capable of producing work.

And I suspect that distinction will matter far more economically than whatever date we eventually decide to attach to AGI.

We are close, but close doesn't necessarily mean tomorrow

After spending more than a month working on agents, the interesting thing is that the remaining problems don't feel impossible anymore. They mostly feel frustrating.

Memory needs to be better. Context needs to survive longer. Tool use needs to fail less often. Agents need better judgement about when to continue and when to stop. Self-evaluation needs to become more dependable. Costs need to keep coming down, and we need better ways of using expensive intelligence only where it actually adds value.

None of these feels individually like a science-fiction breakthrough.

But combining all of them into a system that can be trusted is difficult. Anyone who has tried to turn an impressive agent demo into something that works every day will probably recognise that gap very quickly.

That is why I am simultaneously excited about agents and much more cautious about them after actually building them.

The demos are real. The capabilities are real. The progress is real.

But so is the last 20%.

For product builders, that last 20% is where product judgement, domain expertise and genuine human reasoning still matter enormously. For AI agents, it is where reliability, autonomy and economics have to come together.

We don't need to wait until AI can do everything a human can do. We may not even need anything that everyone agrees to call AGI.

The more important line might be much closer and much simpler.

Can I give an AI a meaningful outcome, walk away, and trust it to keep working until that outcome is achieved, while spending less than it would have cost me to get the work done another way?

When the answer to that becomes consistently yes, we will have crossed the threshold that matters.