How are AI agents changing productivity?

Metrics that do not measure the same thing

Tasks finished in a quarter of the time, five times more commits, eight times more merged code: figures on AI agents abound, and most of them are accurate. Yet they do not measure the same thing. The first reflects a perception, the second a volume of code written, the third code accepted into the project. None of them yet tells us how much software actually reaches production, or whether it is used. For an IT department that has to decide where to invest, that is the decisive question.

In 2026, research is finally measuring the specific impact of development agents, able to write, test and fix complete features, and no longer just that of autocomplete. We chose to start from these large-scale measurements: those of the NBER, the US economic research institute, of METR, an independent lab that evaluates AI, and of Anthropic.

This is also our engineering approach: measure before asserting, and distinguish the gain on the task from the gain on the software delivered. We read the studies through that lens: what they really measure, on which population, and what still holds outside the context studied. Anthropic's publications are illuminating, but they describe a singular company, one that builds its own models: we point out its specific features each time.

Our aim in this analysis is to follow the real gain from the execution of a single task to the product delivered to the user, to identify who actually benefits, to understand how engineering work is changing, and to set out a rigorous framework for delegating to agents.

This article is part of our reference series, "Augmented engineering: what really changes". The intrinsic quality of generated code, the levels of autonomy given to agents and the reorganisation of teams are covered in dedicated articles.

On the task itself, the writing gain is undeniable

The most robust empirical assessment to date comes from researchers at MIT and Wharton. Published by the NBER in May 2026 and updated in September, it follows a cohort of 521,329 developers on GitHub and compares three successive waves of tooling. The figure below shows the rise in commits for each.

The authors observe that these gains grow mainly with the progress of AI models, far more than with developers' learning.

Anthropic's internal observations confirm this raw acceleration. As early as August 2025, its engineering teams reported a 50% productivity gain, against 20% the year before, backed by a 67% rise in merged pull requests, the code changes submitted for review, per engineer per day. Engineers also said that 27% of the work done with Claude would never have happened without it, typically internal tooling or small fixes long put off. In October 2026, the company recorded five times more commits per engineer than six months earlier, along with higher output outside engineering: more contracts handled by lawyers, more invoices processed by accounting, more deals closed.

Even so, these indicators quantify raw activity: code written, tasks handled. They show a much greater production capacity, but do not yet demonstrate that more business value is being delivered. For coding assistants, we estimated in a previous article the net gain for a team at 5 to 15%; agents change the scale of the code written, without transforming the number of features put into service at the same pace.

Related readHow Much Faster Can AI Coding Assistants Really Make Software Delivery?A heuristic, a 150-ticket case study, and 2025-2026 research: real speed gains from AI coding assistants sit between 5 and 15%, far below the marketing hype.Read the article

The cognitive trap: perceived speed above real speed

The controlled experiment run by METR in July 2025 brought this paradox to light: experienced developers were 19% slower when they used AI, while firmly believing they had been 20% faster. The follow-up published in February 2026 could not repeat the measurement: many participants now refused to work without AI, which ruled out any comparison.

Perception also varies widely depending on who answers. In August 2026, Anthropic asked the same question to 1,334 of its employees and to a sample of nearly 11,000 adults, some of whom use AI at work: how long does a task well suited to AI take with it, such as summarising a document or drafting an email, checking and fixing included? This is not a measure of speed, but a comparison of perceptions between those who build these tools and those who use them every day.

The answers are reversed. Anthropic employees are more than twice as likely to say they save a lot of time. Conversely, for nearly a third of other users, checking and fixing wipe out all the time saved, against 5% at Anthropic.

Anthropic itself urges caution about its own figures. One of its reports finds eight times more merged code per engineer than in 2024, but calls this figure "almost certainly an overstatement of the true productivity gain". Its study on how employment is changing at the company puts it plainly: "More code does not necessarily mean better code or business impact."

From code written to software used: the funnel effect

The NBER study follows how the gain carries through the development cycle, from writing code to publishing it. For developers working with an interactive agent, the gap between upstream and downstream is striking: commits jump by 153% and pull requests by 86%, but the number of active projects rises by only 47%, and the number of releases actually published by only 20%. The authors keep 153% for commits; a more cautious comparison method still gives 85%.

This dampening also shows up on consumer app stores. On the App Store, the monthly flow of new iOS apps goes from about 33,000 to 45,000 to nearly 108,000, with no rise in total usage. Meanwhile, the share of apps published without an audience goes from 78% to 87%. The researchers draw a fundamental lesson: gains seen on individual tasks only partly translate into software delivered and used. The constraint appears to be shifting from writing code to reviewing, integrating and distributing it.

On quality, the available measures show no deterioration: the share of commits devoted to bug fixes stays stable in the NBER cohort. Likewise, at Anthropic, for the same author, month and project, code written by an agent is no more often deleted or reverted than code written by hand. The authors point out, however, that errors introduced by AI can be harder to spot in review. We cover the technical debt of generated code in AI-generated code: why “it works” does not mean “it is maintainable”.

Related readIs there still a place for agile in the age of AI-DLC?AI agents can now produce in a few hours what a team used to deliver in a two-week sprint. AWS has turned this into a method, AI-DLC, which replaces the sprint with the "bolt", a cycle lasting a few hours or a few days. Some conclude that agile should be thrown out. We think the opposite: AI-DLC retires the rituals of agile, not its values, and AI makes those values more necessary than ever. This article opens a series in which we will put that claim to the test on a project built for the purpose, with measurements to back it up.Read the article

The expertise premium: expertise as the differentiating factor

In June 2026, Anthropic analysed about 400,000 sessions run on Claude Code by some 235,000 users, between October 2025 and April 2026. The division of responsibilities is clear: users take about 70% of planning decisions, what to do and how to approach it, while the agent takes 80% of execution decisions. The authors sum up this split in one sentence: "People decide what to build, and the agent decides how to build it."

Yet the outcome depends directly on the expertise of whoever is steering. The verified success rate, that is, sessions that end with evidence such as passing tests or committed code, is only 15% for novices, against 28 to 33% from intermediate level upwards. When a session runs into trouble, novices give up in 19% of cases, against 5 to 7% for more experienced profiles. The latter write longer, more precise instructions. Agents do not replace domain expertise: they amplify its effect.

Hiring follows the same trend. At Anthropic, interview assessments for technical roles increasingly value candidates' autonomy and judgement.

Rethinking working time and the rise of "meta-work"

As the constraint moves towards review and integration, the daily work of teams is being redefined. The study Anthropic published in October 2026 on its own organisation offers a first field of observation: in interactive sessions, those where an employee works live with Claude, agents spend about 80% of their time waiting for a human approval or decision. For non-technical roles, reading everything their sessions with Claude produce would take more than half of the working week. Human attention becomes the real limiting factor.

A growing share of work therefore consists of organising and checking the agents' activity: writing their instructions, having one agent review another's work, monitoring automated pipelines. This "meta-work" more than doubled in six months, rising from 10% to 22% of interactive sessions, while the number of written instructions for agents went from a few hundred to 8,500 in a year. Over the same period, time spent in meetings by the median employee rose by 20%: decisions and coordination remain human.

Anthropic has the particular features of a fast-growing model developer, and its authors warn that many of its practices cannot be replicated elsewhere. One lesson nevertheless holds beyond its case: when code production speeds up, it is the ability to decide, check and coordinate that sets an organisation's real speed.

The decision grid: the verification effort rule

Between full autonomy and checking every line, the Anthropic engineers interviewed in 2025 offer a simple criterion. One of them puts it this way: the agent is remarkable "for everything where validation effort isn't large in comparison to creation effort". The easier it is to check a result compared with the effort of producing it, the more delegation pays off.

Combined with the risk of an error, this criterion draws a clear operational line. The grid below lets you apply it to your own tasks.

Even in a company where engineers use Claude for 59% of their work, more than half of them say they can fully delegate only 0 to 20% of their tasks. There is also a paradox: supervising an agent rigorously requires precisely the skills that too much delegation risks eroding. The levels of autonomy to grant agents, and the controls that go with them, are detailed in AI agents: 40% of enterprises could restrict or retire them by 2027.

Performance comes from the organisation, not the tool

A Harvard Business School study published in June 2026 covers 2,891 young companies from the Y Combinator accelerator. These are very different from the IT department of a large group, but one finding is worth keeping: simply handing out individual licences, ChatGPT, Copilot or Cursor, makes no measurable difference to the company's structure. Only those that rethink their product and processes around AI operate with fewer employees and fewer hierarchical levels. The authors conclude that capturing the value of AI "often requires re-engineering processes around it".

An NBER study of 3,454 listed companies observes, in the same direction, that AI adoption goes hand in hand with fewer hierarchical levels, the tool taking over part of the problem solving. The authors stress that this is a correlation, not a causal link.

This finding echoes the analysis by Daron Acemoglu and Pascual Restrepo, in 2019, of so-called "so-so" automation: a technology that replaces human work with only a small productivity gain. Bolting an agent onto an unchanged engineering process would be a contemporary example.

Steering by value delivered rather than the illusion of volume

Since subjective judgement often overstates the return on AI, IT steering must refocus on what actually reaches production. Vendors now provide tools to measure usage and costs, such as the reports Anthropic launched for Claude Enterprise on 28 October 2026. These activity indicators must be cross-checked with objective delivery metrics: deployment frequency, time from commit to production, change failure rate, and real adoption by end users.

Our five recommendations for IT departments

• Measure value delivered rather than volume produced: track features live in production and their actual use, rather than lines of code or commits.
• Decide based on verification effort: delegate widely what is easy to test, and keep control of design, architecture and business rules.
• Strengthen expertise and review discipline: gains go first to engineers able to specify precisely and review without complacency; that is where training effort should go.
• Rework upstream processes before buying licences: a tool alone transforms nothing; it is the quality of requirements and a smooth integration chain that unlock value.
• Protect teams' attention: limit the number of agents running in parallel, and write down context and decisions so that agents work from unambiguous instructions.

The Adservio doctrine: an agent amplifies the chain as it is

The data converge: agents multiply the code written, but it is the maturity of the organisation that determines how much of that code becomes useful software.

An agent does not speed up a project in the abstract; it speeds up the delivery chain as it is, strengths and weaknesses alike. When functional scoping is vague, tests are unstable and reviews are already saturated, the agent clogs the system and piles up technical debt.

For Adservio, success does not start with deploying the tool, but with making what it produces verifiable. As a collective of augmented engineering experts, we apply this principle in our AI-DLC approach, the AI-driven development life cycle:

• executable requirements, written with business teams;
• tests derived from the system's expected behaviour, not from the generated code;
• blocking quality gates from the commit onwards;
• continuous verification in production, on real user journeys.

Our engineers work at the heart of your teams to steer agents alongside you on your projects, and to align automation with what creates measurable value. Our approach is detailed on our dedicated page.

Summary

Agents open an era of abundance in code production, but the gain erodes as you get closer to the product delivered and adopted. Success is not a matter of tooling: it rests on the expertise of those steering, the quality of verification and decision mechanisms, and the ability to redesign engineering processes around human attention. Gaining productivity in the age of agents is possible, provided performance is measured where it really plays out.