# Claude Code saved us 97% of the work, then it became a complete disaster > Adservio's field report on Claude Code: Python support added to CodeConcise in minutes, a complete failure on JavaScript, and the lessons validated in 2026. - Date : 2025-09-21 - Lecture : 9 min - Catégorie : agents-ia - Tags : AI, Coding agents, Claude Code, Legacy code, Testing - URL : https://www.adservio.fr/en/insights/articles/claude-code-nous-a-economise-97-du-travail-puis-c-est-devenu ## TL;DR - In early 2025, an Adservio team trialed Claude Code on CodeConcise, an internal legacy-code analysis tool, to add support for new programming languages. - For Python, the agent and a subject-matter expert produced in a few minutes what used to take two developers two to four weeks, roughly 97% of the work saved. - For JavaScript, three attempts failed: a non-functional ANTLR grammar, nonexistent tree-sitter libraries, purely invented internal packages. - Independent feedback loops, end-to-end tests, human review, continuous integration, remain essential to catch the agent's mistakes before they propagate. - With the hindsight of 2026, the success factors identified, quality of the existing code, library ecosystem, training data, human collaboration, have been confirmed at scale. ## Claude Code on a legacy codebase: the context of the experiment In early 2025, a few weeks after its launch by Anthropic, we experimented with Claude Code on CodeConcise, a tool we built at Adservio to understand legacy codebases. The goal was to determine whether a coding agent could accelerate a well-scoped but time-consuming task: adding support for new programming languages. Typically, this work took a two-developer team two to four weeks. With Claude Code and a subject-matter expert, it took only a few minutes of generation and a few hours of validation for the Python language, roughly 97% of the work saved. But the same approach applied to JavaScript ended in complete failure, across three successive attempts. That spectacular contrast is precisely what makes the experience instructive. We are republishing this field report with the hindsight of July 2026: the tooling has matured considerably, but the underlying lessons, about feedback loops, the quality of the existing code and the structural limits of agents, remain surprisingly current. We have kept the original narrative and updated the way we look at it. ## Claude Code and the 2026 landscape of supervised coding agents Claude Code is a coding agent developed by Anthropic and launched on February 24, 2025. It is an example of what is called a "supervised coding agent": a tool capable of carrying out sophisticated tasks within software development workflows, sometimes autonomously, under the control of a developer who frames the request and validates the key steps. ### Terminal, IDE or cloud: three families of agents Its original distinguishing feature, running from the command line rather than inside an IDE, has become a category in its own right. In 2026, three families of agents coexist: terminal agents (Claude Code, OpenAI Codex CLI, Gemini CLI, and open-source tools such as Aider or Goose), IDE-integrated agents (Cursor, GitHub Copilot, whose Agent Mode has been generally available on VS Code and JetBrains since March 2026,or Windsurf, which became Devin Desktop after its acquisition by Cognition), and cloud agents that take a task end to end in their own execution environment, such as Devin. The terminal remains the best integration point into a tool ecosystem broader than an IDE. ### Models that have changed scale since our tests At the time of our experiment, Claude Code relied on the Claude Sonnet models of the day. By mid-2026, it defaults to Claude Opus 4.8 on Max and API plans, and to Claude Sonnet 5, with its native one-million-token context window, on Pro subscriptions. The Model Context Protocol (MCP), still emerging when we ran our tests, has established itself as the standard for connecting agents to data sources, adopted across the entire ecosystem. An agent's performance still depends on the same fundamentals: orchestrating code context, crafting effective requests to the model, and integrating with context providers. > Related read: [How Much Faster Can AI Coding Assistants Really Make Software Delivery?](https://www.adservio.fr/en/insights/articles/a-quelle-vitesse-les-assistants-de-codage): A heuristic, a 150-ticket case study, and 2025-2026 research: real speed gains from AI coding assistants sit between 5 and 15%, far below the marketing hype. ## CodeConcise: mapping legacy code with ASTs and GraphRAG To understand the experiment, CodeConcise deserves a brief introduction. The tool uses abstract syntax trees (ASTs) not only to break down codebases and extract their structure, but also to navigate those codebases when an LLM is used to summarize and explain them. Concretely, CodeConcise requires language-specific implementations of an abstract class named IngestionTool. Each implementation returns a list of nodes and edges extracted from the code, which feed a knowledge graph. Language-agnostic traversals then enrich this graph with information extracted from the code with the help of an LLM. The graph is finally consumed by an agent and by GraphRAG to deliver insights to users, an architecture that remains, in 2026, one of the reference patterns for large-scale legacy code analysis. ### Why adding a language used to cost two to four weeks Every newly supported language requires writing the code that produces and interprets its AST: finding representative sample codebases, building an automated test suite, then implementing the ingestion tool. Count two to four weeks for a team of two developers. That is why support was only added on demand, and that is the hypothesis we wanted to test: if an agent drastically reduced this cost, we could cover the majority of languages upfront and devote our time to higher-value work. ## The Python success: weeks of work reduced to a few minutes First request: identify the changes needed to add Python support. The question may seem simple for a developer who has known the codebase for a few weeks, but it is invaluable for a newcomer to the project. With access to the code and its documentation, Claude Code produced stunning results: it correctly identified all the necessary changes, and the suggested code showed that it had not merely inspected the existing ingestion tools, but had also absorbed the design patterns used to implement them. ### Three minutes of autonomous work, two defects caught by our tests Second request: implement the changes itself. "I need to build a new tool to load Python code into CodeConcise. Please do this and test it." A little over three minutes later, all the changes were implemented locally, tests included, and every test Claude Code suggested passed. But when we loaded its own source code into a knowledge graph and ran our end-to-end tests, two problems surfaced: the file system structure was not part of the graph, and the edges connecting nodes did not follow our model, call dependencies were missing, which would have prevented the comprehension pipeline from traversing the graph as expected. This discovery illustrates the importance of multiple feedback loops when an AI writes code. Without our own tests, the problem would only have been discovered much later, in a disruptive and costly way, with both the developer and the agent having lost the context of the work in progress. After feedback, the code was fixed within seconds, faithfully following existing patterns such as using Observers to build the file system structure as the code is parsed. It is worth stressing that, for this use case, a large part of the complex thinking had already been done by the tool's designers: the business logic was cleanly separated from the language-specific implementation details. Claude Code "only" had to gather that information and understand, from the existing design, what needed to be built. The bottom line: a few minutes of generation and a handful of hours of validation, where weeks used to be needed. ## The JavaScript failure: broken grammars and invented libraries Encouraged by this success, we tried the same approach for JavaScript, with a precise prompt: use the StageObserver as elsewhere and apply the visitor pattern to the lexer and parser, as in the TSQL loader. First attempt: Claude Code relied on the ANTLR grammar for JavaScript, which we never managed to get working, beyond the agent's responsibility, but it prevented us from verifying the generated code. Second attempt, asking it to use tree-sitter instead: the agent started using libraries that do not exist. Third and final attempt, letting it choose its own approach: it opted for regular-expression parsing… whose code referenced purely invented internal packages. Three tries, three dead ends, no verifiable code. ### Hallucinations delivered with the confidence of facts The takeaway was clear: the agent lacked a robust verification mechanism for the code it produces, as if it deprived itself of the feedback a simple unit test would provide. This behavior was nothing new, we had observed it several times with other coding assistants. To check that the Python success was not a stroke of luck, we repeated the exercise in C: results comparable to Python's, although obtained through regular-expression matching rather than a more reliable AST-based approach. > Related read: [We Need to Treat AI Hallucinations as a Feature, Not a Bug](https://www.adservio.fr/en/insights/articles/il-faut-traiter-les-hallucinations-de-l-ia-comme): AI hallucination isn't a bug but a property of probabilistic models: a risk matrix, RAG grounding, and governance are what it takes to keep it under control. ## Six factors that determine the quality of agent-generated code This experience covers a very specific use case: let us be careful not to draw unwarranted general conclusions. But the spectacular gap between Python and JavaScript is explained by factors that eighteen months of practice have since largely confirmed. The quality of the existing code, first: modular, clean, documented code designed with a clear separation of concerns maximizes the likelihood of good results, what we have always considered important for humans also matters for agents. The library ecosystem, next: Python ships a standard module for converting code into an AST, whereas JavaScript forced reliance on fragile external grammars. Training data, finally: LLMs produce better code for languages massively represented in their original corpus. Add three cross-cutting factors: the underlying model, whose performance directly conditions the agent's; the agent itself, the engineering of its internal prompts, the design of its workflows and the tools made available to it; and human collaboration, because pairing experienced developers with agents yields the best of both worlds, the former knowing the tricks that guide the latter toward better results. > Related read: [How to Build Trust With AI-Powered Coding Assistants](https://www.adservio.fr/en/insights/articles/comment-cultiver-la-confiance-avec-les-assistants-de-codage): Why adoption rates for coding assistants stall, and what actually builds a team's trust in the tool. ## What 2026 confirmed: and what has changed since the experiment Eighteen months later, the core of the diagnosis still holds. Agents have gained built-in verification mechanisms, automatic execution of tests and compilers, validation hooks, review subagents, that would probably have intercepted the invented libraries of our JavaScript failure before they reached the developer. One-million-token context windows and the widespread adoption of MCP have, for their part, reduced the context losses that used to penalize long working sessions. But the fundamental lesson is unchanged: no agent replaces independent feedback loops. End-to-end tests, human review and continuous integration remain the only guarantee that generated code actually does what it claims to do. Generation speed has changed by an order of magnitude; the responsibility for validation has not moved, and that is precisely where we focus our coding-agent adoption engagements today. Thanks to Birgitta Böckeler for her support and guidance during our experiments. Disclaimer: the statements and opinions expressed in this article are those of the authors and do not necessarily reflect Adservio's positions. ## FAQ ### What is Claude Code, and how does it differ from other coding agents? Claude Code is a coding agent from Anthropic, launched in February 2025, that runs from the command line rather than inside an IDE. In 2026 it defaults to Claude Opus 4.8 or Claude Sonnet 5 and coexists with other terminal agents (OpenAI Codex CLI, Gemini CLI, Aider, Goose), IDE-integrated agents (Cursor, GitHub Copilot Agent Mode, Devin Desktop) and cloud agents such as Devin. ### Why did Claude Code succeed on Python but fail on JavaScript? For Python, the agent could rely on the language's standard AST module, on an architecture cleanly separated between business logic and implementation details, and on abundant training data. For JavaScript, it strung together three failures: a non-functional ANTLR grammar, nonexistent tree-sitter libraries, then a regular-expression approach referencing invented internal packages. ### What role do tests play in the reliability of agent-generated code? Automated tests and human feedback loops remain essential: without them, structural defects, such as graph edges that do not match the expected model, could have gone unnoticed for a long time, with a much higher cost to fix once the developer and the agent had both lost the context of the work. ### Are the lessons from this experiment still valid in 2026? Yes. The agents of 2026 embed more automatic verification (test execution, hooks, review subagents), which reduces library hallucinations, but the success factors identified, quality of the existing code, library ecosystem, human collaboration, and the need for independent feedback loops remain fully relevant.