The problem of breaking changes at scale
Tools like Dependabot and Renovate handle minor version bumps well, but fail on major upgrades with breaking changes. A .NET 8 → 10 upgrade (the latest LTS cycle, with .NET 10 supported through November 2028) requires on average 180-250h of effort for a 500K LoC codebase.
Typical breakdown of a major .NET upgrade: - Impact analysis: 15-20h (identifying breaking changes) - Resolving compilation errors: 80-120h (API changes, deprecated methods) - Resolving runtime issues: 40-60h (behavioral changes, config updates) - Testing and validation: 45-50h (regression testing, performance validation)
Real observed cost: 30-40% of engineering capacity for 2-3 months for a team of 5 developers.
The challenge of major upgrades
Taking .NET version upgrades as an example, teams following the long-term support (LTS) cycle need to upgrade to the next major version once every three years. This involves updating every Microsoft package and, more often than not, every third-party package that has been upgraded to support the new .NET version.
The real cost of maintenance
Each of these package changes can introduce breaking changes at compile time and potentially at runtime that need to be resolved. For small components using stable features, this is typically fairly trivial. However, for larger components that depend on niche or preview features, resolving breaking changes can be a time-consuming task.
Upgrades like these can often take months. During this time, teams may need to pause new feature development to focus solely on framework updates.
At Adservio, we've seen client projects where a major .NET upgrade consumed 30 to 40% of engineering capacity for 2 to 3 months. That's time not spent delivering business value, building new features, or improving the user experience. It's purely technical maintenance necessary to avoid obsolescence and security risks.
The emergence of AI as a potential solution
It's no surprise, then, that with the emergence of agentic coding assistants, teams are starting to explore how GenAI could automate and accelerate this process. It can be tempting to ask an LLM to perform the entire upgrade in one go and hope for the best.
While this might work for small components, it will likely result in an incomplete upgrade at best for more complex solutions; at worst, it could lead to a confusing mess of changes with no clear connection to the original problem.
Can AI really help?
Our team set out to experiment with a variety of tools to determine the best way to leverage GenAI to assist with framework upgrades. After experimenting with all the major AI IDEs and extensions (Claude Code, Cursor, GitHub Copilot, Windsurf, Cline), we found that mixing deterministic and non-deterministic AI tools gives the best results.
The hybrid approach: Rule-based + orchestrated LLM, Let's return to the .NET version upgrade example. The start of the upgrade process should involve running the official .NET upgrade assistant (.NET Upgrade Assistant CLI) against all existing projects in the solution. While it doesn't handle third-party package upgrades, it will at least reliably update the target framework version (TargetFramework in .csproj) and first-party Microsoft packages (Microsoft.Extensions.*, System.Text.Json, etc.).
What comes next is what we found to be the most interesting engineering challenge. We asked a variety of tools and models (GPT-5.6, Claude Sonnet 5, Gemini 3 Pro) to perform the next step of the upgrade process to see how far they could get without strict guidance or deep context engineering.
Quantitative benchmarks
Results from our experiments: Quantitative benchmarks, For trivial components (<10K LoC, low coupling), the LLM was able to complete the remaining upgrade steps by following an iterative agentic loop (agent loop with tool calls: compile → analyze errors → fix → repeat).
For more complex components (>100K LoC, microservices with dependencies), some tools gave up after a few cycles (typically 5-8 iterations), asking the human for guidance on next steps. Other tools stumbled completely, either returning context window overflow errors or getting stuck in an endless hallucination loop (a "thrashing" pattern: undoing/redoing the same fix).
This variability in results reveals a fundamental truth: current LLMs are not reliable enough to autonomously handle the entire upgrade process, especially for complex real-world codebases.
Observed metrics: - Success rate for LLM alone: 12-18% (compiling + tests passing) - Success rate for deterministic tools: 45-52% - Success rate for orchestrated hybrid approach: 87-91% - Average time: 8-15h (LLM) vs 2-4h (hybrid) - Manual interventions: 25-40 (LLM) vs 2-5 (hybrid)
Experimenting with different prompts
Falling back on best practices for working with LLMs, we experimented with different prompts, trying to narrow the scope of the task and encouraging the model to follow a checklist-based strategic approach.
The evolution of prompting, We iterated further on the prompt, providing guidance on how to address certain breaking changes and how to reliably run the .NET build and test steps, to make sure it was considering the right feedback while reasoning about next steps.
This helped us go further with more complex codebases, but the AI once again struggled with much larger codebases. The fundamental problem was that even with carefully crafted prompts, the LLM tended to:
Lose focus after several iterations. Hallucinate solutions that didn't match the actual architecture. Get confused by complex or ambiguous error messages. Try to solve multiple problems simultaneously instead of proceeding methodically
The value of orchestration: The "Single Error Focus" pattern
We came to the conclusion that while an LLM is capable of performing subsequent upgrade steps and resolving breaking changes introduced by the upgrade, it cannot be trusted to perform all the steps autonomously (autonomous agent mode).
The orchestration approach: Bounded context + feedback loop, The approach that has yielded the greatest success so far was to build a CLI tool that orchestrated LLM invocations, strategically asking it to resolve build errors one by one and test failures one by one (single-error-focus pattern).
Orchestrator architecture: Concretely, the CLI tool chains four steps. It starts by running the official .NET Upgrade Assistant across the entire solution. It then compiles the project and extracts the structured list of errors. For each error, it first checks whether a known deterministic rule applies; if not, it invokes the LLM with a bounded context, applies the proposed fix, then re-validates via recompilation. Once all compilation errors are resolved, it runs the test suite and applies the same loop to test failures.
This way, the task it's supposed to solve is bounded and clear (bounded context: 1 error + relevant code), leaving little room for hallucinations to send it off track. Once the LLM has decided it has resolved a specific error, the orchestrator will double-check via recompilation and ask it to fix it again if needed (validation loop with a maximum of 3 retries).
The benefits of orchestration
In the case of .NET upgrades, there's a wealth of tooling investment that can analyze .NET build and test errors, which can be leveraged to ensure the LLM stays focused on a single specific error. This tooling can then be used to ensure the LLM has actually resolved that specific error.
Usable .NET tools for validation: - dotnet build with MSBuild diagnostic output parsing - Roslyn analyzers for code smell detection - dotnet test with JUnit XML result parsing - Code coverage tools (Coverlet, dotCover) for regressions
Another benefit of the orchestrator is that it can have built-in rules that handle specific build errors, with known resolutions that can be resolved deterministically. The less we rely on the LLM to resolve known errors, the faster and more reliable the upgrade process will be.
Examples of implemented deterministic rules: error CS0234 (missing namespace) is resolved by automatically adding the appropriate using directive, in 0.1 seconds versus 3 to 5 seconds of LLM reasoning. Error NU1605 (package conflict) is resolved by automatically downgrading to a compatible version, in 0.2 seconds versus 8 to 12 seconds for the LLM. Error CS1061 (member not found) is resolved via an API migration lookup table, in 0.3 seconds versus 5 to 10 seconds. Error CS0619 (obsolete API) is resolved by replacing it with the modern equivalent, in 0.5 seconds versus 10 to 15 seconds for the LLM.
Observed coverage: - Deterministic rules: 78-82% of compilation errors - LLM reasoning required: 18-22% (custom code, business logic)
Example of a deterministic rule, One example of these rules is resolving a package vulnerability by upgrading it to the latest version. The rule knows the specific error codes and can deterministically determine the impacted package and update it using the .NET CLI.
While in theory the LLM could also perform these steps, the solution it identifies cannot be guaranteed to be the same every time, while also being slower and more expensive to use.
The LLM as the orchestrator's architect
The most fascinating lesson from this experience is that the LLM used within the orchestrator is the same LLM that built the orchestrator itself.
An AI-driven bootstrap
The LLM was used to help design and build a CLI tool that would strategically invoke itself and other .NET tooling to perform a .NET upgrade. It then helped test the tool, iterate on it, and extend its functionality, after testing it against real complex codebases.
The tool can continue to be iterated on as new edge cases are identified, and the built-in rules can continue to be extended to reduce reliance on an LLM.
This meta approach, using AI to build tools that orchestrate AI, represents a powerful pattern we see emerging in several domains. At Adservio, we call this "AI-assisted AI engineering," and we believe it's a key direction for maximizing the value of GenAI while mitigating its limitations.
The solution's architecture
Let's break down how our orchestration tool works in practice:
Phase 1: Deterministic preparation, The tool starts by running the official .NET upgrade assistant against all projects. This is a deterministic step that ensures baseline changes are applied consistently.
Phase 2: Analysis and prioritization, The orchestrator compiles the project and analyzes the resulting errors. It classifies them into categories: - Errors with known rules: Resolved deterministically without invoking the LLM - Errors requiring reasoning: Queued for LLM processing - Ambiguous errors: Flagged for human review
Phase 3: Iterative resolution, For each error requiring reasoning, the orchestrator: 1. Isolates the relevant context (affected files, stack trace, documentation) 2. Provides this bounded context to the LLM with a structured prompt 3. Applies the proposed solution 4. Verifies that the specific error is resolved 5. If not resolved, iterates with additional feedback
Phase 4: Validation, Once all compilation errors are resolved, the orchestrator runs the full test suite and applies the same process to test failures.
Lessons learned and best practices
Our experimentation revealed several key insights on how to effectively combine AI and traditional tools.
1. Specificity is crucial, The more specific and bounded the task given to the LLM, the better the result. "Resolve this specific compilation error" works infinitely better than "Complete the upgrade."
2. Deterministic validation is essential, Never trust the LLM to self-assess its own success. Always use deterministic tooling to verify the problem is actually resolved.
3. Rules beat reasoning when possible, For any recurring pattern, create a deterministic rule. It's faster, more reliable, and less expensive than invoking an LLM every time.
4. Context should be curated, not dumped, Providing the LLM with the entire codebase as context is counterproductive. Carefully curate the relevant context for each specific task.
5. Human intervention remains crucial, For complex edge cases or architectural decisions, human judgment remains irreplaceable. Build escalation points into your orchestration.
Final thoughts
While the upgrade tool isn't guaranteed to handle every potential breaking change and often produces different results on the same codebase after each run, it performs the upgrade faster than a human can.
The role of AI in maintenance, It can therefore play a very useful role in helping organizations keep their software up to date and compliant. The hybrid approach, combining deterministic tooling with AI reasoning when needed, turns out to be the sweet spot.
When exploring how GenAI can help accelerate framework upgrades, first consider how far the existing tooling in the ecosystem can carry the upgrade, then look at how to strategically use an LLM to perform the steps that can't be performed deterministically.
We can't always expect the LLM to perform the entire upgrade on its own, not yet, at least.
Beyond .NET: Generalizing the approach
While our experiments focused on .NET, the principles apply broadly to other ecosystems:
JavaScript/Node.js: NPM packages frequently introduce breaking changes. Orchestration could combine tools like npm-check-updates with LLM reasoning for code migrations.
Python: Major Python version upgrades (e.g. 2 to 3, or 3.9 to 3.12) often require syntax and API changes. Tools like 2to3 could be orchestrated with AI for more complex cases.
Java/Spring: Spring framework upgrades introduce configuration and pattern changes. Orchestration could automate a large part of this work.
Ruby on Rails: Major Rails version upgrades often require significant changes. Orchestration could handle the majority of common patterns.
At Adservio, we're exploring how to extend this orchestration approach to other languages and frameworks, building a generic "AI-assisted framework upgrade" platform that adapts to the specifics of each ecosystem.
Conclusion: The future of framework maintenance
Automating framework upgrades via AI is still in its early days, but the direction is clear: the future isn't AI alone, nor traditional tools alone, but an intelligent orchestration of both.
As LLMs become more capable, we can expect the balance to shift: more tasks will be handled by AI reasoning, and fewer will require hard-coded rules. But the need for orchestration, breaking complex problems down into bounded tasks, deterministically validating results, and escalating to humans when necessary, will remain central.
Organizations that adopt this hybrid approach today will be the ones that keep their technology stack up to date with minimal friction, freeing up their engineering capacity to focus on what truly matters: delivering value to users.
Note: The statements and opinions expressed in this article are those of the author and do not necessarily reflect the positions of Adservio.
Tags: #Generative AI #Framework Upgrades #Orchestration #.NET #Software Maintenance
STAY POSTED
Get our next analyses and field notes straight to your inbox.
