# Deep Kernel-Level Root Cause Analysis with MCP > Investigating an incident down to the kernel: how the Model Context Protocol equips precise analysis rather than trial-and-error troubleshooting. - Date : 2025-09-16 - Lecture : 11 min - Catégorie : agents-ia - Tags : AI, Machine Learning, Security, Data - URL : https://www.adservio.fr/en/insights/articles/analyse-de-causalite-profonde-au-niveau-kernel-avec-mcp ## TL;DR - Adservio Managed Services integrated eBPF (via Groundcover) and AI agents connected through the Model Context Protocol (MCP) to accelerate root cause analysis (RCA) on incidents. - MCP lets AI agents move beyond static knowledge retrieval by querying production systems in real time (metrics, logs, traces). - A six-step collaborative workflow was put in place: triggering, triage planning, human validation, real-time execution, iterating on hypotheses, and final synthesis. - Federated search via MCP costs more tokens than classic indexed RAG, and combining both approaches yields the best triage results. - Prompt specificity remains decisive: vague instructions trigger hallucinations, while context-rich prompts produce accurate diagnoses in minutes. ## Introduction Modern distributed systems demand proactive, precise incident investigation to minimize downtime and reduce operational load. At Adservio Managed Services, troubleshooting is not just a technical necessity, it is a business-critical function. Effective root cause analysis (RCA) translates directly into cost savings and improved operational efficiency across downstream support activities. To deepen our investigative capabilities, we adopted technologies such as eBPF, which provide visibility into kernel-level behavior. Groundcover, a leader in this space, makes this detailed information accessible in real time without requiring deep kernel expertise from every engineer. In parallel, we integrated AI agents into our operational toolchain to augment and accelerate the work of our human engineers. With the emergence of the Model Context Protocol (MCP), a new standard is appearing that allows AI agents to move beyond static Q&A and interact directly with live systems and data sources. Access to kernel-level data via Groundcover has become a cornerstone for improving our AI agents toward smarter, more contextual incident investigation. In this blog post, we share the context, our approach, the integrated solution we built with Groundcover, and the results observed so far. ## Agentic AI: solving complex cognitive tasks In our day-to-day support operations, incident investigation typically begins with correlating logs to understand the underlying problem. Once an SRE engineer receives an incident alert from the incident management platform, they generally carry out a multidimensional investigation (Diagram 2) that includes: Metrics, Analyzing time-series data on dashboards to identify performance or resource utilization anomalies. Logs, Searching distributed log sources for error patterns, exceptions, or key terms related to the affected component. Traces, Following request traces to identify slow or failing segments and pinpoint the origin of user experience degradation. Events, Reviewing system and infrastructure events (e.g., restarts and deployments) that could correlate with the incident timeline. Topology, Mapping dependencies between services to understand the potential blast radius and upstream/downstream impact. Changes, Investigating recent configuration, code, or infrastructure changes that could explain behavior shifts. The results of these investigation activities generally lead to one of three categories of solutions: A short-term fix to mitigate the immediate problem. A recommendation to improve observability, such as adjusting alert thresholds or introducing new metrics. A longer-term improvement suggestion targeting underlying architectural or systemic weaknesses. Crafting these resolutions often requires drawing on various knowledge domains, including system design, observability best practices, and service level objectives. While this traditional approach is structured and thorough, it remains time-consuming, particularly when dealing with low-level system signals and fragmented toolchains. To address these challenges, we began adopting AI agents to support our incident workflows. While using agents to retrieve knowledge from multiple sources proved somewhat useful for suggesting solutions, we quickly realized a key limitation: effective investigation requires access to real-time, system-level data. Without up-to-date context from source systems, AI suggestions risk being either too generic or already outdated by the time they are generated. This led us to a critical question: How can we make investigation data collection more real-time and accurate, so AI agents can generate more relevant and actionable insights and solutions? ## MCP: the missing link for real-time AI reasoning This is where MCP, and its federated search capabilities, comes in. In incident investigation, O11y platforms like Groundcover provide a powerful foundation by combining eBPF-based kernel-level telemetry with contextual heuristics through relational retrieval. This gives investigators rich, real-time system context. Paired with AI agents enabled by the Groundcover MCP Server, this paradigm allows reasoning to move beyond simple static knowledge retrieval, enabling dynamic actions such as observability queries, live system lookups, and automated remediation (Diagram 2). The result: faster, more accurate, and more contextual decision-making during incident investigation. Our hypothesis, therefore, is: MCP-enabled AI agents can significantly improve the speed and accuracy of deep system investigations. > Related read: [Why MCP Is Critical for AI-Driven SRE](https://www.adservio.fr/en/insights/articles/pourquoi-mcp-est-essentiel-pour-la-sre-pilotee-par-l-ia): The Model Context Protocol gives an agent access to tools, memory and state. What that changes for AI-operated reliability. ## Co-creation with Groundcover With Groundcover, we built a solution to validate how AI agents can assist us in kernel-level incident investigation workflows. Our integrated solution demonstrates how AI agents use Groundcover and MCP to perform automated, contextual root cause analysis. Defining the use case, An AI agent collaborates with the Groundcover MCP server to perform a real-time, context-rich root cause analysis (RCA) by dynamically retrieving observability data, building and refining triage plans, and synthesizing insights into actionable resolutions. The workflow, Below is the workflow of an AI agent that leverages the Groundcover MCP server to drive a real-time, collaborative incident investigation. It retrieves observability signals, formulates triage strategies, and iteratively pushes toward root cause and resolution (Diagram 3). ### Steps 1 to 3: framing and validating the investigation Step 1. Trigger the investigation. A human agent initiates the investigation by asking a natural-language question (e.g., "Why is payment failing?"), which retrieves the most relevant active alert via Groundcover MCP. > Example prompt: Step 2. Draft the triage plan. An AI agent proposes a triage plan by exploring several investigation paths, each backed by specific data it needs (e.g., logs for error correlation, metrics for latency trends, traces for dependency failures). Step 3. Interactive plan validation. The human agent validates the AI's approach through iterative dialogue. In this experiment, it was found that overly generic prompts could lead to model hallucination. For example, a vague prompt like "using the groundcover mcp, can you diagnose what's happening with the cart workload" led the AI to incorrectly search for local files. The human agent refined the plan by providing a more specific prompt, guiding the AI to compare different time periods and focus on specific services. This correction is a critical part of the collaborative workflow. ### Steps 4 to 6: executing, iterating, and resolving Step 4. Real-time execution. Following the validated plan, the AI agent executes the investigation by making real-time calls to retrieve live telemetry. The Groundcover MCP server acts as a unified API gateway, letting the AI retrieve all necessary observability signals (metrics, traces, logs) from a single source without having to manage multiple endpoints. The AI's internal logs showing "Ran tool" multiple times are evidence of this real-time data retrieval. Step 5. Iterate on insights. Based on initial findings, the agent dynamically adjusts the triage path, looping as needed, incorporating new hypotheses and data to refine the diagnosis. The process is transparent and iterative, as shown in the log of the agent's thinking process. The agent forms a hypothesis (e.g., "let me get more details on the Redis connection errors"), runs a series of queries against different data sources (query_logs, query_traces, query_metrics), then uses these findings to form a new, more refined hypothesis (e.g., "now let me check memory usage patterns"). This iterative loop of "think → query → refined thinking" is what allows the agent to move beyond surface-level symptoms. It is exactly this process that let it pivot from investigating simple connection errors to uncovering the true root cause of memory pressure and synthesizing the "Cascade Failure Pattern" mentioned in the final summary. Step 6. Summarize and resolve. Finally, the agent consolidates all findings into a clear, actionable summary. This includes identifying the root cause, providing evidence, and offering a multi-layered resolution plan with immediate fixes, workarounds, and long-term improvements. This comprehensive summary can then be used for stakeholder communication or ticket creation. ## Key learnings We have definitely drawn a few key learnings from the MCP-enabled incident investigation experience. They include: ### Performance and cost efficiency Federated search via MCP provides powerful real-time context but comes with a higher token cost compared to traditional RAG-based indexed search. This occasionally led to degraded performance and inconsistent results, highlighting the need for smarter cost-control strategies in prompt execution. ### Smarter triage by leveraging RAG Augmenting MCP with RAG-enabled indexed search from historical investigations proved highly effective. It provided context-rich triage plans, improving relevance, efficiency, and the initial direction of investigations. ### Security gaps The current solution lacks essential security capabilities, including access controls, identity management, and auditing. Integrating these safeguards is critical to making MCP solutions enterprise-ready. ### User experience Long, unstructured interactions with AI agents placed a significant cognitive load on human operators. The lack of synthesis, UX cues, decision support, or guided workflows highlighted the urgent need for a better user experience layer. ### Prompt specificity is paramount The accuracy of AI results was directly tied to the clarity of the prompts. Vague instructions like "diagnose the workload" often led to hallucinations, while context-rich prompts, including logs, error codes, and time frames, produced accurate, actionable insights. This reaffirms that prompt engineering is essential to model performance. ### AI excels at synthesizing complex scenarios The model demonstrated a strong ability to move beyond simple data retrieval. It successfully correlated data across different domains (logs, traces, metrics) to build a holistic narrative. Its creation of the "Cascade Failure Pattern" is an excellent example of its ability to synthesize a complex chain of events into an easy-to-grasp concept, a task that is very time-consuming for human agents. ### A transparent thinking process builds trust A key differentiator in successful interactions was the AI narrating its own workflow (e.g., "Let me start by..."). The transparency of its "thinking process" builds trust and lets the engineer understand how the AI reached its conclusions, making it a more effective and trustworthy partner in a diagnostic scenario. ### Drastic reduction in resolution time For well-defined investigations, the experiment confirmed that AI could deliver a detailed RCA with actionable recommendations within minutes. This represents a significant reduction compared to the hours an engineer might typically spend on manual data correlation. There is clearly significant business value in accelerating incident response. > Related read: [Autonomous SRE in 2026: agentic incident response](https://www.adservio.fr/en/insights/articles/sre-autonome-incident-response-agentique): AI agents wired to observability correlate telemetry, code and deployments to triage and remediate incidents. Alert fatigue down 40-60%, MTTR falling. ## Looking ahead: AI as an investigation companion We believe AI is redefining the future of IT operations. Its impact is deeper and more transformative than many realize. While technologies like LLMs and RAG have strengthened AI systems with indexed and contextual search, MCP goes even further, enabling federated, real-time access to live enterprise systems. This shift elevates AI agents from simple responders to proactive, investigative companions that operate with timely, system-aware intelligence. When combined with powerful telemetry technologies like eBPF, MCP-enabled AI agents unlock a new paradigm in incident investigation, moving from static analysis to dynamic, context-rich reasoning. At Adservio Managed Services, we continuously integrate the best AI technologies to improve operational excellence, helping our clients reduce costs while maintaining high service reliability. AI, when properly supported, becomes far more than a tool, it becomes a true partner in detection, diagnosis, and prevention. Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio. ## FAQ ### What does the Model Context Protocol (MCP) bring to root cause analysis? MCP is a standard that allows AI agents to interact directly with live systems and data sources, rather than being limited to static answers. Paired with Groundcover, it gives real-time access to the kernel-level telemetry (metrics, logs, traces) needed for accurate RCA. ### What are the main risks observed with MCP-enabled AI agents? Overly generic prompts can cause model hallucinations, federated search costs more tokens than classic indexed RAG, and the tested solution lacked essential security capabilities such as access controls, identity management, and auditing. ### How much time can AI-assisted RCA save? For well-defined investigations, the AI agent delivered a detailed root cause analysis with actionable recommendations within minutes, compared to several hours for manual data correlation by an engineer.