Introduction
Modern distributed systems demand proactive, precise incident investigation to minimize downtime and reduce operational load. At Adservio Managed Services, troubleshooting is not just a technical necessity, it is a business-critical function. Effective root cause analysis (RCA) translates directly into cost savings and improved operational efficiency across downstream support activities.
To deepen our investigative capabilities, we adopted technologies such as eBPF, which provide visibility into kernel-level behavior. Groundcover, a leader in this space, makes this detailed information accessible in real time without requiring deep kernel expertise from every engineer.
In parallel, we integrated AI agents into our operational toolchain to augment and accelerate the work of our human engineers. With the emergence of the Model Context Protocol (MCP), a new standard is appearing that allows AI agents to move beyond static Q&A and interact directly with live systems and data sources.
Access to kernel-level data via Groundcover has become a cornerstone for improving our AI agents toward smarter, more contextual incident investigation. In this blog post, we share the context, our approach, the integrated solution we built with Groundcover, and the results observed so far.
Agentic AI: solving complex cognitive tasks
In our day-to-day support operations, incident investigation typically begins with correlating logs to understand the underlying problem. Once an SRE engineer receives an incident alert from the incident management platform, they generally carry out a multidimensional investigation (Diagram 2) that includes:
Metrics, Analyzing time-series data on dashboards to identify performance or resource utilization anomalies. Logs, Searching distributed log sources for error patterns, exceptions, or key terms related to the affected component. Traces, Following request traces to identify slow or failing segments and pinpoint the origin of user experience degradation. Events, Reviewing system and infrastructure events (e.g., restarts and deployments) that could correlate with the incident timeline. Topology, Mapping dependencies between services to understand the potential blast radius and upstream/downstream impact. Changes, Investigating recent configuration, code, or infrastructure changes that could explain behavior shifts.
The results of these investigation activities generally lead to one of three categories of solutions:
A short-term fix to mitigate the immediate problem. A recommendation to improve observability, such as adjusting alert thresholds or introducing new metrics. A longer-term improvement suggestion targeting underlying architectural or systemic weaknesses.
Crafting these resolutions often requires drawing on various knowledge domains, including system design, observability best practices, and service level objectives. While this traditional approach is structured and thorough, it remains time-consuming, particularly when dealing with low-level system signals and fragmented toolchains.
To address these challenges, we began adopting AI agents to support our incident workflows. While using agents to retrieve knowledge from multiple sources proved somewhat useful for suggesting solutions, we quickly realized a key limitation: effective investigation requires access to real-time, system-level data. Without up-to-date context from source systems, AI suggestions risk being either too generic or already outdated by the time they are generated.
This led us to a critical question:
How can we make investigation data collection more real-time and accurate, so AI agents can generate more relevant and actionable insights and solutions?
MCP: the missing link for real-time AI reasoning
This is where MCP, and its federated search capabilities, comes in.
In incident investigation, O11y platforms like Groundcover provide a powerful foundation by combining eBPF-based kernel-level telemetry with contextual heuristics through relational retrieval. This gives investigators rich, real-time system context.
Paired with AI agents enabled by the Groundcover MCP Server, this paradigm allows reasoning to move beyond simple static knowledge retrieval, enabling dynamic actions such as observability queries, live system lookups, and automated remediation (Diagram 2).
The result: faster, more accurate, and more contextual decision-making during incident investigation.
Our hypothesis, therefore, is:
MCP-enabled AI agents can significantly improve the speed and accuracy of deep system investigations.
Co-creation with Groundcover
With Groundcover, we built a solution to validate how AI agents can assist us in kernel-level incident investigation workflows. Our integrated solution demonstrates how AI agents use Groundcover and MCP to perform automated, contextual root cause analysis.
Defining the use case, An AI agent collaborates with the Groundcover MCP server to perform a real-time, context-rich root cause analysis (RCA) by dynamically retrieving observability data, building and refining triage plans, and synthesizing insights into actionable resolutions.
The workflow, Below is the workflow of an AI agent that leverages the Groundcover MCP server to drive a real-time, collaborative incident investigation. It retrieves observability signals, formulates triage strategies, and iteratively pushes toward root cause and resolution (Diagram 3).
Steps 1 to 3: framing and validating the investigation
Step 1. Trigger the investigation. A human agent initiates the investigation by asking a natural-language question (e.g., "Why is payment failing?"), which retrieves the most relevant active alert via Groundcover MCP.
> Example prompt:
Step 2. Draft the triage plan. An AI agent proposes a triage plan by exploring several investigation paths, each backed by specific data it needs (e.g., logs for error correlation, metrics for latency trends, traces for dependency failures).
Step 3. Interactive plan validation. The human agent validates the AI's approach through iterative dialogue. In this experiment, it was found that overly generic prompts could lead to model hallucination. For example, a vague prompt like "using the groundcover mcp, can you diagnose what's happening with the cart workload" led the AI to incorrectly search for local files.
The human agent refined the plan by providing a more specific prompt, guiding the AI to compare different time periods and focus on specific services. This correction is a critical part of the collaborative workflow.
Steps 4 to 6: executing, iterating, and resolving
Step 4. Real-time execution. Following the validated plan, the AI agent executes the investigation by making real-time calls to retrieve live telemetry. The Groundcover MCP server acts as a unified API gateway, letting the AI retrieve all necessary observability signals (metrics, traces, logs) from a single source without having to manage multiple endpoints. The AI's internal logs showing "Ran tool" multiple times are evidence of this real-time data retrieval.
Step 5. Iterate on insights. Based on initial findings, the agent dynamically adjusts the triage path, looping as needed, incorporating new hypotheses and data to refine the diagnosis. The process is transparent and iterative, as shown in the log of the agent's thinking process.
The agent forms a hypothesis (e.g., "let me get more details on the Redis connection errors"), runs a series of queries against different data sources (query_logs, query_traces, query_metrics), then uses these findings to form a new, more refined hypothesis (e.g., "now let me check memory usage patterns").
This iterative loop of "think → query → refined thinking" is what allows the agent to move beyond surface-level symptoms. It is exactly this process that let it pivot from investigating simple connection errors to uncovering the true root cause of memory pressure and synthesizing the "Cascade Failure Pattern" mentioned in the final summary.
Step 6. Summarize and resolve. Finally, the agent consolidates all findings into a clear, actionable summary. This includes identifying the root cause, providing evidence, and offering a multi-layered resolution plan with immediate fixes, workarounds, and long-term improvements. This comprehensive summary can then be used for stakeholder communication or ticket creation.
Key learnings
We have definitely drawn a few key learnings from the MCP-enabled incident investigation experience. They include:
Performance and cost efficiency
Federated search via MCP provides powerful real-time context but comes with a higher token cost compared to traditional RAG-based indexed search. This occasionally led to degraded performance and inconsistent results, highlighting the need for smarter cost-control strategies in prompt execution.
Smarter triage by leveraging RAG
Augmenting MCP with RAG-enabled indexed search from historical investigations proved highly effective. It provided context-rich triage plans, improving relevance, efficiency, and the initial direction of investigations.
Security gaps
The current solution lacks essential security capabilities, including access controls, identity management, and auditing. Integrating these safeguards is critical to making MCP solutions enterprise-ready.
User experience
Long, unstructured interactions with AI agents placed a significant cognitive load on human operators. The lack of synthesis, UX cues, decision support, or guided workflows highlighted the urgent need for a better user experience layer.
Prompt specificity is paramount
The accuracy of AI results was directly tied to the clarity of the prompts. Vague instructions like "diagnose the workload" often led to hallucinations, while context-rich prompts, including logs, error codes, and time frames, produced accurate, actionable insights. This reaffirms that prompt engineering is essential to model performance.
AI excels at synthesizing complex scenarios
The model demonstrated a strong ability to move beyond simple data retrieval. It successfully correlated data across different domains (logs, traces, metrics) to build a holistic narrative. Its creation of the "Cascade Failure Pattern" is an excellent example of its ability to synthesize a complex chain of events into an easy-to-grasp concept, a task that is very time-consuming for human agents.
A transparent thinking process builds trust
A key differentiator in successful interactions was the AI narrating its own workflow (e.g., "Let me start by..."). The transparency of its "thinking process" builds trust and lets the engineer understand how the AI reached its conclusions, making it a more effective and trustworthy partner in a diagnostic scenario.
Drastic reduction in resolution time
For well-defined investigations, the experiment confirmed that AI could deliver a detailed RCA with actionable recommendations within minutes. This represents a significant reduction compared to the hours an engineer might typically spend on manual data correlation. There is clearly significant business value in accelerating incident response.
Looking ahead: AI as an investigation companion
We believe AI is redefining the future of IT operations. Its impact is deeper and more transformative than many realize. While technologies like LLMs and RAG have strengthened AI systems with indexed and contextual search, MCP goes even further, enabling federated, real-time access to live enterprise systems. This shift elevates AI agents from simple responders to proactive, investigative companions that operate with timely, system-aware intelligence.
When combined with powerful telemetry technologies like eBPF, MCP-enabled AI agents unlock a new paradigm in incident investigation, moving from static analysis to dynamic, context-rich reasoning.
At Adservio Managed Services, we continuously integrate the best AI technologies to improve operational excellence, helping our clients reduce costs while maintaining high service reliability.
AI, when properly supported, becomes far more than a tool, it becomes a true partner in detection, diagnosis, and prevention.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
