# Observability patterns for distributed systems: logs, metrics, traces and profiles > Logs, metrics, traces and profiles: the observability patterns that make a distributed system understandable, with OpenTelemetry as the common standard. - Date : 2023-03-23 - Lecture : 9 min - Catégorie : devsecops - Tags : Observability, Distributed Systems, Microservices, OpenTelemetry, Logs, Metrics, Tracing, Continuous profiling, eBPF - URL : https://www.adservio.fr/en/insights/articles/patterns-observabilite-systemes-distribues ## TL;DR - Observability is the ability to understand the internal state of a distributed system from its outputs: logs, metrics, traces and now profiles. - Legacy monitoring watches known thresholds; observability lets you query the system when facing failures you had not anticipated. - Structured log aggregation, application metrics and distributed tracing remain the three founding patterns, unified by OpenTelemetry. - Metric cardinality and trace volume are kept under control through label discipline and the right sampling strategy, head-based or tail-based. - Continuous profiling, OpenTelemetry's fourth signal, has been in public alpha since March 2026 with general availability targeted for the third quarter of 2026. - AI is transforming how this data is used: automatic correlation across signals, alert triage and SLO-driven alerting instead of static thresholds. ## Why traditional monitoring no longer keeps up with microservices Implementing observability for modern distributed systems raises real challenges. Legacy monitoring solutions, designed to watch a few servers against thresholds known in advance, fail to provide adequate visibility into the dependencies between microservices, message queues, serverless functions and third-party API calls that make up a current architecture. A user request routinely crosses dozens of services: when it fails, no static dashboard tells you which one is at fault. The question then becomes: how do you collect the relevant data without exploding costs, visualise it meaningfully and correlate it to trace an incident back to its cause? That is exactly what observability patterns are for, proven techniques, now standardised around OpenTelemetry, for instrumenting, aggregating and analysing the signals of a distributed system. Added to this is an economic challenge few teams anticipate: telemetry data grows faster than the traffic it describes, and it is not unusual for a poorly calibrated observability platform to end up costing more than the infrastructure it watches. The patterns presented here therefore answer not only a question of visibility but also one of control: deciding what to collect, at what granularity, for what retention period and, above all, to answer which questions. ## What an observability pattern is and what it brings Observability is defined as the ability to understand the internal state of a system from its outputs: activity records, numeric measurements, execution traces, timestamps and related metadata. An observability pattern is a specific technique for collecting and exploiting those outputs consistently across dozens of heterogeneous services. ### From monitoring to observability: from known failures to unknown ones The distinction from classic monitoring fits in one sentence: monitoring answers questions asked in advance, is CPU above 80%?,while observability lets you ask new questions when facing an unprecedented incident. Its benefits are concrete: troubleshooting fed by correlated information, proactive detection of degradations before they reach users, and vital data for business decisions, conversion rates, perceived latency, cost per transaction. The four signals complement each other more than they compete: a metric detects that a problem exists, a trace locates the service at fault, a log explains what happened there and a profile points to the responsible code. An incident is resolved all the faster when you can navigate from one to the other without switching tools or losing context, that is the whole point of cross-signal correlation, and the reason shared naming and identifier conventions matter more than the choice of any particular backend. ## Structured log aggregation, the foundation of analysis A log is a timestamped record of the events that occur when an application runs. The current standard is the structured JSON log, enriched with correlation identifiers, trace ID, request ID, tenant, that link each line to the other signals. It is the easiest pattern to implement, and yet the most often misused: free-text logs, without structure or context, are barely better than no logs at all. Discipline matters here as much as tooling: log levels kept consistent across services, messages that describe the event rather than the developer's mood, and a strict ban on writing personal data or secrets into them. Sampling repetitive messages at the source also prevents a single error loop from generating terabytes of uninterpretable output on its own, with a matching bill at the end of the month. ### Treating logs as event streams The recommended approach is to treat logs as events: collected by the OpenTelemetry Collector or transported through Kafka, they feed both troubleshooting and business intelligence. On the storage side, column-oriented databases such as ClickHouse and label-indexed stores such as Loki have replaced costly full-text indexes, with tiered retention, hot storage for recent investigation, cold object storage for compliance, that cuts costs without sacrificing history. > Related read: [The 3 pillars of observability: logs, metrics and traces](https://www.adservio.fr/en/insights/articles/les-3-piliers-de-l-observabilite): Logs, metrics and traces form the three pillars of observability. Strengths, limits, correlation and unification through OpenTelemetry: the 2026 guide. ## Application metrics and the cardinality challenge Metrics are numeric measurements aggregated over time intervals: availability, load, CPU and memory usage, error rates. Prometheus and its PromQL language remain the de facto standard for collecting and querying them, with long-term backends such as Mimir or Thanos for multi-cluster scale. Since Prometheus 3, the server natively ingests OTLP data, which simplifies architectures where the two ecosystems coexist. ### From system metrics to business metrics There are three complementary categories: system metrics (host and process saturation), resource metrics (memory, disk, cloud quotas) and business metrics (API errors, processing time, request frequency). The RED framework, requests, errors, duration, for services and USE, utilisation, saturation, errors, for resources provide an immediate reading grid: four or five well-chosen metrics per service are enough to detect most degradations. ### Keeping cardinality under control without losing the signal The main challenge remains cardinality: every label combination creates a distinct time series, and one poorly chosen label, a user identifier, a full URL, can multiply costs a hundredfold. The remedy is a continuously reviewed label discipline, aggregating useless dimensions at collection time and, where detail matters, shifting it to traces or wide events rather than metrics. Exemplars bridge these worlds: attached to measurement points, they link a latency spike visible on a Prometheus graph to the exact traces that caused it. This chaining transforms investigation, you move from the aggregated symptom to the concrete case in one click, instead of rebuilding the correlation by hand across two tools, two windows and two time scales. ## Distributed tracing, the thread running through requests Tracing means following a request end to end through the system: each service crossed produces timestamped spans, linked by context propagated according to the W3C Trace Context standard. The trace reveals the services involved, the error paths and the slowest segments, often an external dependency or an N+1 query that neither logs nor metrics showed. The automatic instrumentation OpenTelemetry provides for common frameworks covers the essentials without touching application code. Beyond one-off diagnosis, aggregated traces draw the living map of the system: the service graph reveals the real dependencies, often different from the documented architecture, the critical paths and the unsuspected couplings between domains. This cartography serves well beyond incidents: architecture reviews, impact analysis before a migration, identification of orphan services nobody calls any more. ### Sampling: head-based or tail-based Tracing every request of a high-traffic system would cost more than the infrastructure it observes. Head-based sampling decides at the entry point, a fixed percentage of requests, simple but blind. Tail-based sampling decides after the fact, in the Collector, and systematically keeps traces that errored or ran abnormally slow: you keep the useful signal while discarding redundant nominal traffic. Combined with RED metrics and SLO-based alerts, tracing becomes a proactive diagnostic tool rather than an expensive archive. > Related read: [Managing observability and resilience in distributed systems](https://www.adservio.fr/en/insights/articles/observabilite-et-resilience-systemes-distribues): Observability, distributed tracing and event-driven architecture: the foundations for making a distributed system resilient, understanding its failures and durably cutting production incidents. ## OpenTelemetry, continuous profiling and AI-augmented observability OpenTelemetry, a graduated CNCF project, has established itself as the single instrumentation standard: one SDK, one protocol (OTLP) and shared semantic conventions for traces, metrics and logs, all three stable. This foundation frees you from proprietary lock-in: instrument once, then choose, and change, your analysis backend freely. The Collector, deployed as a local agent or a central gateway, concentrates the reception, transformation, sampling and export of signals: it is the single control point where filtering, enrichment and routing rules apply to all telemetry. ### Continuous profiling, the fourth signal The fourth signal is arriving: profiles, in public alpha since March 2026 with general availability targeted for the third quarter of 2026, continuously capture CPU and memory consumption at function level through eBPF agents, without application instrumentation. Where a trace shows which service is slow, a profile shows which line of code is burning the resources. What remains is exploiting this mass of data: this is where AI changes the practice. Platforms automatically correlate signals around an incident, group redundant alerts and propose root-cause hypotheses; triage agents prepare the context before a human even opens the dashboard. Alerting shifts from static thresholds to error budgets: you alert when the promised user experience is at risk, not when a CPU flickers. Humans remain the arbiter: AI prepares the diagnosis and shortens the investigation, but it replaces neither the understanding of the system nor the decision to act. > Related read: [Bridging the SRE Gap: Toward Autonomous Observability and AI-Agent Root Cause Analysis](https://www.adservio.fr/en/insights/articles/combler-l-ecart-sre-vers-l-observabilite-autonome): Autonomous observability: how an AI agent correlates logs, metrics and traces to automate root cause analysis and cut MTTR from hours down to minutes. ## Implementing these patterns with Adservio Before distributed systems, a single log file was enough to diagnose a problem. The cloud changed that: hundreds of services spread across several environments generate massive data volumes, and the risk is no longer lacking information but drowning in it, or paying to store it without ever using it. The right starting point is never the tool but the questions the team must be able to answer within three minutes in the middle of the night: which service is at fault, since when, for which users and because of which change. At Adservio, we help organisations implement these observability patterns in the right order: OpenTelemetry instrumentation, cardinality discipline, sampling suited to real traffic and SLO-based alerting, while avoiding the costly traps, runaway cardinality, default retention, ignored alerts, that turn so many observability projects into cost centres. ## FAQ ### What is an observability pattern? It is a specific technique for collecting and exploiting the outputs of a distributed system consistently: structured log aggregation, application metrics, distributed tracing and, more recently, continuous profiling, all unified by OpenTelemetry. ### What is the difference between monitoring and observability? Monitoring answers questions asked in advance, such as a CPU threshold. Observability lets you query the system when facing an incident you had not anticipated, by correlating logs, metrics, traces and profiles. ### What is the main challenge of metrics? Cardinality: every label combination creates a distinct time series, and one poorly chosen label can multiply costs a hundredfold. The remedy is label discipline, aggregation at collection time and shifting detail to traces. ### Should traces be sampled head-based or tail-based? Head-based sampling decides at the entry point with a fixed percentage, simple but blind. Tail-based sampling decides in the Collector and keeps errored or slow traces: it is the recommended choice once traffic makes exhaustive tracing too expensive. ### Where does continuous profiling stand in OpenTelemetry? Profiles are OpenTelemetry's fourth signal: in public alpha since March 2026, with general availability targeted for the third quarter of 2026. They capture CPU and memory consumption at function level through eBPF agents.