SRE & observability
Seeing more is not the goal. What an operations team needs is to be woken for what customers actually feel, and to be left alone the rest of the time. That starts with an objective, not with a tool.
An alert nobody acts on teaches everyone to ignore alerts.
Sixty to eighty per cent of alerts are false positives, and an on-call engineer takes a median of forty-two pages a week. The result is not a tired team, it is a team that stops reading, and the one signal that mattered goes past unnoticed among the sixty-three that did not.
So we start from the other end. An objective negotiated with the business, an error budget that makes the trade-off explicit, alerting on what the customer feels rather than on what the machine reports, and a monthly review that removes any alert nobody acted on.
What we do
Four workstreams so an operations team is woken for the right things.
Service levels and error budget
An objective negotiated with the business, measured from the customer's side, with an error budget that says how many failures are acceptable. That is a business choice, not a technical one.
measured on the customer journey, not on the server
- An SLO per critical journey
- An error budget with its arbitration rule
- Delivery arbitrated against the budget left
End-to-end observability
Traces propagated from the browser to the last system, metrics per step and sampled logs, so an incident reads on one thread rather than across four consoles.
with a cardinality budget, which is where the bill hides
- Traces propagated across every hop
- Metrics per step of the journey
- A cardinality budget, set before the bill
Alerting that earns the wake-up
Alerts on symptoms rather than causes, graded by how fast the error budget burns. Anything that does not call for an action becomes a ticket, or disappears.
no alert is created without another being removed
- Two levels: wake up, or ticket
- Monthly review, actionless alerts removed
- No threshold without a documented action
AIOps and root cause analysis
Signals correlated automatically instead of being added up, so investigation starts from a hypothesis rather than from a dashboard. What is gained is not detection, it is the time before the first order.
decision time is measured just like detection time
- Automatic correlation of traces, logs and metrics
- Root cause analysis presented, then confirmed by a human
- Blameless post-mortems, with an owner
What you get
One project runs through the four deliverables below: putting a payment journey under SLO. Each step states what is actually handed over, in the order it is handed over.
The service level and its error budget
Availability and latency measured from the customer's side, over a rolling window, with a budget of forty-three minutes a month and the rule that says what happens when three quarters of it are gone.
parcours-paiement
- 01-slo.yaml
- 02-instrumentation.yaml
- 03-alertes.yaml
- 04-incident.yaml
The instrumentation of the journey
Traces propagated at every hop, metrics per step, sampled logs, and a cardinality budget. One label per customer id multiplies the bill by a thousand and helps nobody diagnose anything.
parcours-paiement
- 01-slo.yaml
- 02-instrumentation.yaml
- 03-alertes.yaml
- 04-incident.yaml
Alerting that deserves a wake-up
Two levels only, and a removal criterion written in: any alert that has fired five times without an action is removed or requalified. Creating one requires removing another.
An incident, replayed
Forty-one minutes of partial outage against a monthly budget of forty-three. What the replay finds is not a missing signal, and that changes the whole conclusion.
How we deliver
Define
depending on the number of critical journeys and systems crossed
- SLOs negotiated with the business
- Error budget and arbitration rule
- Map of the critical journeys
Instrument
depending on trace depth and the systems to connect
- Traces propagated end to end
- Alerts on symptoms, not on causes
- A first incident handled on that basis
Cut the noise
depending on the existing alert volume and the number of on-call teams
- Actionless alerts removed or requalified
- Automatic correlation of signals
- On-call sized on useful pages
Hold
service commitment defined with you
- Error budget tracked and arbitrated
- Blameless post-mortems
- Monthly alert review
What on-call actually looks like
A production you hold, not one you watch
A frame holds because each portal was sized for its load, not because someone is watching it. That is what operations should produce: a system where attention goes where it is useful, and nowhere else.
Objectives before tools. One SLO per critical journey and an error budget that makes the delivery trade-off explicit, rather than a dashboard nobody arbitrates on.
Service levels that hold
Scaling to 200,000 active users without losing the service level
99.99% SLO reached · P1 MTTR divided by three
An online bank of the Crédit Agricole group scaling towards 200,000 active users, where a slowdown at a peak is not a performance topic but customers unable to reach their accounts.
A cross-functional performance and observability team: peaks anticipated rather than absorbed, detection time divided by five, and a service level that holds as the user base grows fourfold.
Editorial performance that stops eroding release after release
−45% load time · 5 of 6 Core Web Vitals green
An editorial site whose performance eroded release after release, on a title where reading comfort is part of the product and where nobody could say which change had cost what.
Measurement wired into the delivery chain rather than run as an audit: what degrades shows up on the next release, and the maintenance keeps the measurement running after the rebuild.
Insights & Perspectives
Closing the SRE gap: towards autonomous observability
How an AI agent correlates logs, metrics and traces to automate root cause analysis, and what it takes for that to cut MTTR rather than add another dashboard.
Kernel-level deep causality analysis with MCP
Going below the application to read what the kernel says, and turning that into an explanation an on-call engineer can act on at three in the morning.
Chaos engineering: practices for resilient systems
Breaking things on purpose, in production, under control. What the discipline actually requires before the first experiment is worth running.
Be woken for the right things
Service levels negotiated with the business, observability that reads on one thread, and an on-call rota sized on useful pages.
Frequently asked questions
With objectives. A tool shows what you point it at, and without an agreed service level every incident is arbitrated in the moment. An SLO negotiated with the business is what turns a dashboard into a decision.
To make the trade-off explicit. Aiming for zero incidents means either shipping nothing or shipping and absorbing the consequences. A budget says how many failures are acceptable, and stops non-critical delivery when three quarters of it are consumed.
Because a cause that harms no customer does not deserve a wake-up, and a symptom always does. CPU usage on its own says nothing about what a customer experiences, which is why it is one of the alerts we remove first.
By removing what nobody acted on. An alert that has fired five times with no action is removed or turned into a ticket. The risk is not missing a signal, it is that the useful one goes unread inside a stream nobody follows any more.
No. It correlates signals and proposes a root cause, which shortens the time before the first order. The decision stays human, because acting on a critical system on a machine's suggestion is exactly the kind of thing a post-mortem later regrets.
Mostly cardinality. One label per customer identifier multiplies volume by a thousand and helps nobody diagnose anything. The budget is set at design time, per label, which is far cheaper than discovering it on the invoice.
Defining them takes 2 to 6 weeks depending on the number of critical journeys and systems crossed, and produces the SLOs, the error budget and the map of critical journeys. Instrumentation and a first incident handled on that basis follow in 4 to 10 weeks.