Data observability without incident response is a log nobody reads

The ERP system I worked on as a backend software developer ran three layers of support. The first handled configuration questions and common errors; the second investigated whether an issue was a software defect; the third was engineering, brought in when the investigation needed code-level analysis or a fix. Every incident had an owner, a classification, and a resolution path. When I moved into the data engineering domain, none of that existed — even as the data products had become more consequential to the business than the ERP reports ever were.

The data platform had monitoring. Alerts fired when data pipelines failed. They fired into a Slack channel that engineers checked when they had time. Nobody was on call. Nobody was accountable for resolution within a defined window. And when a data quality issue surfaced, it usually arrived not as an alert but as a question from someone in finance — by which point the data had been wrong for days, not hours.

Data observability tooling is a detection mechanism. What happens after — how fast it gets resolved, what happens to the data that was incorrect, and whether the same failure produces any learning — is determined by process, not tooling. Most data teams have invested in the first and not the second.

SLAs

Not all data failures carry the same cost. A stale dashboard in a weekly review is a different incident from a corrupted feature store feeding a production model. Without a severity classification, every alert carries the same urgency — which means, in practice, that none of them do.

Defining SLAs starts with a conversation between the data team and the business: which data products are decision-critical, which data pipelines feed them, and what "available" and "accurate" mean for each. Data lineage analysis maps the dependencies. The output is a first version of severity tiers — low, medium, high — with SLAs for time to detect, time to resolve, and time to communicate. It does not need to be perfect on the first attempt. It needs to be written down.

Runbooks

The first time a data incident is investigated, the process is improvised. Symptoms are ambiguous, the investigation path is non-obvious, and the resolution takes longer than it should. Without a runbook, the second occurrence of the same failure starts from the same place.

A runbook does not need to be comprehensive. It needs to capture three things: where to look first, what to rule out, and who to escalate to if the investigation stalls. That is enough to reduce resolution time on the second occurrence significantly — and enough to give an AI agent a starting point when the same failure pattern surfaces outside business hours.

Backfill policy

When a bug is found in a data pipeline, two questions arise: is the pipeline fixed, and what happens to the data that was wrong? Most data teams answer the first question formally and the second one informally. Historical data gets corrected sometimes and left inconsistent other times, depending on who noticed and how much effort the correction required. There is no policy — only precedent, which varies by engineer and by incident.

A backfill policy does not need to be complex. It needs to answer: under what conditions a backfill is triggered, who authorises it, and which downstream consumers need to be notified. Without those answers, the same decision gets made differently each time, and the data platform accumulates inconsistencies that are difficult to trace later.

Postmortem

The question a postmortem forces is not "why did the data pipeline fail?" It is: how long was the data wrong before anyone noticed, and what decisions were made in that window?

A data pipeline that produced incorrect data for six hours before a monitoring alert fired is a different kind of failure from one that was wrong for three days before a question from the head of finance surfaced it. The tooling might classify both as incidents. The postmortem distinguishes them — and creates a record of which business decisions were potentially affected, not just which technical components failed. Without a postmortem process, only the second type of failure gets investigated, and only informally.

A useful data postmortem asks four things: when was the incorrect data first introduced; when was it detected; which data products drew from it in that window; and which business decisions referenced those products. That record serves two purposes — it creates accountability for the detection gap, and it gives the business a way to assess whether any decision made during the window needs to be revisited. Most software postmortems stop at the first question. A data postmortem cannot.

The leadership mandate

These are not processes a data engineer can establish alone. SLA definitions require the business to say which data products are critical. Backfill decisions require someone with authority over downstream consumers. Postmortems require leadership to treat a data incident as a process failure, not just a technical one — which means it has to matter to someone above the data team.

Data observability tooling identifies the failure. The incident response process determines what the organisation does with that information. Without the second, the first is a detection mechanism with no one listening.

The path forward

The first on-call rotation I set up for a data team was an uncomfortable conversation — not technically, but organisationally. It required agreeing on what "urgent" meant, who was responsible at 2am, and what a resolved incident actually looked like. The monitoring had been in place for months. That conversation had not happened yet. It is usually the conversation that needs to happen first.