Data quality is not an engineering problem

A dashboard metric shifts. A stakeholder asks why a KPI has changed when nothing in the business has changed. The investigation leads back to a new data source added the previous week — one that writes zero where the existing source writes null for the same field, because of how the application was configured.

Neither data source is wrong. The aggregation treats null and zero differently, which is why the total moved. But the question that surfaces during the investigation is not which configuration is correct. It is why nobody wrote down what null was supposed to mean in that field before the second data source was ever connected.

Most data quality problems are not engineering failures. They are quality decisions nobody made — thresholds that were never set, definitions that were never agreed, policies that defaulted to whatever the framework happened to do.

This is a harder problem than a broken data pipeline, because a broken data pipeline produces an error. A missing quality decision produces outputs that look plausible — and often are, until they are not.

The semantic default

Null and zero are not interchangeable, but most data platforms treat them as if they are. In a sales field, a null might mean "no sale was recorded" in one table and "zero sales occurred" in another, depending on how each application wrote its records. Downstream aggregations silently produce different totals depending on which table they read from.

The null rate in that field compounds the problem. A 15% null rate in a key metric is common enough to be treated as normal — it has always been that way, so it becomes the baseline. Nobody decided that 15% was acceptable. Nobody decided what acceptable meant. A model that reads from that field silently excludes those rows, and the output is wrong in a way that is difficult to detect because the input looked entirely normal.

What null means in a given field, and what null rate is acceptable, are two questions that should have written answers before the first aggregation runs. They rarely do.

The duplicate policy

A count reconciliation between two systems shows a 2–3% discrepancy. The assumption is missing data — records present in one system but absent from the other. The investigation reveals the opposite: duplicate records in the data platform, from a data source that retried failed requests on timeout and produced the same event twice.

The data pipeline had no deduplication because the data source was not supposed to emit duplicates. The question of whether to deduplicate, and on which key, had never been asked explicitly. The framework default was to ingest what arrived.

A 2% duplicate rate might be expected behaviour from a data source with retry logic. It might be a critical defect in a financial reconciliation. The difference between those two situations is a written policy, not a technical one.

The completeness threshold

What percentage of expected records arriving in a load is acceptable? Without a stated threshold, engineers make the call individually. One treats a 97% load as a pass; another raises an alert. Neither is wrong — they have different intuitions about what the downstream use case requires, and neither intuition has been written down.

Freshness is the same category of problem. A record that arrives on time but reflects a state from six hours ago is not a data pipeline failure. It is a question about whether the data pipeline's behaviour matches what the business actually needs the data for — and those two things are often never compared.

The leadership mandate

These are not decisions a data engineer can make alone. What null means in a sales field is a product decision. Whether a 2% duplicate rate is acceptable depends on the data source system's behaviour and the downstream use case — knowledge that sits across teams, not within the data engineering function. What constitutes a complete load depends on what the business does with the data.

Engineering teams make these calls by default because the data pipeline has to run and the field has to have a value. The defaults accumulate, and the data platform ends up calibrated to inherited behaviour rather than stated requirements. When a quality issue surfaces, the investigation uncovers not a mistake but an assumption — one that was made implicitly, by a framework configuration nobody reviewed, and then never revisited.

The path forward

Data quality tooling — monitoring thresholds, anomaly detection, freshness alerts — is easier to configure when these decisions already exist. Without them, the tooling detects deviations from an inherited default rather than from a defined standard. That is a meaningful difference: one catches drift from a baseline nobody chose, the other enforces a commitment the business has actually made. Most data platforms have the tooling. The written decisions are what is usually missing.