Your data team is one role short

I have walked into more than a few data teams that were quietly drowning. Five, six, seven engineers. Solid technical talent. Data pipelines running. Dashboards delivered.

And yet the AI roadmap was stalled, the data models were a mess, and the engineers were frustrated — not because they were bad at their jobs, but because they had been asked to make decisions that were never theirs to make.

The instinct, when a data platform starts to slow down, is to hire more engineers. More hands, more throughput. It is a reasonable assumption. It is also usually wrong.

The problem is rarely capacity. It is clarity.

What a data engineer actually does

Data engineers are specialists in flow. They build and maintain the data pipelines that move data from source to destination reliably, efficiently, and at scale. They are exceptionally good at this.

What they are not equipped to do — and should not be expected to do — is answer questions like:

• Which datasets are mission-critical for the business cases consuming this data?
• Which tables should refresh daily and which can run quarterly?
• How should storage be structured to optimise for the queries that actually matter?
• Which roles need access to personally identifiable data, and which absolutely should not?

These are not engineering questions. They are architectural and business questions. And when no one else answers them, engineers answer them anyway — as best they can, with the context they have.

What a data architect brings that engineers cannot

A data architect, working closely with subject matter experts from the business, creates the data model that gives engineers the answers they need.

They map data domains to business processes. They define relationships between entities — which are core, which are derived, which are disposable. They establish the governance rules that determine who sees what and why.

Crucially, they translate business priorities into technical direction:

• This dataset drives the revenue forecast. It must be accurate and fresh.
• This reference table changes quarterly. Daily refreshes are waste.
• This dimension contains personally identifiable data. Access must be restricted at the storage layer, not the application layer.

None of this is intuitive to someone hired to build data pipelines. It requires a different lens — one trained on the business model, not just the data model.

The two costs of a missing data architect

The first cost is technical. Without architectural guidance, data engineers optimise for what they can measure: data pipeline speed, job completion, query time. Over time, the data platform accumulates decisions that made sense locally but compound into a system that is slow to query, expensive to run, and brittle under load. By the time anyone notices, unpicking it requires months of rework.

The second cost is human. I have seen teams of five or more engineers operating for years without meaningful access to architects or subject matter experts, making schema decisions and business judgement calls they were never trained for. The frustration is real. Talented engineers, doing technically competent work, with no clear sense of whether any of it matters. The work becomes mechanical. Motivation erodes quietly. With no solution visible on the horizon, I have seen a few quietly move to other teams.

The contrast is stark. Teams with easy access to data architects and subject matter experts can focus entirely on what they do best — because someone else has already done the hard work of telling them what to build and why it matters.

The leadership mandate

As a CTO, the temptation is to treat this as a headcount question. It is not. It is a collaboration design question.

Data architects are often already present in the organisation — siloed, underutilised, or disconnected from the engineering teams that need their input most. Data engineers cannot reach them. Decisions cannot wait. Work proceeds on assumptions.

Your role is to create the conditions where architectural thinking and engineering execution happen in the same space, on the same timelines. Not as a handoff, but as a conversation.

Until that happens, your data engineers will keep answering questions they were never hired to answer — and your data platform will keep accumulating the debt of every wrong guess.

The path forward

If your data team is technically strong but your AI roadmap is moving slower than it should, the gap is probably not in the data pipelines.

It is in the space between what the business needs and what the data engineers know.

As a Fractional CTO with 15 years of experience building data foundations across healthcare, energy, and telecommunications, I help teams close that gap — by getting the right roles collaborating and the right decisions made in the right order.

Let’s connect to build your data moat. 🛡️