Streaming is not the starting point. It is the upgrade.

I worked with a data team that had built their data ingestion layer on Apache Kafka. Six months in, I asked how many of their data sources actually required real-time processing. We went through the list. Ninety per cent of them were refreshed at source twice a day. The streaming infrastructure was running continuously to ingest data that arrived on a schedule.

Nobody had asked what freshness the business needed before the architecture was decided.

The choice between batch and streaming is usually presented as a technical architecture decision. It is a product maturity decision. Streaming infrastructure — Kafka, Flink, real-time data pipelines — is expensive to operate, requires specialist knowledge to maintain, and introduces failure modes that batch processing does not. Reaching for it before the business has told you what freshness it actually needs is the same instinct that leads teams to ingest every available data source on day one. Both substitute future optionality for present clarity, and both create overhead that compounds.

The default

Batch-first is not a fallback. It is the correct starting point for a data platform that does not yet have evidence of real-time requirements.

At the early stages of a data platform, business users typically do not know what freshness they need. They know they want data. Whether that data should be four hours old or from yesterday is a question they will answer once they start using the system, not before. Building streaming infrastructure to serve requirements that have not been stated yet is building for a problem you are guessing at.

Batch processing built with flexibility in mind gets further than most teams expect. A data pipeline that runs on a schedule can run multiple times a day — every four hours, every hour, every thirty minutes — without any architectural change. That range covers the majority of business requirements that surface in the first year of a data platform and gives the team time to learn where real-time freshness genuinely changes a product outcome before committing to the infrastructure that delivers it.

The upgrade

The batch-first default does not mean streaming is never the right choice. It means the decision should be made with evidence.

Some data sources genuinely justify real-time data pipelines: high-frequency transaction events, user behaviour streams, operational feeds where a one-hour delay affects a business decision. When those requirements are present and validated, a hybrid approach is the right architecture — batch for the data sources that do not need real-time freshness, streaming for the ones that do. The hybrid also prevents the trap of streaming everything once the infrastructure exists.

For a data platform that has no immediate streaming requirement but expects to add one, the right step is not to build streaming now and use it later. It is to run a proof of concept: validate the streaming infrastructure against real production data sources, understand the operational overhead, and leave the integration point ready without coupling the production data platform to a capability it does not yet need.

The architecture

Whether the current requirement is batch or streaming, the data platform should be designed for that decision to change without a rebuild.

The practical approach is a configuration-driven data flow: a file that controls how data moves through the system, with an ingest_mode flag that can be set per data source. When a new real-time requirement emerges — and it will — the operations team flips the flag rather than refactoring the data pipeline. The architecture stays the same. The parameter changes.

This is not premature flexibility. It is the difference between a data platform that adapts to changing business requirements and one that locks in a decision made before the evidence was available.

The leadership mandate

The batch vs streaming choice is the kind of decision that happens without leadership visibility because it looks purely technical. It is not.

When a team defaults to streaming because it is what they saw at a conference, or what their previous employer used, or because it sounds more modern, the cost does not appear immediately. It appears six months later in operational overhead nobody budgeted for, in on-call alerts for streaming infrastructure serving data that arrives twice a day, in capacity that was supposed to go to the product roadmap but went to keeping Kafka running instead.

The leadership role is not to make the architecture decision. It is to make the question visible: what freshness does the business actually need, and what is the evidence? When leaders model that kind of deliberate questioning, it sets the standard for every infrastructure decision the team makes. When they do not, "it might be useful" becomes the default reasoning — and that mindset accumulates.

The path forward

Streaming infrastructure is not wrong. It is a deliberate upgrade with a specific business case. The teams that introduce it well start from batch, discover through use where real-time freshness actually changes a product outcome, and build the streaming capability when that case is clear. That evidence-first conversation is worth having before the Kafka cluster is provisioned. It is where I usually start.

Let's connect to build your data moat. 🛡️