The data platform decision that comes before build or buy

I worked with a team running a proof of concept on a cloud data platform vendor solution that cost roughly one million dollars a year. A year in, the assessment was clear: the data integrations, data models, and business insights the team had built were genuinely valuable. What was not clear — what had never been asked — was why the team had chosen that particular vendor solution to build on. There had been no architectural assessment. There had been no comparison against open-source alternatives. The contract had been signed before anyone knew how much data they would be processing or what the data platform needed to do.

The cost of that gap was not just financial. It was a year of building on a foundation that nobody had deliberately chosen.

The build vs buy debate in data infrastructure is usually framed as a vendor comparison: which has the best feature set, the best integrations, the best support contract. That is the wrong starting point. The right starting point is whether any third-party data platform component is needed at the current stage of the product at all.

The default

In the early stages of a data-driven product, the correct default is to avoid committing to any significant third-party data platform component vendor until the value of the data itself has been demonstrated.

The reason is straightforward: early-stage data products are discovery exercises. The team is building to find out what the data can do, not to optimise a known data pipeline. In that context, the open-source ecosystem combined with public cloud usage-based services is almost always sufficient. Apache Airflow for orchestration, Polars/Pandas for data transformation, object storage for persistence — this combination can carry a data platform further than most teams expect, at a fraction of the cost of an enterprise contract, and without the lock-in.

This is not a permanent position. It is a starting point that keeps options open while the team learns what the data platform actually needs to do.

The modular structure

The instinct to avoid third-party lock-in does not mean building something rigid. The right approach is to modularise the data platform from the start — treating each functional component as an independently replaceable unit.

The components worth treating as separate modules from the beginning are: data storage, data ingestion and transformation, orchestration, data cataloguing, data lineage, and data governance. Each of these can start as an open-source implementation. As the platform matures and the team has operational evidence of where the open-source solution is reaching its limits, individual modules can be swapped for a managed cloud offering or a third-party product — without touching the rest of the data platform.

A team that makes each of those decisions based on evidence is in a very different position from a team that signed a data platform component vendor contract before the product existed.

The buy decision

When the evidence does justify buying, the decision should be grounded in measurable criteria rather than market enthusiasm.

The metrics worth evaluating vary by module — cost at current and projected data volumes, scalability under the team's specific access patterns, user extensibility for the integration requirements already visible on the roadmap. The point is to agree on criteria before comparing solutions, not to reverse-engineer a justification for a vendor solution the team is already excited about.

The hype bias in data infrastructure is real. When every conference talk in a domain features the same tool, it becomes genuinely difficult to separate signal from noise. I have seen teams push for Databricks on data volumes where a managed cloud warehouse would have served them for another eighteen months at a fraction of the cost — not because the engineers were wrong about Databricks being a strong product, but because the pressure of industry momentum made the timing feel more urgent than the data warranted. Bringing business and engineering together to evaluate the actual metrics, rather than the assumed trajectory, is how that conversation gets resolved productively.

The leadership mandate

The data platform component build vs buy decision is not primarily a technical decision. It is a timing decision.

Buying the wrong data platform components is expensive. Buying the right ones at the wrong stage is nearly as expensive — because it anchors the architecture before the team knows what the architecture needs to do, and it creates switching costs at exactly the moment when the product is still discovering what it is.

The teams that get this right share a common approach: they build on the minimum viable data foundation, modularise deliberately, and let operational evidence — not vendor demos or conference momentum — drive the buy decision. When they do buy, they know precisely which module they are replacing, what it needs to do, and how they will measure whether it is working.

The path forward

If your team is approaching a data platform component build vs buy decision, the most useful question is not which vendor to choose. It is whether this is the right moment to choose one at all — and if it is, which specific component the decision is actually about.

As a Fractional CTO with 15 years of experience building data foundations across healthcare, energy, and telecommunications, I help engineering teams make data platform decisions that are grounded in where the product is, not where the vendor roadmap says it should be.

Let's connect to build your data moat. 🛡️