The non-negotiable data stack

There is a floor under data readiness, and it is made of software engineering, not data science.

I have argued that ready data is the ceiling on any AI system, that a practitioner has to own it, and I have shown the stack I would build a data agent on top of it. This post is the layer underneath all of that: the technical stack that actually produces ready data, and the short list of it that is non-negotiable.

The distinction that runs through the whole thing: some of the stack is a choice, and some of it is a floor. Below the floor you do not have a slower or cheaper version of ready data. You have data that is not ready. Here is the floor, drawn, with the optional gold-plating sitting on top of it where it belongs.

The floor, and the gold-plating

Six layers make data ready, and you do not get to skip any of them.

A warehouse separate from the production database, so there is one place the numbers live and analytics and AI never run against prod. A transformation layer in version control, so every change to a number is a diff someone reviewed. One semantic layer, so each metric is defined exactly once. Tests and contracts that gate the merge, so a broken definition cannot ship. Continuous integration with isolated environments, so nothing is hand-edited in production. And lineage, so the blast radius of any change is visible before you make it. That is the minimum viable readiness stack, and it is smaller and more boring than most teams expect.

You also pour it narrow before you pour it wide. The floor goes down for the tier-1 metrics and dimensions first, the handful the business actually runs on, not the whole warehouse modeled at once. Done that way the floor is a few weeks of work, not a quarter, which is exactly why treating it as optional is inexcusable: it is cheap, and skipping it is the expensive choice disguised as the fast one.

Everything else is gold-plating, and gold-plating is fine once the floor is poured. A fancy orchestration platform, a data catalog, observability and freshness monitoring, reverse ETL, a feature store, a mesh: all genuinely useful, all optional, and all worthless bolted onto a stack with no version control or no tests. The common mistake is buying the polish before pouring the floor, because the polish is what vendors sell and the floor is what you have to build.

Put simply: Six layers are the floor: a warehouse, version-controlled transforms, one semantic layer, tests as a merge gate, CI with environments, and lineage. Everything else is polish. Pour the floor before you buy the polish.

Below the floor, it is not ready

The reason these six are non-negotiable and not just recommended is that each one, skipped, does not degrade readiness a little. It moves you across a line, from data you can trust to data at rest that happens to be queryable.

Skip version control and a change becomes invisible: you cannot review it, revert it, or even name who moved the number. Skip the single semantic layer and revenue means three different things in three dashboards, so correctness stops being knowable. Let tests only warn instead of blocking the merge, and you have theater, not a gate. Hand-edit production and a quiet Tuesday becomes an outage nobody can trace. Each of these is not a smaller version of ready. It is unready with extra steps.

Put simply: The non-negotiables are non-negotiable because skipping one is binary, not gradual. It does not make the data less ready. It makes it not ready.

The floor is practices, not products

Here is the part that matters most, and the part that makes "which tool should we buy" the wrong first question.

The non-negotiables are engineering disciplines, and they are tool-agnostic. dbt or SQLMesh, MetricFlow or Cube, dbt tests or Great Expectations: pick whichever you like. What you do not get to pick is whether transforms are version-controlled, or whether tests gate the merge, or whether a metric is defined once. The discipline is fixed; the tool is a preference. That is why the right first question is not "which tool" but "which disciplines does this tool enforce for me, and which do I still have to enforce myself." A tool that lets you hand-edit production is not a shortcut. It is a way to skip the floor and not notice. Once you accept the floor, which tool to pick at each layer is its own question, and it turns on your constraints, not the feature matrix.

Put simply: The floor is a set of practices, not a shopping list. Any tool that enforces the discipline works, and no tool compensates for skipping it. Choose the tool second.

What it adds up to

The stack that makes data ready is cheap, mostly free, and almost entirely known. That is exactly why it is not where teams fail. They fail by treating the floor as optional and the polish as the point, then wondering why an AI system built on top keeps returning confident, wrong answers. Pour the six-layer floor, keep the disciplines it encodes, and put a practitioner in charge of holding it. Then, and only then, buy the polish. And once the floor is poured, measure whether people actually query the gold layer: the share that does is the fastest read on whether any of this is trusted.

Put simply: Data readiness has a floor of six boring, non-negotiable layers, and it is made of software-engineering discipline. Build the floor first, choose your tools second, and add the gold-plating last.