What a data agent is, and what makes one good

My data-readiness essay argued the point I want to carry through all of this: the ceiling on a data agent is not the model, it is data readiness, whether the meaning of your columns and metrics is captured well enough for anyone, human or agent, to trust. Rent a better model next quarter and that ceiling does not move. It is the thread this whole post keeps returning to. The build post is the constructive sequel, one opinionated stack, assembled and measured in public. This is the piece in between, the part I skipped the first time and should not have. What a data agent even is, why it is worth building, what separates a good one from a demo, and how the answer changes depending on the company asking. No stack, no vendor names yet. Just the concept, stated plainly, so the build has something to be judged against.

What a data agent is, and how it differs from a standard agent

A data agent answers questions about a business by querying its warehouse, through a governed layer of definitions, and hands back a number a human can trust. That is the whole job. It is not a chatbot that happens to know SQL, and it is not a coding agent pointed at a database.

The difference from a standard agent is not the model or the harness, which are the same. It is what "done" means. A coding agent succeeds when the task is complete and the tests pass, and a wrong turn along the way is cheap, because you can read the diff. A data agent succeeds on one axis only: is the number right. A wrong number is worse than no number, because it looks exactly like a right one, and nobody double-checks the chart that confirms what they already believed.

Three things follow from that, and they shape everything downstream.

• The input is not open-ended. A coding agent works over a whole repository and the open web. A data agent works over one private, structured asset: the warehouse and the definitions on top of it. Its world is small and specific, which is a gift, because you can enumerate what correct means.
• The bottleneck is context, not reasoning. The model can write the SQL. What it cannot do unaided is know that revenue means bookings net of refunds in your company but gross in the one it read about on the internet. That gap is the whole ballgame, and it is why the good version of this is mostly a data problem wearing an AI hat.
• Verification has to be built, not assumed. A coding agent gets a free oracle: the test suite already exists. A data agent has no oracle unless you write one. Golden questions with known answers are not a nice-to-have; they are the only thing standing between "it answered" and "it was right."

The loop is the same shape every agent uses: gather context, act, verify, repeat. The data agent just makes the first and last steps the hard part. The context is business meaning in files, and the verify is an exam you had to write yourself.

And here is the part that decides everything else: a data agent inherits the readiness of the data beneath it. Point the best model in the world at a warehouse whose meanings live in three analysts' heads, and you get confident wrong answers faster. Point a mediocre one at a foundation where every metric is defined, governed, and documented, and it can already be useful. The agent is a thin layer of reasoning on top of however ready your data is. The readiness work is not a prerequisite you clear once and move past; it is the permanent upper bound on the answer, and no model release raises it for you.

Put simply: A data agent answers questions over your warehouse through governed definitions. It differs from a normal agent in one way that changes everything: it is judged on whether the number is right, a wrong number is invisible, and nothing verifies it unless you build the thing that does. And its ceiling is set before the model ever runs, by how ready the data beneath it is.

Why build one, and how to prove it is worth it

The case for a data agent is the same case every self-serve tool makes, and it is real: the person with the question stops waiting on the person who can write the query. The data team stops being a help desk, and the exec stops guessing between meetings. When it works, the answer arrives at the speed of the question instead of the speed of the next sprint.

The trouble is that "when it works" is doing enormous load-bearing work in that sentence, and the field is full of demos that never survived contact with a real warehouse. So the honest version of why is not a promise, it is a measurement. There is a public leaderboard for exactly this, and the numbers are sobering and clarifying at once. On Spider 2.0, the benchmark built for this, the lighter track tells the whole story in two numbers: a near-stock agent with no governed context posts 61.2%, while the leader, a context product, reaches 74%. The harder track, full repository-level tasks over a working warehouse, still tops out around 66% for the entire field. Put any of those next to a saturated older benchmark where models score in the low nineties, and the argument is a few numbers wide: on toy questions the model is done; on your questions it is not, and what moves the number is context, not a bigger model.

That is why "why build one" and "prove it with benchmarks" are the same section. The value is not that an agent can talk to your data, because a raw model can already do that, badly. The value is the delta: how much a governed, well-fed data agent beats the raw baseline on questions with known answers. And that delta is mostly a readiness delta. The governed agent wins because someone did the work of making the data mean something; strip that work away and the two agents converge at mediocre. Read honestly, the leaderboard is a readiness meter. If that delta is large, you have a reason to build. If it is not, you have saved yourself a platform. Either way you leave with a number, which is more than most of this category ships.

Put simply: Build a data agent so the question stops waiting in the queue. But do not take that on faith. A raw model already scores about 61% on real warehouse questions and stalls there; the reason to build is the measurable gap between that baseline and a governed agent, and you should insist on seeing it before you commit.

What makes a good one: the properties, not the product

Every serious system in this space, from the cloud warehouses to the open-source leaders, disagrees about product shape and agrees about the skeleton. Strip the branding and a good data agent has the same handful of properties. This is the bar; the build post is one way to clear it.

• An enforced semantic contract. The agent can only name things that have been modeled. Undeclared columns and legacy tables are invisible to it, so the most common failure, confidently querying the wrong field, becomes structurally impossible instead of something you catch after the fact.
• Validate before execute. The generated SQL is compiled and checked against the real schema before it ever runs, so an error is a rejected draft, not a wrong answer already sitting on someone's screen.
• Least privilege by default. Read-only, with row and timeout caps, and no tool anywhere in the query path that can send data outward. A warehouse agent holds private data by definition, and the boring credential discipline is the difference between a tool and an incident.
• Grounded, traceable answers. Every number carries its receipt: the query that produced it and the definitions it leaned on. An answer you cannot trace is a rumor with a chart, and the trace is also what lets a human review the agent's reasoning when a figure looks off.
• Correctness that is tested, not vibed. Golden questions with expected results, run like a test suite. This is the property that turns "seems right" into "passed," and it is the one most demos skip, because it is the only one that can also tell you that you failed.
• Context that improves under a gate. The business drifts, so the agent decays unless its context can be updated, and updates are dangerous unless something checks them. The mature version treats a context change like a code change: proposed with evidence, gated by the eval suite, merged by a human.

Notice what is not on that list: which model, which framework, which vendor. A good data agent is defined by these properties in whatever combination you assemble. Miss one, and you have a convincing demo with a soft spot exactly where trust is supposed to live.

And notice what the whole list stands on. Every property assumes a foundation it cannot supply. A semantic contract can only enforce meanings that already exist, grounded answers can only trace to definitions someone wrote, and tested correctness needs a known-right answer to test against. The six properties are how you spend data readiness, not a way around needing it.

Put simply: A good data agent is a set of properties, not a purchase: a semantic contract it cannot violate, SQL checked before it runs, least-privilege access, answers that carry their receipts, correctness under a real test suite, and context that improves through a gate. The vendors converged on this skeleton. Copy the skeleton, not the logo.

How the requirements change by org

The properties are universal; how much each one matters is not. The same six-item bar produces very different builds depending on who is asking, and pretending otherwise is how reference architectures turn into shelfware. A few dimensions move the weights.

• Data maturity, which dominates the rest. The first question is whether a trustworthy semantic layer even exists. If it does not, the data agent is not step one, it is step three: the readiness work comes first, and what that work actually looks like is seven disciplines of its own. A team that skips it is building on sand. A mature org with governed metrics can point an agent at them next week; an immature one has to earn the right. This is less one dimension among five than the one that decides whether the other four matter, because it sets the ceiling every later choice operates under.
• Risk and governance. A startup answering internal questions can run loose. A bank or a hospital cannot: access control, audit trails, and hard limits on what can leave the building stop being good hygiene and become the license to operate. The least-privilege property is the same everywhere; the amount of it is not.
• Breadth and scale. One warehouse and twenty metrics is a different animal from a dozen systems of record and thousands of definitions across teams. The more meanings there are, the more the semantic contract and the context management carry the whole system, and the earlier they have to be taken seriously.
• Who it answers, and how badly wrong hurts. Internal self-serve for analysts tolerates a slower, chattier agent. A customer-facing or embedded agent raises the correctness and latency bar sharply, because the wrong number now ships to someone outside the building. Regulated reporting raises it again, because now the wrong number is a filing.
• Who keeps it running. A one-person kit and a platform team want different tooling for the same properties. The solo build leans on headless evals in CI; the platform on a governed service with a UI and on-call. The bar is identical; the machine that holds it up is sized to the org.

The through-line is that none of these dimensions changes what good looks like. They change how much of each property you buy, and in what order. Which is the honest answer to "what should we build": tell me your data maturity, your risk, your breadth, your audience, and who keeps it alive, and the same skeleton resolves into a very different first ninety days.

Put simply: The properties do not change from company to company; their weights do. Data maturity decides whether you can start at all, risk decides how much governance, breadth decides how hard the semantic layer has to work, audience decides the correctness bar, and team size decides the tooling. Same skeleton, different build.

That is the concept, kept deliberately vendor-free. The build post takes this bar and clears it with one specific stack, assembled from parts that all went free this year, measured against the leaderboard above, in public, whichever way the number lands.