Choosing your data pipeline tools for a lean team

The non-negotiable data stack makes one argument twice: the six layers that produce ready data are a fixed floor, and the tools that satisfy each layer are an interchangeable preference. So it deliberately refuses to answer the question everyone actually asks, which is which tool to buy. This is that answer, and it is scoped on purpose to a lean data team: an early-stage, growth, or lower-middle-market company with real revenue, a data team of one to a handful, a budget that is finite but not zero, and no platform team standing by to babysit infrastructure. The stage label matters less than those constraints.

That scope matters because at this size the thing that decides your tool is rarely the feature matrix. It is your team's size, your tolerance for operating software, and whether a tool's cost is justified by a pain you actually have rather than one you imagine having at ten times your revenue. What follows is neutral: for each of the six floor layers, the real options, the tradeoff that decides, and when each one fits a company your size. There is no universal winner. There is a winner for your constraints.

The warehouse: where the numbers live

The options that matter are BigQuery, Snowflake, Databricks, and DuckDB by way of MotherDuck. The tradeoff that decides is the operating and pricing model, not the SQL.

BigQuery is serverless and priced by the data each query scans, with no clusters to size and nothing to keep running. It fits a lean team on Google Cloud with spiky, ad-hoc analytics, and its risk is a runaway scan bill you control with partitioning, clustering, and a cost cap. Snowflake separates storage from compute and bills per second on warehouses you size yourself, which suits steady concurrency, strong workload isolation, or a multi-cloud footprint, at the cost of more knobs and the easy mistake of a warehouse left running idle. Databricks is a lakehouse built for Spark, notebooks, and machine learning, and for most analytics at your size it is answering a question you do not have yet; reach for it when real ML, streaming, or large unstructured data is the actual job. DuckDB, run locally or through MotherDuck, is the cheapest thing that works, and it genuinely works when your data is tens of gigabytes, single-team, and free of heavy concurrency.

Put simply: Choose the warehouse by your data size, your cloud, and whether you truly do ML. DuckDB when the data is small, BigQuery when it is spiky and on GCP, Snowflake when concurrency and isolation matter, Databricks only when the lakehouse is the job and not the aspiration.

Transformation, in version control

Here the options are dbt Core, dbt Cloud, SQLMesh, and Dataform, and the tradeoff is a triangle: the size of the hiring pool and ecosystem, the raw technical capability, and who is on the hook to run the operations.

dbt Core is free, open source, and the de facto standard, which means the largest hiring pool and the widest integration surface; the catch is that you run your own scheduling and continuous integration. dbt Cloud wraps that in a managed IDE, scheduler, and CI for a per-seat fee, and it earns its cost precisely when you have nobody to run infrastructure and would rather buy the operations away. SQLMesh brings genuinely better engineering, with virtual data environments, column-level lineage, and rebuilds that only recompute what changed, and it is the right bet when compute cost and environment management are real pains and you have an engineer who will own a smaller ecosystem. Dataform is free and native to BigQuery, which suits an all-Google-Cloud team that wants zero extra tooling, paid for with a much smaller community.

Put simply: The transformation choice is about operations and hiring more than dialect. dbt Core is the safe default, dbt Cloud buys away the ops, SQLMesh wins on cost and environments if you can absorb the smaller ecosystem, and Dataform fits a pure-GCP team that wants nothing extra.

One semantic layer

The options are MetricFlow, Cube, and whatever semantic model is native to your business-intelligence tool. The tradeoff is headless and open versus embedded and cached versus already sitting inside the tool you use.

MetricFlow is now Apache-licensed and native to dbt, so a metric is defined once and can be queried from anything, which fits a team already on dbt that wants governed metrics without adopting a new vendor. Cube is API-first with a caching layer, and it earns its place when you are embedding metrics into a product or a customer-facing app. If you have standardized on a single BI tool, its own semantic model, LookML in Looker or a Power BI or Tableau model, may simply be enough; the price you pay is lock-in and metrics that do not travel outside that tool. The trap here is buying a standalone semantic layer you do not need because a single BI tool already governs every metric you have.

Put simply: On dbt, MetricFlow. Embedding metrics in a product, Cube. Standardized on one BI tool with no headless need, its native model is often enough, as long as you accept the lock-in with your eyes open.

Tests and contracts, as the merge gate

The options are dbt tests, Elementary, and the standalone data-quality tools Great Expectations and Soda. The tradeoff is whether your quality checks live inside the transformation graph or beside it.

dbt tests are built in, free, and run in the same graph as your models, which is enough to gate a merge for most teams your size. Elementary layers dbt-native anomaly detection and freshness and volume monitoring on top of those tests, and it is light enough to add without standing up a platform. Great Expectations and Soda bring richer expectation libraries and reach data sources that dbt never sees, at the cost of more to operate; they are worth it when you have quality needs outside the dbt graph or a compliance bar to clear. For a lean team, the honest default is dbt tests to gate the merge and Elementary to watch for drift, with the heavier tools held in reserve.

Put simply: Gate the merge with dbt tests and watch for drift with Elementary; that covers most teams your size. Reach for Great Expectations or Soda only when quality checks have to live outside dbt.

Continuous integration and isolated environments

The options are GitHub Actions, GitLab CI, and dbt Cloud CI, and this is the layer where the tool matters least. The tradeoff is simply whether you already have a runner or you bought one with dbt Cloud.

GitHub Actions or GitLab CI is free at the tier you will use and already attached to wherever your code lives; you run a dbt build on every pull request against a staging dataset, and you use slim CI that only rebuilds changed models to keep the compute bill down. dbt Cloud CI is the zero-setup managed version, and the natural pick if you have already paid for dbt Cloud. The shape of the environments is identical no matter which you choose: separate dev, staging, and production datasets or schemas, nothing hand-edited in production, and promotion only by merge. The tool is almost incidental; the discipline it enforces is the entire point.

Put simply: Do not buy a CI tool. Use the one attached to where your code already lives, run dbt build on every pull request against staging, and promote to production only by merge.

Lineage

The options run from dbt docs and exposures, through the open standard OpenLineage, to the open-source catalog DataHub, up to enterprise catalogs like Atlan and Collibra. The tradeoff is free and dbt-scoped versus an open cross-tool standard versus a full catalog you have to run.

dbt docs and exposures are free, generated from the code you already write, and give you column-level lineage within the dbt graph, which answers "what breaks if I change this" for most teams your size. OpenLineage is an open standard that captures lineage across tools dbt cannot see, and you add it when your pipeline genuinely spans systems beyond dbt. DataHub is an open-source catalog with lineage, search, and discovery, heavier to operate and worth it once dbt-only lineage stops answering your questions. The enterprise catalogs are real capability at an enterprise price and rollout, and for a team your size they are almost always premature: you are buying governance tooling for an organization you are not yet.

Put simply: dbt docs until it stops answering "what breaks if I change this," then OpenLineage or DataHub when your pipeline outgrows dbt's view. Skip the enterprise catalog until you are much bigger than you are now.

What it adds up to

The pattern is the same across all six layers, and it is the one that fits a lean team. Every layer has a free or managed default that already satisfies the discipline, a warehouse you do not operate, dbt, its tests, its docs, the CI attached to your repository, and a set of paid or heavier tools you add only when a specific pain actually appears: operations you cannot staff, machine learning you genuinely do, metrics you embed in a product, quality checks outside the graph, lineage dbt cannot see. The mistake is buying the enterprise tool for the pain you imagine having at ten times your size, and paying for it in dollars, setup, and a team's attention while the pain never arrives. Match the tool to the constraint you have now, keep the discipline the floor requires no matter which tool you pick, and upgrade a layer the week its pain becomes real, not the quarter you first read about the tool.

Put simply: Every layer has a cheap default that satisfies the discipline and a heavier tool you add only when a real pain shows up. Buy for the constraint you have, not the enterprise you imagine becoming, and the whole stack stays affordable and boring, which is exactly what a lean team needs it to be.