If you work in data, you might have recently tried using generative AI to answer natural language questions from your analytics database. What you might have discovered in the process is that in order to get better accuracy in the answer, the solution is not always to upgrade to a better and more expensive AI model.
Instead, the difference and the biggest improvements in accuracy come from somewhere else: the context the model is given or the extra tools it can call to answer you. Practitioners have started calling this surrounding layer the “harness,” and the discipline of building it well “harness engineering.”12
I find this trend pivotal because it sits in a longer arc of organizational transformation. Let me explain. Earlier this year my book Data as a Product Driver (Apress, 2026) came out, describing three phases that data teams go through as companies grow: 1) establishing data foundations, 2) operating as a support function, and finally 3) data becoming a product driver.
Since publishing it, I have been trying to understand whether generative AI is pushing the most mature teams into a fourth phase. The more teams I look at and the more I read about it, the more convinced I am that harness engineering is a strategy that, done well, is at the center of such transition. I also believe that harness engineering is significantly more complex when automating data workflows than when automating standard day-to-day tasks or even software engineering workflows. Sadly, most data organizations don’t have the capabilities in place to make it work.
This is why in this article, I break down what harness engineering actually looks like for data teams. I cover:
The anatomy of a standard AI harness: The six core components every system needs.
The data complexity layer: Why working with data introduces unique hurdles like state, environments, data quality, and the need for metadata.
The data-specific harness: How to adapt standard AI components to solve these data-specific problems.
Let’s dive in!
The Anatomy of a Harness
At a high level, drawing on LangChain’s anatomy of an agent harness², I see six core components, illustrated in the figure below and explained next.
1. Context Injection
By context we refer to the prompts, memory, behavior instructions, persona definitions, and basically any kind of information an AI agent might need to perform the task.
It includes files like AGENTS.md or CLAUDE.md that encode team conventions, architecture decisions, table metadata, semantic information, dbt style guides, SQL naming conventions, data lineage information, and any data modeling patterns you use internally.
2. Skills
Every skill is a collection of step-by-step instructions that encode a workflow definition for a specific task that you execute often. You (or an agent) might have a skill to run a data quality test suite, to generate metadata information for a specific data product, or to create a new dbt model.
3. MCP Servers (Model Context Protocol)
MCP servers (think of them as APIs) give the model access to other tools and the data they produce. If needed, your agents can easily connect to your issue tracker (e.g., Linear, Jira), your company knowledge wiki (e.g., Notion, Confluence), your async communication tool (e.g., Slack) or a data catalog tool.
Thanks to this connection, you can allow agents to perform multiple operations: write new tickets, build charts, send messages, add metadata to the data catalog…
4. Hooks
Hooks are rule-based checks that operate entirely outside of the AI model. They are launched at the specific step of the task you decide (depending on the type of job). They are used to intercept actions and mechanically enforce strict rules, like enforcing naming conventions or checking data quality on a specific field. They are 100% predictable and deterministic as they don’t rely on AI.
5. Observability
Observability includes logging, tracing, metrics, and dashboards that expose human, machine, or agent behavior to understand what is actually happening.
6. Subagents
In a nutshell, subagents are specialized AI agents that work together within a larger system. For example, an agent that answers a natural language question with a chart might need two subagents to accomplish this: one that has the ability to transform the natural language query into SQL and another that has the ability to plot data. This separation keeps each task focused and token-efficient.
The Data Team Complexity Layer
Building a harness for your day-to-day at work or for coding is already difficult, as you need to build your company and product knowledge. But working with data adds a unique set of challenges to building a harness. These hurdles aren’t entirely new in the industry. We faced similar issues when transitioning from pure BI teams to Product Analytics and Data Science teams. Still, they are definitely worth revisiting here.
The State Problem
Application code is largely stateless, or has simple, well-defined state. A function takes inputs, produces outputs, and can be tested in isolation. Data pipelines, by contrast, are all about state transformation. Data moves through different layers, gets aggregated, joined, filtered, and enriched. Which data you have in each layer depends on the quality and structure of what came before, and the transformations you apply.
An AI agent can write code that compiles and runs without errors but still stores completely wrong data because it misunderstood how data is captured and transformed in your system.
The Environment Problem
Software Engineers need development, staging, and production environments for their code. Analytics Engineers need all of that plus development, staging, and production data warehouses or data lakes.
It is neither easy nor cheap to spin up a dev environment and start testing data transformations against production data. You need realistic test data at realistic volumes, with realistic distributions and edge cases. Most data teams I’ve worked with don’t have proper development data warehouses. They test in staging (if they’re lucky) or just ship to production and cross their fingers.
Without a safe sandbox, AI agents are as blind as the engineers who lack the sandbox, and every change becomes a potential production incident.
The Metadata Problem
Software has structure: classes, functions, APIs with documented contracts. Data has structure too (schemas, relationships, business semantics), but it’s often implicit, scattered in human brains, or missing entirely.
An AI agent needs to know:
What tables and columns exist.
What they actually mean (like “user_id” means “the unique identifier for a registered user, distinct from anonymous visitor IDs”).
How they relate to each other.
Which are the source of truth and which are derived.
What the business definitions and calculation rules are.
Without comprehensive metadata and a semantic layer to make sense of all the distinct tables in a data warehouse, an analytics AI agent will confidently write queries that are technically correct but semantically nonsense. It will join the wrong tables or misinterpret what a metric actually measures. This already happens with human analysts, and with AI, it happens faster and at scale.
The Quality Problem
In software engineering, if your code compiles and passes tests, you have reasonable confidence it works. In data, pipelines can run successfully and produce wrong results. For example, a join that silently drops 10% of rows because of a type mismatch.
Testing data transformations requires domain knowledge, realistic test data, and a certain degree of manual inspection. An AI agent needs explicit guardrails and validation rules, but a vast majority of data teams haven’t yet formalized this because humans have been doing the validation so far.
The Data-Specific Harness Anatomy
So, how do we adapt the harness components (Context Injection, Skills, MCP Servers, Hooks, Observability, and Subagents) to solve the data-specific complexities (State, Environment, Metadata, and Quality)? Here is the mapping:
Let’s go through each one:
1. Context Injection: Metadata, Conventions, and Lineage
This component absorbs the Metadata Problem and half the State Problem. BI and Data Engineering teams have already been working on these problems for years, in reporting and Data Science, well before generative AI made them relevant again.
Having your data assets, a glossary for business terms, and the semantic relations between them documented is a must, even if it’s very simple and AI-generated.
Every table, column, and metric needs to be documented instead of living in an Analytics Engineer’s head. This can live in data-asset-specific documentation, an AGENTS.md file, or a centralized semantic layer. For most data orgs, it’s a far bigger, messier corpus that they haven’t finished writing down, and that was true before generative AI entered the picture. This ensures both agents and humans stop reinventing the same logic and avoid those situations where you have three conflicting definitions of “net revenue”.
Finally, a deliberate architecture for organizing your assets (think medallion) and the ability to track lineage between assets and layers help keep things in order. They also tell the agents what transformations and filters already happened to the data, closing half the State Problem (Observability closes the other half).
Metadata, conventions, and lineage are important to have because they make the whole system more maintainable, robust, and traceable. And you should care about these qualities in the long term, even if they come at the expense of development velocity.
2. Skills: New Wrapper for a Good Old Practice
Skills were introduced by Anthropic for Claude in 2025 as a way to document repeatable flows with a set of instructions so that agents can run them. Anthropic opened the format as a public standard shortly after. In less than a year, dozens of other platforms adopted the same format.
We’ve always documented repeatable flows: by using dbt macros, for example, or by having the typical folder for those queries we execute once a month. Mature data platform teams that believe in self-serve also implemented “golden paths,” coined and popularized by Spotify3, to guide their stakeholders in how to use the platform. Golden paths are step-by-step tutorials that describe how to complete a specific task like registering a new dataset into the catalog.
The difference between those and skills is that the latter are specific to agents. However, these workflows don’t need a specific reinvention just because of that. A skill is just a documented workflow, and a workflow doesn’t care if the output is a dbt model or a slide deck, or if it is followed by a human or an AI agent.
You can have a skill to standardize how you document metadata or one to scaffold a new model following your team’s conventions.
The Skills practice is maturing fast, from individual ones to team and company ones. If I can recommend only one thing now, it’s to version your skills like you’d version code. It helps avoid duplicated work and keeps everyone following the same standards. Don’t let skills become a new source of the Quality Problem this whole harness is supposed to solve.
3. MCP Servers: Making Metadata and Quality Easier
MCPs are a new way for the different tools to interact with each other. They don’t solve any of the data Problems per se, but they make the Metadata and Quality Problems easier to manage and navigate. I see plenty of companies rushing to patch their metadata and quality gaps this way, simply because it’s suddenly so easy to do.
For example, adding a step at the end of your pipelines that updates the metadata for your data product is now much easier with agents and MCP connections than if you needed to code a new step in your pipeline. The same is true for adding a quality check.
4. Hooks: Enforcing Data Quality
As I mentioned in the Quality Problem, in data, it is not enough that your code or query compiles. Quality can degrade even with a perfectly well-written join: it can still multiply rows and break a primary key constraint downstream.
We all know the best way to avoid that is to run these checks programmatically. But there are two caveats here:
First, having full coverage is really hard. Teams start with good intentions and add checks wherever they need them most and whenever things fail, and the effort stops there because there isn’t a systematic approach to testing.
Second, you should use data contracts agreed between Product teams (backend and frontend) and Data and Analytics to enforce data quality, which is also a practice that needs perseverance and coordination with others.
If you have data contracts, then it is super easy for hooks to enforce them. You place the hook in the path of an action or workflow (e.g., before performing an aggregation for a table that feeds a daily report) and an agent cannot refuse to run it.
Teams that already use data contracts, or at least have a programmatic approach to data quality, have a massive advantage here. Agents can reuse what they have, and improve on it pretty easily given a good foundation that also helps in reaching full coverage faster.
5. Observability for Data: the Art of Ensuring Data Correctness
When introducing observability, I described it as logging and tracing actions. When applied to data, though, observability gains a whole new meaning. For data professionals, observability means ensuring the data that lands in a data lake is actually correct.
A pipeline can pass a data contract (via a hook), yet the data it produces can still drift from the truth in ways a static check would never catch. This is where distribution monitoring and anomaly detection on business metrics come in: catching drops and spikes that cannot be checked just with today’s batch of data, to give an example.
The industry has historically solved this with data quality and observability tooling. Some teams implement their own checks for key metrics and several tolerance ranges, and others go all in on a third party, and either way is fine.
The point is that with generative AI, this work does not disappear as we still need to ensure data is in a good state throughout its lifecycle. However, AI supercharges our abilities here: we can develop checks faster and we can get the benefit of AI intelligence to autonomously surface drops and spikes before human eyes can. If you combine this with subagents (see next section), then you can have a huge advantage.
6. Subagents and the Importance of Evaluators
In organizations that heavily use AI, making sure data is correct (the example from the previous section) no longer only happens after the daily batch job. Query results are now also generated during natural language interactions where an agent tries to answer with a chart, as I explained before.
This means the observability capabilities I discussed before need to be available for these type of use cases.
Subagents are how you solve this. In the example above, before sending the query result to the visualization subagent, we can include an evaluator subagent that checks against the original question, confirming that the SQL actually answers what was asked, not just that it runs cleanly.
This works because the evaluator subagent has been specifically built for making this ad hoc judgment call. If we were to use the same subagent that built the query, that would be cheating.
Thanks to the other components of the harness, the evaluator subagent can be trained with similar questions (with their correct answers), with a lot of curated context like golden datasets and trust rules.
The data-specific harness I described in this section addresses three of the four problems from the data complexity layer section, but I haven’t addressed the Environment Problem at all.
Unfortunately, and to make things even more complex for data teams, none of the six harness components will give you a safe warehouse to test against. None of them can sample your production data and keep it fresh at realistic volumes and distributions. Building it is an investment. My advice is for you to go build this prerequisite, let it be a sandbox or a full replica database or something in between, but analytical roles (human or AI agents) deserve a safe playground to test.
Final Words
The foundation of every component I described for the data-specific harness isn’t new work created for generative AI. Yes, you might need to translate a golden path into a skill, but the meat of the golden path stays the same.
Actually, these foundations are the same disciplined approach required to treat data as a product driver in the first place.
This is why teams that already treat data as a product driver, the third and most mature phase I describe in my book, are best positioned to build and benefit from a real harness. They already have the ownership, the data product documentation, the contracts. For them, harness engineering is simply surrounding an agent with existing practices.
On the other hand, data teams still operating as a support function have to build both at once: the product discipline and the harness on top of it.
If generative AI is pushing the most mature teams into a fourth phase, it will be thanks to everything I walked you through in this article. If you are still in the support-function phase, the fastest lever is to start writing down everything.
Enjoyed this post? You might like my book, Data as a Product Driver 🚚.
https://mitchellh.com/writing/my-ai-adoption-journey
https://www.langchain.com/blog/the-anatomy-of-an-agent-harness
https://engineering.atspotify.com/2020/8/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem