Data Contracts: A Missed Opportunity
Newsletter Content
Data Contracts: A Missed OpportunityThe Conversation We Should Have Had—Before Thought Leadership Replaced System Design
Over the last couple of years, the data industry has been having a conversation about data contracts that never quite went where they needed to. There was no shortage of activity around the topic. Definitions were proposed and refined. Conceptual boundaries were drawn and redrawn. Data contracts were compared to APIs, governance frameworks, data mesh primitives, and ideas teams already “sort of” implemented in practice.
What was largely absent was sustained engagement with the engineering consequences of taking data contracts seriously. Questions about enforcement, evolution, compatibility, and failure modes appeared only briefly before the conversation moved on. The result was an industry consensus that data contracts were “interesting,” without a shared understanding of what it would actually mean to build platforms around them. In hindsight, this matters—not because the debate was unproductive, but because of what happened in parallel. The Shift Happening Elsewhere
While the data community was debating what data contracts were, the software engineering world was converging—quietly and pragmatically—on a different organizing principle: specifications as the primary unit of system design. This wasn’t a philosophical shift so much as an operational one. As systems became more distributed, more automated, and more interdependent, informal agreements stopped scaling. Documentation drifted. Assumptions diverged. Human coordination became the bottleneck. The response was not more process, but more precision. APIs began with schemas rather than code. Infrastructure moved from scripts to declarative specifications. Compatibility rules were automatically encoded and enforced. In these systems, the specification was no longer an artifact produced alongside the system—it was the system. More recently, AI agents have accelerated this trend. Agents do not operate on intent, convention, or context. They operate on explicit, machine-readable, and verifiable data. Where specifications exist, agents can reason deterministically. Where they do not, agents approximate—and approximation is rarely acceptable in core infrastructure. This is where the connection to data contracts becomes unavoidable. Data Contracts as Specifications, Not Concepts
Viewed through the lens of spec-driven development, data contracts stop looking like a data-specific innovation and become a familiar pattern applied to a different domain. A properly implemented data contract is a specification:
This is exactly what spec-driven development has optimized for in software engineering. The difference is not conceptual—it is operational. Software engineering treated specifications as executable constraints. The data industry often treated contracts as descriptive artifacts. As a result, contracts were discussed as governance tools or communication mechanisms, rather than as interfaces with failure semantics. That framing limited how far the idea could go. Where the Two Worlds DivergedSpec-driven systems force early clarity around hard problems.
In software systems, these questions are answered in code and tooling. In data systems, they were often answered socially. Producer–consumer agreements existed, but they lived in tickets, meetings, and tribal knowledge rather than in executable form. Many teams compensated by building partial solutions: strong schemas, upstream quality checks, and informal SLAs. These patterns worked, but they relied heavily on human intervention. They were resilient, but not legible to machines. As long as humans were the primary integrators, this was manageable. As soon as AI agents enter the workflow, it becomes a constraint. Why Spec-Driven Thinking Changes the Next PhaseAI agents make an implicit demand of data platforms: make your rules explicit. Agents can generate schemas, propose transformations, reason about compatibility, and enforce policy—but only if the platform exposes contracts in a form they can execute against. Without that, agents revert to inference, which introduces uncertainty precisely where determinism is required. This is the practical implication of spec-driven development for data engineering. It’s not about adopting a new paradigm. It’s about recognizing that the platform already behaves like a system of interfaces—and formalizing those interfaces accordingly. Teams that have already internalized contract discipline will find this transition incremental. Teams that have not will experience it as friction. What We Should Do NextAt this point, the terminology matters less than the mechanics. Whether we call them data contracts, data interfaces, or executable schemas, the path forward is the same:
This is not about adding process. It is about making systems legible to other systems. Closing ThoughtThe original data contracts conversation wasn’t wrong. It just stopped too early. Spec-driven development has shown that explicit, enforceable interfaces are not optional in complex, automated systems. Data platforms are now at that same inflection point. Data contracts were never the destination. They were the missing layer that would have made everything else easier to build. The opportunity is still there—but only if we’re willing to treat contracts as infrastructure, not ideas. Referenceshttps://www.thoughtworks.com/insights/podcasts/technology-podcasts/data-contracts-what-why https://airbyte.com/data-engineering-resources/data-contracts https://soda.io/blog/what-are-data-contracts https://atlan.com/data-contracts/
|
Details
Source:
Other
Date:
Jan 21, 2026 01:31
Category:
Technical