The OCC changes a date format in one production export. Nobody tells you. Your pipeline reads the file, parses what it can, drops the rows it can’t, and lands the rest in PPDM. Every task shows green. The DAG runs clean for three weeks. Then close comes, an analyst notices a lease is short about 4,000 barrels, and you spend two days tracing it back to a format change that happened on the 6th.
That gap between “the pipeline ran” and “the pipeline ingested correct data” is exactly what a data contract closes. It is a small, boring artifact that would have failed the load on the 6th and paged whoever owns the OCC feed, instead of surfacing as a reconciliation mystery in front of the comptroller.
What a data contract actually is
A data contract is an explicit, machine-checkable agreement about the data crossing a boundary between a producer and a consumer. It says: here are the fields, here are their types, here is how fresh the data will be, and here are the assertions that have to hold for this data to count as valid. If reality violates the contract, the check fails before the data lands.
That is the whole idea. The value is in where the check runs: at the boundary, on the way in, before anything downstream depends on it.
It helps to be clear about what a contract is not, because the term gets stretched.
It is not a legal document. Nobody signs it. It does not obligate the OCC or your SCADA vendor to anything, and it never will. The producer in an upstream shop is usually a regulator or a vendor who has never heard of you and will change their format whenever they like. The contract is your side of the boundary. You are encoding what you currently believe the producer sends, so that when that belief becomes false, you find out on the 6th instead of at close.
It is also not a schema migration. A migration changes your tables. A contract checks incoming data against an expectation and fails loudly when they diverge. When the OCC adds a column, your migration is one response. The contract is what tells you a response is needed at all.
And it is a different thing from two neighbors it gets confused with. Quality gates are the automated tests running inside your pipeline against data you already trust the shape of, covering transformation logic like allocation math and daily-to-monthly rollups. We cover those in automated data quality gates. A data product is the consumer-facing packaging on top of clean data, with an owner and published SLOs, which we get into in why your well data is not a data product. The contract sits earlier than both. It guards the front door.
The OCC-to-PPDM example
Here is the failure a contract catches, in the shape it actually takes.
You pull monthly production from the OCC. Your extractor reads a CSV, your parser maps columns, and your load writes rows into a PPDM-aligned production table keyed on well and reporting period. This is the pipeline we walked through in OCC data ingestion, and it runs nightly without anyone touching it. That reliability is the trap. A silent pipeline that keeps returning green trains everyone to stop looking.
Three ways this feed breaks without failing:
The producer changes a format. A date field that came in as MM/DD/YYYY starts arriving as YYYY-MM-DD, or a volume column picks up thousands separators. A permissive parser coerces what it can and nulls the rest. Row counts look normal. The nulls do not show up until an allocation sums low.
A field gets renamed or reordered. The same problem we flagged with vendor SCADA APIs in SCADA reliability: a column you join on quietly moves or gets a new header, and a positional parser reads the wrong column into the right field. Now you have plausible numbers in the wrong place, which is worse than a crash.
The data is stale. The OCC posts late, or the source URL 200s with last month’s file. The pipeline succeeds. It just succeeded on old data, and nothing in a green run distinguishes fresh from stale.
None of these throw an exception. That is the point. The pipeline is doing exactly what you told it to. A contract adds the one thing the pipeline was missing: a statement of what correct looks like, checked before the load.
A minimal contract has three parts
You do not need the full specification to get value. A useful contract for an OCC feed has three sections, and you can write it in an afternoon.
Schema. The fields you depend on, their types, and which ones cannot be null. For OCC production that is the API number as a string (never a number, or your leading zeros vanish), the reporting period as a date, and the volume as a decimal. If the API number stops being 10 or 14 digits, or the date stops parsing, the contract fails.
Freshness SLA. How current the data has to be to count. For a monthly feed, the max reporting period in the file should be no older than the last closed month. A freshness assertion is what catches the stale-file case that a green run hides.
Quality assertions. The domain checks that a generic type system will not give you. API numbers match the format for your state. Volumes are non-negative. Reporting periods fall inside a sane window, not 1970 and not next year. A well appears once per period, not three times. These are cheap to write and they are the ones that actually fire.
That is it. Three fields, one freshness rule, four assertions. Small enough that keeping it current is not a project, specific enough that it catches the real failures.
Tooling: Data Contract CLI and ODCS
You can hand-roll these checks, and plenty of teams do. But there is a standard and a tool worth knowing before you write your own.
The Open Data Contract Standard (ODCS), now at v3.1.0 and Apache-2.0 licensed under the Bitol project, defines a YAML structure for a data contract with sections for fundamentals, schema, data quality, and service levels.[1] It gives you a common shape so the contract is not one engineer’s bespoke JSON.
The Data Contract CLI is a Python tool that reads that standard and does the work.[2] The command that matters is datacontract test <contract>, which validates the contract file, connects to a data source (Postgres, Snowflake, BigQuery, Kafka, S3, and others), checks that records comply with the declared schema, and verifies the quality and freshness assertions. It installs with pip and runs standalone, in CI, or as a library. ThoughtWorks put it in the Assess ring on their Technology Radar, which is the right posture: worth trialing on a real feed, not yet a default you adopt blind.[3]
A rough sketch of what the schema and quality sections look like for the OCC feed:
schema:
occ_production:
properties:
api_number:
type: string
required: true
pattern: "^[0-9]{10}([0-9]{4})?$"
reporting_period:
type: date
required: true
volume_bbl:
type: number
required: true
minimum: 0
quality:
- type: sql
query: >
SELECT count(*) FROM occ_production
GROUP BY api_number, reporting_period HAVING count(*) > 1
mustBe: 0
Treat that as illustrative, not copy-paste ready: check the current ODCS field names against the spec before you commit one, because v3 renamed things from v2. The shape is the takeaway. Fields with types and a regex, plus a SQL assertion for the duplicate-well check that no type system expresses.
Wiring the check into an Airflow DAG
The contract earns its keep when it runs automatically, in the right spot: after extract, before load. You want to reject a bad file, not clean it out of PPDM afterward.
The order is extract, then contract check, then load. If the check fails, the load never runs, and the failure pages the feed’s owner instead of showing up at close.
from airflow.decorators import dag, task
from datacontract.data_contract import DataContract
@dag(schedule="@daily", catchup=False)
def occ_production_ingest():
@task
def extract() -> str:
# pull the OCC file, land it in staging (Postgres, S3, wherever)
return "occ_staging.production"
@task
def check_contract(staged: str):
result = DataContract(
data_contract_file="contracts/occ_production.odcs.yaml"
).test()
if not result.has_passed():
raise ValueError(f"OCC contract failed: {result.result}")
@task
def load():
# merge staging into the PPDM-aligned production table
...
staged = extract()
check_contract(staged) >> load()
occ_production_ingest()
The mechanics are ordinary Airflow. The discipline is the sequencing. check_contract runs against the staged data, and the >> dependency guarantees load only fires after the check passes. When the OCC changes that date format on the 6th, the contract task goes red that night, the owner gets paged, and the bad rows never reach a table anyone reports off.
Two things to decide up front. First, staging: the contract needs the raw data somewhere it can query it, so land to a staging table or object store before the check rather than validating in memory. Second, severity. Not every violation should halt the DAG. A renamed required field should hard-fail. A freshness miss on a source that is often a day late might warrant an alert-and-continue instead. Wire the response to the severity, the same call we make on quality gates: fail the task, route to a dead-letter path, or alert and keep going.
Where it stops helping
A contract is honest about a narrow job. It validates data at a boundary against what you declared. It does not clean the data, it does not fix the producer, and it does not tell you the OCC’s numbers are wrong in the first place, only that they changed shape from what you expected.
It also adds a maintenance surface. A contract that drifts out of date is worse than none, because it fails on changes you already accepted and everyone learns to ignore the alert. Somebody owns the contract, the same way somebody owns the pipeline. If no name is attached, it rots.
The reason to write one anyway is the arithmetic every operator already knows. The cost of a contract is an afternoon and a YAML file per feed. The cost of not having one is the two days you spend in a war room during close, tracing 4,000 missing barrels back to a format change nobody caught. Write the boring artifact. Start with the single feed that has burned you most, put the check in front of the load, and give it an owner.
Bitol, “Open Data Contract Standard (ODCS),” v3.1.0, Apache 2.0 license. https://github.com/bitol-io/open-data-contract-standard ↩︎
Data Contract CLI, project documentation and repository. https://github.com/datacontract/datacontract-cli ↩︎
ThoughtWorks, “Data Contract CLI,” Technology Radar (Assess). https://www.thoughtworks.com/radar/tools/data-contract-cli ↩︎