The medallion pattern made its name in business intelligence. Bronze, silver, gold. Preserve raw fidelity, enforce quality before promotion, build consumer-ready models. It is the default architecture for serious data platforms now, and most data engineers know it well enough to recite the structure in their sleep.
What gets discussed less often is that the discipline matters more for operational data than it does for business data. The source data is messier. The consumers are making real-time decisions. The blast radius of bad data is larger. The same patterns that produced cleaner finance reports produce safer field operations, and they pay off faster.
This post is a case for treating SCADA, historian, and other operational data with the same architectural discipline you already apply to ERP exports.
Why the pattern fits
Business data tends to be clean at source. ERP systems enforce schema. CRMs validate inputs. Vendor master data is curated by humans. The source data still has problems, but they are problems of definition and reconciliation, not of physics. A row in the GL is either there or not. The number is either right or wrong. You can argue about which interpretation matters. You can’t argue about whether the row exists.
Operational data is different in kind. Sensors drift. Transducers fail. Pumpers enter manual readings days late. Quality flags shift after the fact. Tag names change when a workover happens. Two vendors report the same physical measurement under different names, in different units, with different quality semantics. The data is noisy because the world it describes is noisy.
The medallion pattern was built for exactly this. The promise is that the raw signal is preserved in bronze, the cleaning and normalization happen once in silver, and the consumers query gold without having to reinvent any of that work. The promise pays off more when the raw signal is messier, which is precisely the case in OT.
Bronze: raw, untouched, replayable
The bronze layer for operational data should look almost identical to the bronze layer for business data, with one additional requirement.
Raw tag data as ingested. Timestamp, tag name, value, quality flag, source system identifier. Whatever the source provided, exactly as it arrived. No interpretation. No filtering. No conversion. The schema mirrors the source.
Never modified, never deleted. The bronze layer is the source of truth for backfill and replay. If a silver-layer normalization rule turns out to be wrong, the fix is to rerun the silver transformation against the same bronze data. If bronze gets modified, that capability is gone.
Partitioned by ingestion date and source. The partitioning matters. Tag history at one-minute resolution gets large fast. Partitioning by ingestion date gives you efficient backfill queries and predictable storage costs. Partitioning by source gives you the ability to reprocess a single vendor’s data without touching the others.
Provenance attached. This is the operational-data-specific requirement. Every row should carry not just its source system but enough metadata to reconstruct where it came from. Which historian. Which connector version. Which polling interval. Which ingestion run. When the inevitable question comes (“why does this well’s pressure history look weird in May?”), the answer needs to be queryable, not investigative.
Retention policy decided here. How far back does the business actually need to replay? Three years? Seven? Forever? Decide once, encode it as a retention policy, and move on. The default of “keep everything forever” is fine if storage is cheap and queries are scoped, but it is a decision worth making consciously.
The bronze layer should be boring. If it is interesting, something is wrong.
Silver: where the real work happens
The silver layer is where the medallion pattern earns its keep on operational data. It is also where most teams cut corners and pay for it downstream.
Quality flag translation. Every vendor has their own scheme for flagging sensor data. OSI PI has its codes. Ignition has its quality bits. SaaS vendors have their own conventions. The silver layer translates all of them into a common schema (good/bad/uncertain at the most basic level, with finer-grained categories where the consumers need them). The translation is documented. The translation is tested. The translation lives in one place.
Gap handling. A sensor that goes offline for an hour can be represented in five different ways. Null. Zero. Last-known value. Linear interpolation. Vendor-specific gap marker. Each of these is correct for some use case and wrong for others. The silver layer makes one decision per tag class, encodes it, and stops every downstream consumer from having to make their own choice. This is the single most common place where five engineers end up with five different answers to the same question, and the silver layer is what prevents that.
Tag normalization. The work we covered in Why dbt Belongs in Your OT Data Stack and in the post on vendor sprawl lives here. Three different tag names from three different vendors that all refer to the same physical measurement on the same physical well get mapped to the same canonical concept, with the same unit, on the same well. The mapping is versioned. The mapping has lineage. The mapping is queryable.
Unit normalization. PSI versus kPa. MCF versus MMBTU. Barrels versus cubic meters. Every operator has at least one historical incident where a unit conversion happened in the wrong layer and a report ended up reading reservoir pressures off by a factor of seven. The silver layer is where conversions happen, once, deliberately, with tests that catch drift.
Asset contextualization. Tags belong to wells. Wells belong to pads. Pads belong to fields. The asset hierarchy lives in your master data and gets joined into the silver layer here. Without this, cross-asset analysis is not possible at all. The hierarchy is also the thing that breaks most often when assets get acquired or divested, which is why the join logic has to be tested and the hierarchy has to be governed. We covered the governance side in data governance without a data team.
The silver layer is opinionated. That is the point. The opinions are made once, by people who know what the right answer is, and every downstream consumer inherits them.
Gold: what the business actually queries
Gold should be unsurprising. It is asset-centric, consumer-shaped, and optimized for the specific questions the business asks.
Well performance summary. Daily and hourly volumes, pressures, runtimes, alarm counts, per well, with allocation context joined in. This is what the surveillance dashboard queries.
Facility status. Tank levels, throughput, equipment status, per facility, per shift. This is what operations queries.
Production by basin by day. Allocated volumes against SCADA-derived volumes, with variance flagged. This is what the production engineering team and the allocation analysts query during close.
Economics joined in. Production volumes tied to commodity prices, operating costs, royalty structures. This is what the leadership team queries when they want to understand the portfolio.
Aggregations at the right granularity for the consumer. Field engineers want fifteen-minute intervals. Executives want daily rollups. The same underlying silver data feeds both. The aggregation logic lives in gold, not in every dashboard.
The gold layer is where the question “what would I want to query if I could query anything” gets answered. Done right, the answer is “the well or the field, by the metric I care about, over the period I care about, joined to the context I care about, in one query, in one place.”
The engineer-built production surveillance app should be querying gold, not raw tags. If it isn’t, the medallion pattern is not yet doing what it was supposed to do.
Why this matters more here than in business data
Worth saying explicitly, because the case is often made implicitly and then dismissed.
The source data is noisier. Business data sources enforce a schema and validate inputs. Operational data sources report what the physical world looked like, and the physical world includes failed sensors, calibration drift, and pumpers entering manual readings two days late.
The consumers move faster. A field engineer dispatching a truck cannot wait until tomorrow for the dashboard to update. A reservoir engineer investigating a deferral has hours, not weeks. The latency tolerance is lower, and the cost of latency is higher.
The blast radius is larger. A wrong number in a finance report gets caught at close and corrected. A wrong number on a critical well can cascade into a workover decision, a deferral, or a safety incident. The cost of bad data is more immediate. The discipline of medallion makes that less likely.
Lineage is operationally critical. When the dashboard shows a well dropped thirty barrels of oil per day yesterday, the engineer needs to know whether the signal is real. Without lineage, that question takes a day to answer. With lineage, the answer is in the model graph. dbt’s docs site is a debugging tool more than a documentation tool, and the value of that tool scales with how noisy the upstream data is.
A common failure mode
The pattern we see most often is teams that built bronze and gold and skipped silver. Raw tags land in the warehouse. A few business-facing models get built directly on top of the raw tags. The dashboards work. From the outside it looks like a working medallion.
What actually happened is that every transformation that should have lived in silver got duplicated across the gold models. Quality filtering is reimplemented in three places. Unit conversions live in dashboards. Tag normalization happens in dbt models that nobody knows about. The first time someone asks why the well performance dashboard disagrees with the allocation report, the answer is that they made different opinions about the same raw data.
The fix is not to rebuild everything. The fix is to extract the duplicated logic into a silver layer, point the existing gold models at it, and stop new models from being built on the bronze layer directly. The conventions get enforced through code review and through the structure of the dbt project itself. The work is incremental. The payoff is a layer of consistency that the downstream consumers don’t have to think about.
The summary version
If your operational data is going straight from ingestion to a dashboard without a transformation layer in between, you are not running a data platform. You are running a very expensive CSV with logos on it.
The medallion pattern is what makes the platform worth the cost. The discipline is the same as in business data. The payoff is faster, larger, and easier to point at, because the downstream consumers can tell you the moment the data goes wrong.
Build bronze the boring way. Build silver as if your career depends on the opinions you encode in it. Build gold for the specific questions the business is asking today, not the hypothetical ones it might ask tomorrow.
The teams that do this stop fighting fires every month and start answering questions that matter.
Further Reading
- Why dbt Belongs in Your OT Data Stack
- Bringing SCADA Data Into Your Data Warehouse Without Breaking Either
- Your SCADA Data Is Already in Snowflake. That Doesn’t Mean It’s Reliable.
- The Hidden Cost of SCADA Vendor Sprawl
- Data Pipeline Patterns: A Practical Reference
- Crawl, Walk, Run: A Realistic Sequence for SCADA Ingestion
- Data Governance for Energy Companies That Don’t Have a Data Team