Expert insights on data engineering
A SCADA tag count is not an instrument count. A six-tier taxonomy shows why a fifth of your tags are configuration, not time series.
RAG over your well files is a data problem before it's a model problem. What chunking, embeddings, metadata, and pgvector vs Qdrant actually require.
You can't big-bang a legacy upstream system that runs monthly close. Here's how to displace P2, Enertia, or a homegrown stack incrementally instead.
Replace SSIS packages, stored-proc ETL, and cron scripts with Airflow and dbt. Wrap the legacy job, run in parallel, diff the output, then retire it.
A quality gate stops bad data before it reaches production, unlike monitoring after the fact. Build one with Great Expectations, dbt tests, and Airflow.
When the OCC changes a format or a SCADA API renames a tag, your pipeline stays green and bad data lands in PPDM. A data contract catches it first.
How to model an Airflow SCADA ingestion DAG for historian and time-series data when you have no dedicated OT middleware layer.
A Unified Namespace gives every SCADA tag one permanent address. Here's when a UNS is worth it for an upstream operator and when a tag dictionary wins.
IT/OT convergence stalls on the org chart, not the tech. OT (Operational Technology) keeps the field running; IT (Information Technology) runs the business. Why they split, what convergence really means, and a read-only historian path.
You don't need an MDM platform for one clean list of wells. Build per-source well dimensions and conform them into a master with a surrogate key.
The discipline that made medallion architecture the default in business intelligence matters more for operational data, not less. The source data is messier, the consumers move faster, and the blast radius of bad data is larger. Here's how bronze, silver, and gold should actually look when the input is SCADA and historian data.
Most data teams treat SCADA and OT (Operational Technology) data as a special case that lives outside the normal data stack. The same discipline that makes dbt valuable for business data is exactly what OT data is missing, and the consequences of getting it wrong are higher. Here's the case for medallion-on-dbt against OT data and the tests that catch real-world problems.