Expert insights on data solutions, database administration, data engineering, and DevOps
A SCADA tag count is not an instrument count. A six-tier taxonomy shows why a fifth of your tags are configuration, not time series.
A tag census found roughly 70% of SCADA tags belong to facilities, not wells. If you scope and price per well, you have costed a third of the job.
RAG over your well files is a data problem before it's a model problem. What chunking, embeddings, metadata, and pgvector vs Qdrant actually require.
LLM extraction pipelines fail quietly, not loudly. Here's how to catch AI pipeline drift with ground truth sets, regression suites, and distribution monitoring.
Emissions reporting gets treated as a spreadsheet task. It's a data engineering problem: flare data, allocation, and an equipment inventory nobody cleaned.
Most operators have well data. Very few have a well data product. The difference is owners, consumers, freshness SLOs, and a contract, not a platform.
You can't big-bang a legacy upstream system that runs monthly close. Here's how to displace P2, Enertia, or a homegrown stack incrementally instead.
Replace SSIS packages, stored-proc ETL, and cron scripts with Airflow and dbt. Wrap the legacy job, run in parallel, diff the output, then retire it.
A quality gate stops bad data before it reaches production, unlike monitoring after the fact. Build one with Great Expectations, dbt tests, and Airflow.
When the OCC changes a format or a SCADA API renames a tag, your pipeline stays green and bad data lands in PPDM. A data contract catches it first.
"Digital twin" covers everything from a well record to a physics simulation. Here's an honest four-level spectrum and where a mid-size operator should stop.
How to model an Airflow SCADA ingestion DAG for historian and time-series data when you have no dedicated OT middleware layer.