Expert insights on data engineering
IT/OT convergence stalls on the org chart, not the tech. OT (Operational Technology) keeps the field running; IT (Information Technology) runs the business. Why they split, what convergence really means, and a read-only historian path.
You don't need an MDM platform for one clean list of wells. Build per-source well dimensions and conform them into a master with a surrogate key.
The discipline that made medallion architecture the default in business intelligence matters more for operational data, not less. The source data is messier, the consumers move faster, and the blast radius of bad data is larger. Here's how bronze, silver, and gold should actually look when the input is SCADA and historian data.
Most data teams treat SCADA and OT (Operational Technology) data as a special case that lives outside the normal data stack. The same discipline that makes dbt valuable for business data is exactly what OT data is missing, and the consequences of getting it wrong are higher. Here's the case for medallion-on-dbt against OT data and the tests that catch real-world problems.
Most operators are further along on SCADA-to-Snowflake than they realize architecturally, and closer to the edge than they realize operationally. The gap between a pipeline that runs and a pipeline you can trust is observability, error handling, ownership, and recovery. Here's where the fragility actually lives and how to harden what you already have without starting over.
Full-refresh ETL against a vendor-hosted SCADA source is the easy default and the wrong one. Here's why backfills push pipelines toward full refresh, what it actually costs the source, and the layered incremental pattern (overlap window, updated_at pass, reconciliation, planned deep pulls) that gets the same correctness without the call from the vendor.
Most operators are sitting on years of SCADA history that never made it into their analytical systems. The data is rich, the integration is unglamorous. Here's why the gap persists, what a real SCADA-to-warehouse pipeline looks like, and the volume, downsampling, and reconciliation decisions you have to make on the way.
Field tickets, run tickets, JIBs, and service invoices still arrive at most mid-size operators as paper or PDFs. Here's why the digitization gap has persisted, what a real ingestion pipeline looks like, and where LLM-assisted extraction earns its keep versus where it just creates silent errors in close.
What does a real PPDM ingestion pipeline actually look like? Here's the architecture: Airflow DAGs pulling from OCC and other source systems, landing normalized data into PPDM-aligned PostgreSQL tables, and DuckDB handling the analytical layer on top. Design decisions, common mistakes, and what a working monthly cycle looks like.
Most Oklahoma operators are still pulling Oklahoma Corporation Commission data by hand every month. There's no technical reason for that. Here's what a proper OCC ingestion pipeline looks like, and what it takes to get one running.
We put our entire SDLC in git. Requirements, decisions, task assignments, everything. Then we cancelled standup. Nobody complained. OK, I complained, which is apparently how you get assigned the blog post about it.
A practical reference for data pipeline patterns: loading strategies, slowly changing dimensions, change data capture, Lambda, Kappa, and Medallion architecture, reliability fundamentals like idempotency and atomic swap, and orchestration patterns.
The hyperscale data platform pitch is compelling, but it was designed for a different customer. Here is an honest look at where open source tools like PostgreSQL, DuckDB, Airflow, and dbt outperform proprietary platforms for most Oklahoma organizations, and when the proprietary option is actually the right call.
Stop coupling DAGs by time or ExternalTaskSensor. Airflow's dataset scheduling lets you wire pipelines together through the data they produce and consume, so the right DAGs run at the right time without the fragility.
Oklahoma energy companies are sitting on enormous amounts of data spread across systems that were never designed to talk to each other. PPDM gives you a standard. Data engineering makes it actually work.
We've seen a trend of small-to-mid size Oklahoma businesses outgrowing their data setup. Here's how to tell if it's time to bring in a real data engineer.
If your Airflow Variables, Connections, and secrets only exist in the UI or someone's memory, you don't have a config strategy, you have a time-bomb. Here's how to actually fix that.
Learn how to build portable, testable data pipelines by containerizing your ETL logic and using Airflow purely as a scheduler, keeping your code independent from any specific orchestration tool.
Stop fighting for inbound VPN access. Put your Airflow workers where the data lives and let them call home.
This guide takes an existing Meltano proof-of-concept and elevates it by using a true data source and target database. We will be working with public REST API endpoints as an extractor, and we will use PostgreSQL as a loader.
Containerize a Meltano EL pipeline with Docker to get a reproducible, self-contained workflow that produces a JSONL artifact.