Expert insights on data engineering
Most operators are further along on SCADA-to-Snowflake than they realize architecturally, and closer to the edge than they realize operationally. The gap between a pipeline that runs and a pipeline you can trust is observability, error handling, ownership, and recovery. Here's where the fragility actually lives and how to harden what you already have without starting over.
Full-refresh ETL against a vendor-hosted SCADA source is the easy default and the wrong one. Here's why backfills push pipelines toward full refresh, what it actually costs the source, and the layered incremental pattern (overlap window, updated_at pass, reconciliation, planned deep pulls) that gets the same correctness without the call from the vendor.
Most operators are sitting on years of SCADA history that never made it into their analytical systems. The data is rich, the integration is unglamorous. Here's why the gap persists, what a real SCADA-to-warehouse pipeline looks like, and the volume, downsampling, and reconciliation decisions you have to make on the way.
Field tickets, run tickets, JIBs, and service invoices still arrive at most mid-size operators as paper or PDFs. Here's why the digitization gap has persisted, what a real ingestion pipeline looks like, and where LLM-assisted extraction earns its keep versus where it just creates silent errors in close.
What does a real PPDM ingestion pipeline actually look like? Here's the architecture: Airflow DAGs pulling from OCC and other source systems, landing normalized data into PPDM-aligned PostgreSQL tables, and DuckDB handling the analytical layer on top. Design decisions, common mistakes, and what a working monthly cycle looks like.
Most Oklahoma operators are still pulling Oklahoma Corporation Commission data by hand every month. There's no technical reason for that. Here's what a proper OCC ingestion pipeline looks like, and what it takes to get one running.
We put our entire SDLC in git. Requirements, decisions, task assignments, everything. Then we cancelled standup. Nobody complained. OK, I complained, which is apparently how you get assigned the blog post about it.
A practical reference for data pipeline patterns: loading strategies, slowly changing dimensions, change data capture, Lambda, Kappa, and Medallion architecture, reliability fundamentals like idempotency and atomic swap, and orchestration patterns.
The hyperscale data platform pitch is compelling, but it was designed for a different customer. Here is an honest look at where open source tools like PostgreSQL, DuckDB, Airflow, and dbt outperform proprietary platforms for most Oklahoma organizations, and when the proprietary option is actually the right call.
Stop coupling DAGs by time or ExternalTaskSensor. Airflow's dataset scheduling lets you wire pipelines together through the data they produce and consume, so the right DAGs run at the right time without the fragility.
Oklahoma energy companies are sitting on enormous amounts of data spread across systems that were never designed to talk to each other. PPDM gives you a standard. Data engineering makes it actually work.
We've seen a trend of small-to-mid size Oklahoma businesses outgrowing their data setup. Here's how to tell if it's time to bring in a real data engineer.