Expert insights on data solutions, database administration, data engineering, and DevOps
A Unified Namespace gives every SCADA tag one permanent address. Here's when a UNS is worth it for an upstream operator and when a tag dictionary wins.
IT/OT convergence stalls on the org chart, not the tech. OT (Operational Technology) keeps the field running; IT (Information Technology) runs the business. Why they split, what convergence really means, and a read-only historian path.
You don't need an MDM platform for one clean list of wells. Build per-source well dimensions and conform them into a master with a surrogate key.
How we centralized a fragmented upstream data estate spanning 20+ vendor systems into one master well table with resolved well identity.
Moving fast has real value, especially in upstream. But there's a predictable inflection point where the systems built for speed become the thing slowing you down. The earlier you build with handoff in mind, the cheaper the inflection is. Here's what 'built for handoff' actually means and how AI-assisted development changes the calculus.
The discipline that made medallion architecture the default in business intelligence matters more for operational data, not less. The source data is messier, the consumers move faster, and the blast radius of bad data is larger. Here's how bronze, silver, and gold should actually look when the input is SCADA and historian data.
Most SCADA ingestion programs fail because they try to boil the ocean. A phased approach that proves a repeatable pattern on the cleanest asset first is almost always faster end-to-end. Here's a realistic sequencing guide for someone who has been told they own SCADA ingestion and is trying to figure out where to start.
Most data teams treat SCADA and OT (Operational Technology) data as a special case that lives outside the normal data stack. The same discipline that makes dbt valuable for business data is exactly what OT data is missing, and the consequences of getting it wrong are higher. Here's the case for medallion-on-dbt against OT data and the tests that catch real-world problems.
Every acquisition comes with somebody else's SCADA stack. After enough deals you have eight to twelve platforms, no common namespace, and a field team that lives in browser tabs. The real cost isn't licensing, it's the analytics you can't run and the integrations you keep rebuilding. Here's why and what to do about it without replacing the SCADA vendors.
Most operators are further along on SCADA-to-Snowflake than they realize architecturally, and closer to the edge than they realize operationally. The gap between a pipeline that runs and a pipeline you can trust is observability, error handling, ownership, and recovery. Here's where the fragility actually lives and how to harden what you already have without starting over.
Full-refresh ETL against a vendor-hosted SCADA source is the easy default and the wrong one. Here's why backfills push pipelines toward full refresh, what it actually costs the source, and the layered incremental pattern (overlap window, updated_at pass, reconciliation, planned deep pulls) that gets the same correctness without the call from the vendor.
Most operators are sitting on years of SCADA history that never made it into their analytical systems. The data is rich, the integration is unglamorous. Here's why the gap persists, what a real SCADA-to-warehouse pipeline looks like, and the volume, downsampling, and reconciliation decisions you have to make on the way.