There is a moment in most growth-stage data programs when the speed that got you here starts to be the thing slowing you down. It happens later than people expect and gets noticed later still. The team that built fast, picked good tools, and shipped useful infrastructure is the same team that wakes up at some point and realizes they cannot find a clean way to explain how any of it works to the next hire.
This is not a failure of speed. Moving fast was the right call. In upstream, where the pace of acquisition outstrips the pace of IT integration, slow is expensive in ways that show up on the deal model. The teams that win this period are the teams that move fast.
But there is a predictable inflection where the systems built for speed become the systems built around one or two people’s heads. The earlier you build with handoff in mind, the cheaper the inflection is. The later you wait, the more of your existing infrastructure becomes a cleanup project.
This post is about how to keep building at speed without producing systems that nobody else can operate.
The pattern we see
The shape is consistent across companies.
A small, capable team stands up the initial data infrastructure. One or two engineers, maybe a fractional resource on top of that. Heavy use of AI tooling for scaffolding pipelines, writing dbt models, debugging SQL. Things ship in days that used to take weeks. The output is genuinely good. The team has reason to be proud of it.
Other engineers in the organization notice. They start consuming the pipelines, building dashboards on top of the models, integrating workflows into the warehouse. The infrastructure transitions from “useful internal tool” to “load-bearing system” without anyone announcing it.
Then the person who built it leaves. Or gets promoted onto a different project. Or just gets pulled onto something more urgent and stops being the active maintainer. The system keeps running, because it was built well. From the outside, nothing has changed.
The next person who has to touch it discovers what nobody noticed during the handoff. There are no runbooks. The documentation lives in three Slack threads and an outdated Notion page. The credentials are in someone’s local password manager. The cron jobs are scattered across machines whose existence is not in any inventory. The dbt project has eighteen models and twelve of them have no documentation. The original author had all of this in their head, and they took it with them.
The next incident takes days to recover from. The one after that takes a week.
We have seen this play out enough times that we treat it as the default outcome of growth-stage data infrastructure, not an exception. The fix is not technical. It is a decision about how to build, made early enough that it doesn’t have to be retrofitted.
What “built for handoff” actually means
This is the term that gets used in talks and almost never gets defined. The components are mundane individually and cumulatively transformative.
Documentation that lives next to the code. Not in Confluence. Not in someone’s head. In the repo, near the code it describes, where it gets updated as part of the same pull request that changes the code. dbt docs for the dbt project. README files at every level of the directory tree. Inline comments on the non-obvious decisions, the ones a future reader would otherwise spend an hour reverse-engineering. The bar is not “what would I want to remember in three months.” The bar is “what would the next person, who has not yet been hired, need to know.”
Runbooks for the failure modes you can predict. Brokers go down. Vendor APIs change. Credentials expire. Clocks drift. None of these are surprising. All of them happen. A runbook does not need to be fancy. Five pages covers most of the common incidents. The deliverable is that the person on call at 2 AM can fix the problem without having to find the original author.
Alerting that fires before engineers notice. Freshness checks. Row count anomalies. Null rate spikes. The job ran but produced no data. The job has not run on schedule for three intervals in a row. The alerting matters less than the principle behind it: the team that built the system should know about the problem before the consumer does, because the consumer will tell the wrong person and the recovery takes longer.
Orchestration that makes dependencies visible. Not a cron job nobody can find. Not a scheduled task on the engineer’s laptop. A real orchestration layer where every pipeline lives, every dependency is visible, and every run has a record. We have written about Airflow patterns and the same point applies to Dagster, Prefect, or whatever the team picks. The choice of tool matters less than the choice to use one.
Secrets management that does not involve a spreadsheet. This sounds obvious. It is. We still find spreadsheets. The bar is that the next engineer can rotate credentials, audit access, and onboard themselves without needing the original author in the loop. AWS Secrets Manager, HashiCorp Vault, Bitwarden Secrets Manager, whatever fits the stack. The same principle we covered in the Airflow config post applies to credentials specifically: if the only source of truth is in someone’s head, you do not have a system, you have a time bomb.
A clear owner for each pipeline. One name. Not “the data team.” Not “whoever built it.” A person who gets paged when freshness drops, who has the authority to fix it, and who has the responsibility to escalate when they can’t. This is the single most commonly missing element, and the most consequential. The pipeline without an owner is the pipeline that breaks for three days before anyone notices.
None of these are clever. All of them are work the team often deprioritizes because the current author doesn’t need them. The whole point of building for handoff is that the future maintainer does.
The AI tooling complication
This is the part that didn’t exist five years ago, and it is the part that is worth being honest about.
AI-assisted development genuinely accelerates data engineering. Pipelines that took a week now take a day. dbt models that took an afternoon take an hour. Debugging that used to require a Slack thread now happens inside an editor. The productivity gain is real. The teams that don’t take advantage of it are leaving speed on the table.
The risk is specific. AI-generated code tends to solve the immediate problem. It does not always account for the edge cases, the operational failure modes, or the maintainability concerns that a senior engineer with scar tissue would naturally think about. The pipeline that gets generated quickly and merged without a careful review is often the pipeline that breaks in six months in a way nobody fully understands, because nobody fully understood it in the first place.
SCADA data, specifically, will find every edge case in the code. Sensors that report implausible values. Timestamps that jump backward when an RTU is restarted. Vendor APIs that silently change their schema. Brokers that drop connections under load. The code that handles the happy path well does not, by default, handle any of those.
The point is not to slow down. The point is to use the speed of AI tooling for the construction work and to spend the time saved on the parts that AI tooling does worse: thinking about how the system will behave when it breaks, writing the documentation that explains the non-obvious choices, building the tests that catch the failure modes, and reviewing the code with the assumption that the person reading it next does not have the original author’s context. Speed and discipline are not in tension. The same hour you save on the boilerplate is the hour that buys you the runbook.
The organizational layer
The technical work is the easy part to write down. The harder part is the organizational shift that has to happen alongside it.
Companies in the middle of transitioning from outsourced IT and data functions to internal ownership need to be deliberate about what they are taking on. Owning a pipeline is not the same as having one. It means owning the incidents, the on-call rotation, the vendor relationship, the upgrade cycle, and the inventory of what exists. Each of those is its own discipline. Each of those is a place where the previous arrangement (outsourced, fractional, contractor-led) created muscle memory the internal team doesn’t yet have.
The temptation in this transition is to keep building at the same pace as before, because the team is capable and the AI tooling is good and the work is exciting. The risk is that you end up six months later with twice as much infrastructure as you have operational discipline to support.
The cheaper move is to slow the rate of new build slightly and use the spare cycles to install the operational discipline behind what’s already there. Runbooks for the existing pipelines. Owner assignments. Documentation passes. Monitoring on the load-bearing models. None of this is exciting work. All of it pays for itself the first time something breaks and the recovery takes hours instead of days.
The companies that get this right are not the ones who stop moving fast. They are the ones who treat the operational work as a first-class part of the engineering work, not a thing to do later. The same point we made about the crawl phase of SCADA ingestion applies here. The documentation is the deliverable, not a side effect.
The conversation worth having now
If the team that built your data infrastructure is one or two people, and most of the institutional knowledge about how it works is in their heads, the question worth asking is concrete. If they took a job somewhere else next month, how long would it take the next person to be fully productive in their place?
If the answer is more than a few weeks, the systems are not yet built for handoff. That is not a crisis. It is a thing to start working on now, while the original authors are still around to make the work cheap.
The best time to build for handoff is at the start. The second best time is now, before the person who built everything takes a different job and leaves you to reverse-engineer their code at 2 AM.
The teams that figure this out early keep their speed. The teams that don’t end up paying for it twice: once when they have to rebuild what they don’t understand, and once more when the rebuild slows them down for a quarter while the rest of the business keeps moving.
The handoff isn’t a special event. It’s just the time when everything you didn’t write down catches up with you.