There’s a certain kind of pain that every data team eventually runs into. You’re on call, something’s broken at 11pm, and after twenty minutes of digging you realize the DAG is failing because a Variable that used to exist… doesn’t anymore. Nobody deleted it on purpose. It just kind of drifted away. Maybe someone was cleaning up. Maybe a new environment got spun up and nobody ported it over.
This is config drift, and it’s genuinely one of the more insidious failure modes in data engineering workflows because it tends to be invisible right up until it isn’t.
Airflow Variables: Convenient Until They Aren’t
Airflow Variables are fine for what they are, a quick way to parameterize DAGs without hard-coding values. But the way most teams use them creates problems:
# Easy to write, easy to forget where the value actually came from
threshold = Variable.get("daily_row_threshold", default_var=10000)
The issue isn’t the code. It’s that daily_row_threshold lives in the Airflow metadata DB, probably got set via the UI,
and is not tracked anywhere. You can’t git diff it. You can’t review it. You can’t easily verify that staging has the same value as prod.
A better pattern: export your Variables to a JSON file and commit it.
{
"daily_row_threshold": "10000",
"pipeline_owner_email": "data-team@yourcompany.com",
"s3_output_bucket": "your-bucket-prod"
}
Then import on deploy:
airflow variables import variables.json
This won’t work perfectly for everything, you probably don’t want prod bucket names in your repo, but for non-sensitive config, it’s a massive improvement. Now there’s a paper trail. PRs require review. New environments can be bootstrapped consistently.
For the sensitive stuff, that’s where secrets backends come in.
Secrets Backends: The Right Way to Handle Sensitive Config
Airflow has supported pluggable secrets backends since 1.10.10.
The idea is simple: instead of storing sensitive values in the Airflow metadata DB, you delegate to an external secrets manager. Airflow checks there first when resolving Variables and Connections. The lookup order is: secrets backend → environment variables → metadata DB. You can use that ordering to your advantage, in local dev, set things as env vars; in prod, the backend takes precedence. Your code doesn’t change.
AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, and Azure Key Vault all have first-class provider support. But if you’re already using something like Bitwarden Secrets Manager and don’t want to bolt on a whole new piece of infrastructure just for this, the backend interface is pluggable enough that you can write your own without much effort.
Regardless of which backend you pick, the config in airflow.cfg follows the same pattern:
[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}
Now when your code calls Variable.get("db_password"), Airflow checks the secrets backend first before falling back to the metadata DB.
Your secrets are versioned, auditable, and access-controlled. You’re not storing plaintext passwords in the Airflow DB. Everyone is slightly less stressed.
Connections Are the Other Half of This Problem
Variables get most of the attention, but Connections are at least as bad when it comes to drift. Teams routinely hand-configure connections in the Airflow UI and then forget about them. New environments are missing connections. Credentials change and nobody updates all the environments. You get the idea.
Pluggable secrets backends can support connections too. Depending on your backend you can store something like this using Airflow’s URI format:
# Secret name in Bitwarden, etc:
airflow/connections/my_postgres
# Secret value:
postgresql://airflow_user:your-secret-password@your-db-host:5432/analytics
If the URI format doesn’t capture everything you need (extra SSL params, SSH tunnels, etc.), you can store it as a JSON blob instead. Again, check your implementation.
For non-sensitive connections in local dev and CI, you can skip your secrets backend entirely and use environment variables. Airflow picks these up automatically with the AIRFLOW_CONN_ prefix:
export AIRFLOW_CONN_MY_POSTGRES="postgresql://user:pass@localhost:5432/dbname"
Put safe placeholder values in a .env.example that gets committed to the repo. Then you can use your secrets backend for production environments,
recall that it would then take precedence over environment variables.
Treat It Like Code
The phrase “treat X as code” gets thrown around a lot, but here’s what it actually means in practice for Airflow config:
Nothing should only exist in the UI. If it’s configured through the Airflow UI and not represented somewhere in your repo or a managed secrets store, it’s invisible. When something breaks, you’ll have no idea what changed.
Bootstrap scripts should be idempotent. Write a script that imports your Variables, creates your Connections, and seeds any other state your Airflow environment needs. Run it in CI. Run it on new environment setup. Make it so that spinning up a fresh Airflow instance gives you something that actually works.
#!/bin/bash
airflow variables import config/variables.json
airflow connections import config/connections.json
Use environment-specific config files, not environment-specific UI state. It’s okay to have variables.prod.json and variables.staging.json if they need to differ. What’s not okay is having the difference live only in someone’s muscle memory.
Document what’s expected. At minimum, a README that says “this DAG requires a Variable called X and a Connection called Y” is better than nothing. Ideally, your bootstrap script serves as that documentation.
The Airflow.cfg Problem
One more thing that often gets overlooked: the Airflow configuration file itself. airflow.cfg has dozens of settings that affect behavior — parallelism, scheduler heartbeat, email config, executor settings. These absolutely need to be version-controlled.
Don’t just copy the default cfg and make changes by hand. Commit your cfg, or better, manage settings through environment variables (Airflow supports AIRFLOW__SECTION__KEY format for all config values) and commit those to your deployment config.
export AIRFLOW__CORE__PARALLELISM=32
export AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=5
export AIRFLOW__WEBSERVER__EXPOSE_CONFIG=False
This makes environment differences explicit and reviewable rather than buried in a file on some server.
Putting It Together
None of this is particularly complicated to implement, but it does require some intentionality. The rough checklist:
- Non-sensitive Variables → JSON file in version control, imported on deploy
- Sensitive Variables and Connections → Pluggable Secrets Backend (Bitwarden, Vault, AWS SM)
- Local dev Connections → Environment variables,
.env.examplein repo - Airflow.cfg settings → Environment variables managed in your deployment tooling
- A bootstrap script that ties it all together and runs in CI
The goal is that if your metadata database got wiped tomorrow, you could rebuild it from your repo in under an hour without hunting through someone’s email history trying to figure out what a connection was supposed to point to.