Forty companies. Different sizes, different stacks, different clouds. GitHub Actions, CircleCI, GitLab CI, Jenkins (yes, still), Bitbucket Pipelines. After running or auditing CI/CD for all of them, three things stand out as the clearest predictors of whether a team has pipeline-related incidents.
Not the tool. We went in expecting the tool to matter more than it does. It doesn’t. A well-run Jenkins install beats a badly-run GitHub Actions setup every time, and we have seen both. What separates the teams with quiet pipelines from the teams with a pipeline-related incident every fortnight is three habits. None of them require a migration.
1. They treat their pipeline as production code
The teams with the fewest incidents have pipeline configuration that’s reviewed, tested, and versioned the same way their application code is. No “quick YAML tweak” that bypasses review. No pipeline changes deployed on Friday afternoons. The pipeline is part of the system, not a wrapper around it.
What this looks like in practice: the workflow files live in the same repository as the application, changes go through pull requests with a named reviewer, and the reviewer actually reads the YAML rather than approving it on sight. Reusable pieces — composite actions, GitLab includes, Jenkins shared libraries — are versioned and pinned to a tag or commit SHA, never to main or latest.
The teams that do this well also test their pipelines. Not exhaustively — nobody writes unit tests for a YAML file — but they run a change through a non-production branch before it touches the path to production, and they lint the config on every commit. actionlint for GitHub Actions and the built-in GitLab CI linter catch most of what breaks.
The teams that don’t do this share a specific failure. Someone edits the pipeline directly in the web UI (Jenkins is the worst offender here, because the UI makes it so easy), the change never reaches version control, and six weeks later nobody can explain why the build behaves differently on Tuesdays. We have spent whole days reconstructing pipeline state that should have been a git log.
Pin everything
One more thing under this heading, because it causes more surprise breakages than anything else: third-party actions and base images drift. A pipeline that references actions/checkout@v4 gets whatever v4 is today. A pipeline that references a commit SHA gets exactly what was reviewed. Same for Docker base images — node:20 is a moving target, node:20.11.1-bookworm is not. The supply-chain argument for pinning is well known; the reliability argument is the one that convinces busy teams.
2. They have exactly one deployment path to production
Not two. Not a fast-track and a slow path. One. Every deployment to production goes through the same gates — automated tests, environment-based approvals, deployment verification. The teams who have multiple paths always end up using the fast path when it matters most, which is exactly when you want the gates most.
“A second deployment path is a pressure valve. It’ll get used when pressure is highest, which is when you can least afford to skip your checks.”
We hear the same justification every time: “we need a way to ship a hotfix quickly.” Fine. Make the one path quick. If the full path takes forty minutes, the answer is to cut it to ten by parallelising the test suite, caching dependencies properly and dropping the steps nobody can defend — not to build a second path that skips them.
The other common shape is the manual override. A senior engineer with production credentials who can kubectl apply or push an image tag by hand. That is a deployment path, whether or not it has a name. The worst incidents we have been called into almost all involved a change that reached production without going through the pipeline, and in most cases the person who made it was trying to fix something.
What one path buys you
Once every change flows through the same gate, the gate becomes trustworthy. You can put a real deployment verification step behind it — a smoke test against the live environment, a check that error rates haven’t moved in the first five minutes — and know it always runs. You get a complete audit trail for free. And you stop having the argument about whether this particular change is “small enough” to skip review, because there is no mechanism for skipping.
3. Rollback is tested and fast
Not theoretically possible. Tested. Monthly. With a timer. The teams with the lowest mean time to recovery practice rollback the same way good teams practice incident response — deliberately, before they need it. If rolling back takes more than 10 minutes, it’ll get deprioritised when an incident is live and the option you’ll reach for is forward-fix instead.
Ask a team how they roll back and most will describe it fluently. Ask when they last did it, and the room goes quiet. A rollback that has never been run is a hypothesis. The database migration that isn’t backward-compatible, the config value that changed shape, the feature flag defaulting to on in the old image — none of these show up until you actually try.
The teams that do this well keep it mechanically simple. Rollback means re-deploying the previous known-good artefact through the same single path described above, not rebuilding from an old commit. Artefacts are immutable and retained. Migrations are written expand-then-contract, so the previous version of the application can run against the current schema. Nobody is editing anything during a rollback; they are pressing one button that was pressed last month in a drill.
The monthly drill
Pick a non-critical service, deploy a deliberately broken but harmless change, start a timer, and roll back. Write down the time. If it is over ten minutes, that is your next piece of engineering work. If a step needed a human to remember something, automate the step or document it where the on-call engineer will actually look. Do it again next month.
Where to start
These three patterns aren’t complicated. They’re discipline. And discipline compounds.
If you’re starting from a pipeline that has none of the three, don’t try to do everything at once. This is the order we’d go in:
- Put the pipeline config under review. This costs nothing and stops the drift immediately. Pin your actions and base images while you’re in there.
- Find and close every route to production that isn’t the pipeline. Rotate credentials if you have to. Then make the one route fast enough that nobody misses the shortcuts.
- Run one rollback drill. Whatever it exposes is your backlog for the next month.
Do that and you will have covered the three habits that separated the calm teams from the rest, across every stack and every tool we’ve seen.
