Docker in production: mistakes we made so you don't have to

There's a gap between "Docker works on my machine" and "Docker is running reliably in production at 2am without anyone watching it." This post is about that gap.

We've run containers for clients for long enough to have made most of the avoidable mistakes at least once. None of them were exotic. Every one is documented somewhere, and every one still catches teams shipping their first containerised workload, because the tutorials stop at docker run and production starts several steps after that. Here are the four that cost us the most.

1. Not setting resource limits

Containers without CPU and memory limits will compete for resources. One misbehaving container can starve others on the same host. This seems obvious in retrospect — it wasn't obvious when we first started running containers in production and the documentation was more focused on getting things running than on operating them safely.

Set limits. Always. Even generous ones are better than none.

What it looks like in practice: a service with a slow memory leak sits on a shared node for a fortnight, growing quietly. Then the host runs out of memory and the kernel's OOM killer picks a victim — usually not the leaking process, but whichever neighbour happens to hold the most pages. In Kubernetes that can take the whole node to NotReady.

With a limit, the leaking container is the one that dies, it restarts on its own, and you get a clear OOMKilled to investigate in the morning rather than the middle of the night.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    memory: "512Mi"

Two refinements. First, requests and limits are different things: requests drive scheduling, limits drive enforcement. Set both. Second, we are stricter about memory limits than CPU limits. CPU limits throttle rather than kill, and aggressive throttling produces latency spikes that are hard to diagnose, so we set memory limits everywhere and CPU limits only where a noisy neighbour is a real risk.

2. Running as root inside the container

The default for most base images. A container running as root with a misconfigured volume mount or a container escape vulnerability can cause disproportionate damage. Use a non-root user in your Dockerfile. It's three lines and it matters.

FROM node:20-alpine
RUN addgroup -S app && adduser -S app -G app
WORKDIR /app
COPY --chown=app:app . .
USER app
CMD ["node", "server.js"]

The objection we hear most is "but the container is isolated". It is isolated by kernel namespaces, not by a hypervisor. Root inside the container is root on any file you bind-mount in, root on the Docker socket if someone mounted it, and a much shorter path to root on the host when a runtime bug turns up.

The two things that break when you switch are predictable. Ports below 1024 need a capability or, more simply, a higher port behind your load balancer. And files the application writes to at runtime need to be owned by the new user, which is what the --chown above is for. Then enforce it: runAsNonRoot: true in the pod's security context makes Kubernetes refuse to start anything that regresses. A read-only root filesystem is the natural next step.

3. Ignoring graceful shutdown

SIGTERM handling. This tripped us on our first production deploy. The orchestrator sends SIGTERM to your container when it wants to stop it. If your application doesn't handle it, the orchestrator waits for the timeout, then sends SIGKILL. Mid-request. This is the most common cause of "mysterious 502s during deployments" we see.

"Every container that can't explain its shutdown behaviour to the orchestrator is a 502 waiting to happen."

There are two halves to this, and teams usually fix only one.

The first half is whether the signal reaches your process at all. If your CMD is written in shell form — CMD node server.js — Docker wraps it in /bin/sh -c, the shell becomes PID 1, and the shell does not forward SIGTERM to its child. Your carefully written handler never runs. Use the exec form, CMD ["node", "server.js"], or add a minimal init such as tini as the entrypoint.

The second half is what your application does when the signal arrives: stop accepting new connections, finish the requests already in flight, close database connections cleanly, then exit zero.

process.on('SIGTERM', () => {
  server.close(() => process.exit(0));
});

On Kubernetes there is a third wrinkle. When a pod is terminated, the SIGTERM and the removal from the Service's endpoints happen in parallel, not in sequence, so for a second or two the load balancer can still send traffic to a container that has stopped listening. A short preStop sleep — five seconds is usually enough — lets the endpoint update propagate before shutdown begins.

Test it: run time docker stop against your image. If it takes ten seconds, that is Docker's default timeout expiring and your application never handled the signal.

4. Fat images in production

Multi-stage builds exist for a reason. Your production image should not contain your build toolchain, test dependencies, or development utilities. Smaller images mean faster pulls, smaller attack surface, and faster cold starts. The effort to implement multi-stage builds is about two hours the first time, then it's a template you copy.

FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/api

FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

The cost of a fat image is paid in two places. Autoscaling: every new node pulls the image before it can serve, and an image well over a gigabyte turns a thirty-second scale-out into several minutes. Security: your vulnerability scanner will flag CVEs in compilers, package managers and shells that your application never executes, and someone has to triage every one.

Two habits go with the multi-stage pattern. Keep a .dockerignore so the build context doesn't ship your .git directory and node_modules. Pin base image tags — node:20.11-alpine, not node:latest — so a rebuild next month produces the same image.

The pattern behind all four

Each mistake is the same mistake. Docker's defaults are optimised for getting a developer from zero to a running container with the least friction: no limits, root, no signal handling, everything in one image. Those are the right defaults for a laptop and the wrong defaults for a host you aren't watching. Production is the act of reversing them, deliberately, one at a time.

A checklist before your next deploy

• Every container has a memory request and limit; CPU requests are set, CPU limits are a considered choice.
• The Dockerfile has a USER line and the pod spec has runAsNonRoot: true.
• CMD and ENTRYPOINT are in exec form, or tini is PID 1.
• The application handles SIGTERM, and docker stop returns in well under ten seconds.
• A preStop hook and a grace period longer than your slowest request are configured.
• The production image is built in a separate stage from a pinned base, and a .dockerignore exists.

It is the difference between a container that runs and a container that can be left alone, and the second one is the only kind worth paying to host.