Why Docker pushes to Artifact Registry time out at exactly 60 seconds

TL;DR Link to heading

Our image pushes from GitHub Actions to Google Artifact Registry kept failing with failed to authorize: DeadlineExceeded: context deadline exceeded, always about 60 seconds into the push, and usually passing on a rerun. The 60 seconds is BuildKit’s limit on how long it waits for the buildx client to hand back a registry token. The client asks Artifact Registry for that token with a POST first. Artifact Registry doesn’t support it and answers 405, which normally takes 20ms, after which the client falls back to a GET that works. Occasionally the 405 reply never finishes, the client sits on it until BuildKit gives up, and the GET never runs. Setting BUILDKIT_NO_CLIENT_TOKEN=1 where buildx runs makes BuildKit fetch tokens itself, and BuildKit tries GET first.

The symptom Link to heading

Every Dockerfile step passes, then the push dies:

#20 pushing layers 60.2s done
#20 ERROR: failed to push europe-north1-docker.pkg.dev/<project>/<repo>/<image>:<tag>: failed to authorize: DeadlineExceeded: context deadline exceeded

Sometimes it’s a cache export instead, error writing layer blob: failed to authorize, after the same 60 seconds. The google-github-actions/auth and docker/login-action steps before it succeed every time. All of it ran on our self-hosted ARC runners, using docker/build-push-action with the docker-container buildx driver.

GitHub keeps job logs for about eight days, so that’s as far back as anyone can count. Grepping every failed job in the org for the error over that window:

That’s 49 jobs across 12 repos, with Python, Node and Go images alike. 41 were image pushes and 8 were cache exports. In the week before the fix, our runner metrics counted 1,523 successful and 125 failed build jobs. Roughly a third of those failures were this error, about 3% of all builds.

A rerun usually passed, so nobody looked at it properly. I handed it to a Claude Code session to dig into, and this is what it found.

Where 60 seconds comes from Link to heading

With the docker-container driver, the BuildKit daemon runs in a container and never sees your Docker credentials directly. When it needs a registry token, it asks the buildx client over the build session, and the client does the HTTP request. The daemon caps that wait in session/auth/auth.go:

const sessionAuthTimeout = 60 * time.Second

So “failed to authorize: DeadlineExceeded” after 60 seconds means the client didn’t answer the daemon’s FetchToken call in time. Something held the client up for the full minute.

The build record has the answer Link to heading

docker/build-push-action uploads a build record as a workflow artifact (<owner>~<repo>~<id>.dockerbuild), including for failed builds. It’s a gzipped OCI layout, and one of its layers, with media type application/vnd.buildkit.otlp.json.v0, is an OpenTelemetry trace of the build. It includes spans from the buildx client, not only the daemon.

gh api repos/<owner>/<repo>/actions/artifacts/<artifact-id>/zip > rec.bin
mkdir rec && tar -xf rec.bin -C rec
# the manifest in index.json lists the layers; find the otlp one

The trace is a stream of JSON span objects. Pulling out the token requests from a failed push gave this:

12:43:47.503  60.00s  Auth/FetchToken      DeadlineExceeded
12:43:47.505  60.00s  HTTP POST /v2/token  405  context deadline exceeded
12:44:47.504   0.00s  HTTP GET  /v2/token       context deadline exceeded

The POST got a status code back (405) but its span lasted the full 60 seconds. The status line arrived and the rest of the reply never did. The GET fallback started after the deadline had passed and failed straight away.

A passing build of the same repo looks like this:

13:42:46.256  0.02s  HTTP POST /v2/token  405
13:42:46.273  0.03s  HTTP GET  /v2/token  200

Same 405, in 20ms, then a GET that works. Every passing build pays for one pointless POST per token.

Of the 49 failures, 37 still had a build record with a trace. Every token request in them, counted:

Each failed build has exactly one stalled POST, and no GET ever stalled.

Why the client sends a POST Link to heading

The buildx client’s auth provider (from session/auth/authprovider) uses the OAuth POST endpoint whenever it has a password, and only falls back to GET on a 404, 401 or 405:

// credential information is provided, use oauth POST endpoint
resp, err := authutil.FetchTokenWithOAuth(ctx, httpClient, nil, "buildkit-client", to)

On a non-2xx status the containerd helper reads the response body to build the error. Since the 405 status had already arrived, that read is the likeliest place it hangs. The HTTP client has no timeout of its own, so only BuildKit’s 60 seconds ends it.

The daemon’s own token code does it the other way round:

// try GET first because Docker Hub does not support POST
resp, err := auth.FetchToken(ctx, ah.client, nil, to)

The fix Link to heading

BUILDKIT_NO_CLIENT_TOKEN=1 in the buildx client’s environment makes the client refuse to act as a token authority. The daemon treats that as “no client tokens”, asks the client for the username and password instead (an in-memory lookup, no HTTP), and fetches the token itself, GET first. The POST is never sent.

We set it on the runner container of the ARC scale sets, so every repo picks it up without a workflow change:

containers:
  - name: runner
    env:
      - name: BUILDKIT_NO_CLIENT_TOKEN
        value: "1"

If you can’t touch the runners, setting it as env on the docker/build-push-action step does the same for one workflow.

The trade-off: the registry credential (here a short-lived access token) now goes to the BuildKit container, which does the token exchange instead of the client. On an ephemeral runner where BuildKit lives in the same pod and is deleted after the job, that seemed fine.

What doesn’t fix it Link to heading

• Upgrading buildx. Ours was pinned at v0.25.0 against a v0.33.1 daemon, which looked suspicious. Newer buildx wraps the token fetch in a retry, but the retry only fires on errors such as a 5xx or a connection reset. A reply that never finishes isn’t an error, so it still sits there for 60 seconds.
• Cloud NAT. With Private Google Access on the subnet, traffic to *.pkg.dev doesn’t go through NAT at all. Our NAT error logs had no dropped connections to Google addresses.
• Smaller images. One repo started passing after its image shrank by 1.5GB, which pointed at layer size. The stall happens right at the start of the push, though, and a smaller image only shortens how long the push runs.

Did it work? Link to heading

The traces from the first hour after rollout, against the failed builds from before:

After the change the client only answers the daemon’s credential lookups, which need no network call.

That shows the request that hung is no longer sent. Whether the failures are gone for good depends on the stall being specific to the POST. Nothing so far suggests the GET stalls, but at about five failures a day before the fix, it’ll take a week of builds to say for sure.

What’s still open is why Artifact Registry sometimes starts a 405 reply and never finishes it. This change stops sending the request that triggers it, and the stall itself is still there to find.