"failed to authorize: DeadlineExceeded" pushing to Artifact Registry with BuildKit

TL;DR Link to heading

Image pushes from GitHub Actions to Google Artifact Registry kept failing with failed to authorize: DeadlineExceeded after 60 seconds, because the buildx client’s token POST gets a 405 and sometimes doesn’t complete. Setting BUILDKIT_NO_CLIENT_TOKEN=1 on the runners stops the client sending that POST, and I haven’t seen the error in the first day since.

Motivation Link to heading

Our image builds run on self-hosted GitHub Actions runners and push to Artifact Registry. A few times a day, a build would get through each Dockerfile step and then die at the push:

#20 pushing layers 60.2s done
#20 ERROR: failed to push europe-north1-docker.pkg.dev/<project>/<repo>/<image>:<tag>: failed to authorize: DeadlineExceeded: context deadline exceeded

Some failed during cache export instead, with error writing layer blob: failed to authorize after the same 60 seconds. The google-github-actions/auth and docker/login-action steps before the push passed each time, and most reruns went green, so nobody looked into it.

Failed jobs with this error from 15 to 27 October:

That came to 49 jobs across 12 repos, with Python, Node and Go images alike. In the week before the fix, our runner metrics counted 1,637 build jobs, 125 of them failed. This error caused 38 of those 125, about 2% of all builds. A typical failure held up an image push by about 40 minutes, and a quarter sat for over 15 hours before someone reran them.

I handed the investigation to a Claude Code session.

The 60 seconds comes from BuildKit Link to heading

With the docker-container buildx driver, the BuildKit daemon runs in its own container and doesn’t hold your registry credentials. When it needs a token, it asks the buildx client over the build session, and the client makes the HTTP request. The daemon caps that wait in session/auth/auth.go:

const sessionAuthTimeout = 60 * time.Second

In our traces, the daemon’s FetchToken call to the client is the one that hit the 60-second deadline.

The build record shows the stall Link to heading

docker/build-push-action uploads a build record as a workflow artifact (<owner>~<repo>~<id>.dockerbuild), failed builds included. It’s a gzipped OCI layout. One layer, with media type application/vnd.buildkit.otlp.json.v0, holds an OpenTelemetry trace of the build, with spans from the buildx client as well as the daemon.

gh api repos/<owner>/<repo>/actions/artifacts/<artifact-id>/zip > rec.bin
mkdir rec && tar -xf rec.bin -C rec
# follow the manifest digest in index.json into blobs/sha256/,
# then find the otlp layer in that manifest

The token requests from a failed push:

12:43:47.503  60.00s  Auth/FetchToken      DeadlineExceeded
12:43:47.505  60.00s  HTTP POST /v2/token  405  context deadline exceeded
12:44:47.504   0.00s  HTTP GET  /v2/token       context deadline exceeded

The POST span lasted the full 60 seconds and recorded a 405. The GET fallback started after the deadline and failed at once.

A passing build of the same repo:

13:42:46.256  0.02s  HTTP POST /v2/token  405
13:42:46.273  0.03s  HTTP GET  /v2/token  200

The 405 comes back in 20ms here, and the GET returns a token. Each passing build sends one wasted POST per token.

37 of the 49 failures still had a build record with a trace. Counting the token requests in them:

Each failed build had one stalled POST, and none of the GETs stalled.

The POST comes from the buildx client Link to heading

The client’s auth provider (session/auth/authprovider) tries the OAuth POST endpoint first when it has a credential secret. It falls back to GET on a 404 or 401, and on a 405 when it also has a username:

// credential information is provided, use oauth POST endpoint
resp, err := authutil.FetchTokenWithOAuth(ctx, httpClient, nil, "buildkit-client", to)

On a non-2xx status, the containerd helper reads the response body to build the error. Since the span recorded a 405, my best guess is that the body read hangs. I haven’t proved it. The HTTP client has no timeout of its own, which leaves BuildKit’s 60 seconds to end it.

The daemon’s own token code tries GET first:

// try GET first because Docker Hub does not support POST
resp, err := auth.FetchToken(ctx, ah.client, nil, to)

The fix Link to heading

BUILDKIT_NO_CLIENT_TOKEN=1 in the buildx client’s environment stops the client acting as a token authority. The daemon then asks the client for its configured credentials and fetches the registry token itself, GET first. The client sends no POST.

We set it on the runner container of our ARC scale sets, so each repo picks it up without a workflow change:

containers:
  - name: runner
    env:
      - name: BUILDKIT_NO_CLIENT_TOKEN
        value: "1"

If you can’t change the runners, I’d set it as env on the docker/build-push-action step instead, which covers that one workflow.

The trade-off: the BuildKit container now receives the registry credential (here a short-lived access token) and does the token exchange. Our runners are ephemeral, and BuildKit lives in the same pod and goes away after the job, so I was happy with that.

Things that didn’t fix it Link to heading

• Upgrading buildx. We had pinned buildx at v0.25.0 against a v0.33.1 daemon, which looked suspicious. Newer buildx retries the token fetch on errors such as a 5xx or a connection reset. It adds no shorter timeout per attempt, so a stalled read can still use up the 60 seconds before a retry happens. We upgraded later anyway.
• Cloud NAT. With Private Google Access on the subnet, traffic to *.pkg.dev skips NAT. Our NAT error logs had no dropped connections to Google addresses.
• Smaller images. One repo started passing after its image shrank by 1.5GB, which pointed at layer size. In the failed traces, though, the token request stalled before any layer upload, and image size doesn’t explain that.

Results so far Link to heading

Traces from builds after the rollout, against the failed builds from before. I picked the before sample for failure, so it shows the change in token requests and says nothing about how often the stall happened:

Build jobs in the first 24 hours after the change, against the week before:

The drop in failures is bigger than this error can explain, and I can’t attribute the rest from these figures. The failed image builds I checked from that day broke in their own Dockerfile steps, none at the push.

At around five failures a day before the change, I’d want a week of data before calling it gone. I also still don’t know why Artifact Registry sometimes starts a 405 reply and doesn’t finish it. The change stops sending the request that stalls but doesn’t explain the stall.

Further reading Link to heading

• BuildKit v0.33.1 session/auth/auth.go, where the 60-second timeout lives
• BuildKit v0.23.0 authprovider.go (the version buildx v0.25.0 embeds), the client’s POST-first token fetch and BUILDKIT_NO_CLIENT_TOKEN
• Docker build summaries for GitHub Actions, which explains the build record artifact
• Distribution OAuth2 token spec and token spec
• Artifact Registry Docker authentication