I needed to check what was in a GCP secret to debug why an application wasn’t starting. Here’s how we inject secrets into GKE workloads, plus some CLI commands for debugging.

Debugging secrets with gcloud Link to heading

These commands are useful for checking secret values during debugging - not how you’d access them in production.

List all secrets:

gcloud secrets list

Filter by name:

gcloud secrets list | grep MY_SECRET

Get the latest version of a secret:

gcloud secrets versions access latest \
    --secret="MY_SECRET" \
    --project my-project

Get a specific version:

gcloud secrets versions access 3 \
    --secret="MY_SECRET" \
    --project my-project

List versions of a secret:

gcloud secrets versions list MY_SECRET --project my-project

Be careful - this prints the secret to your terminal. Don’t do it if someone’s looking over your shoulder.

How to inject secrets into applications Link to heading

Reading secrets via gcloud is fine for debugging, but not how you should get secrets into production apps. Here are the methods I actually use:

1. Environment variables (simplest) Link to heading

For Cloud Run:

gcloud run deploy my-service \
  --set-secrets=DATABASE_URL=my-secret:latest

For GKE, use the Secret Manager CSI driver. It injects secrets directly into pods without storing them in etcd - much better from a security perspective.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-secrets
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/my-project/secrets/database-url/versions/latest"
        path: "database-url"

Then mount it in your pod:

volumes:
- name: secrets
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: "app-secrets"

Both the CSI driver and kube-secrets-init use Workload Identity to fetch secrets - no service account keys needed.

Another nice thing about the CSI driver: it can sync secrets periodically. If your application watches the mounted file for changes, you get secret rotation without pod restarts.

Why CSI driver over alternatives:

  • Secret Manager CSI Driver - Secrets injected directly via Workload Identity, never stored in etcd. Supports periodic sync for rotation. The right choice for GKE.
  • External Secrets Operator - Syncs to Kubernetes Secrets which end up in etcd. Avoid for sensitive data.
  • kube-secrets-init - Also uses Workload Identity, but no longer actively developed. Migrate away if you’re using it.

2. Volume mounts (for files) Link to heading

Sometimes you need the secret as a file (like a service account key or TLS cert). Mount it:

volumeMounts:
- name: secrets
  mountPath: /secrets
  readOnly: true
volumes:
- name: secrets
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: "app-secrets"

3. Application code (most flexible) Link to heading

For Python apps, the Secret Manager client library works well:

from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()
name = f"projects/my-project/secrets/my-secret/versions/latest"
response = client.access_secret_version(request={"name": name})
secret = response.payload.data.decode("UTF-8")

This lets you fetch secrets on-demand and cache them in memory. Good for secrets that change infrequently but might need to be rotated without redeploying.

Secret rotation workflow Link to heading

We rotate secrets quarterly (or immediately if compromised). Here’s the process:

  1. Create new version:

    echo -n "new-secret-value" | gcloud secrets versions add my-secret --data-file=-

  2. Test with new version: Update one service to use the new version explicitly:

    gcloud run deploy my-service \
  --set-secrets=DATABASE_URL=my-secret:2

  3. Roll out to all services: Once confirmed working, update services to use latest:

    gcloud run deploy my-service \
  --set-secrets=DATABASE_URL=my-secret:latest

  4. Disable old version: Wait a few days, then disable (not delete) the old version:

    gcloud secrets versions disable 1 --secret=my-secret

  5. Delete after retention period: After 30 days (or your compliance requirement), delete:

    gcloud secrets versions destroy 1 --secret=my-secret

The key is disabling before deleting. You can re-enable if something breaks.

Common mistakes Link to heading

Mistake 1: Using version 1 forever

Always use latest in production unless you have a specific reason not to. Otherwise rotation requires updating every service.

Mistake 2: Not setting up alerts

Set up an alert for secret access failures:

gcloud logging metrics create secret-access-failures \
  --description="Failed secret accesses" \
  --log-filter='resource.type="secretmanager.googleapis.com/Secret" severity>=ERROR'

Mistake 3: Over-sharing secrets

Don’t give roles/secretmanager.admin to service accounts. They only need roles/secretmanager.secretAccessor for specific secrets:

gcloud secrets add-iam-policy-binding my-secret \
  --member=serviceAccount:[email protected] \
  --role=roles/secretmanager.secretAccessor

Further reading Link to heading