Two Writers on One Cloud Run Job Break It

We have a Cloud Run job that applies DB migrations. Two paths update it. One is the DB deploy workflow, which runs gcloud run jobs update; the other is the Terraform apply workflow.

I wrote about deployments showing success while reflecting nothing in The Deploy Succeeded, but the Revision Never Switched. A bullet at the end of that post surfaces this post’s subject in a single line. This post is the substance of that line.

The plan diff that kept coming back

A particular plan run started it. Against the migrate job, the same diff appeared on every apply.

client         = "gcloud"   -> null       （イメージ）
client_version = "568.0.0"  -> null

client and client_version are administrative metadata that gcloud run jobs update attaches to the job. Terraform’s configuration lacks these attributes, so plan produces a diff nulling them each time. Apply strips them off once, and the next gcloud run jobs update attaches them again.

Env ordering is the second diff. gcloud saves env vars in the order passed on the CLI. Terraform expands a map through for_each, which sorts alphabetically. All values match, yet plan shows a diff of ordering alone.

Neither diff indicates a mistake in either configuration. gcloud writes the job in its own style, and Terraform plans the difference against its config. While two writers share the job, every apply regenerates these diffs.

etag as a timing collision

The diffs escalated beyond cosmetics. From a single push, the DB deploy and Terraform apply can run concurrently. When both update the same job, Cloud Run’s optimistic locking (etag) collides and the DB deploy workflow fails.

ERROR: ABORTED: Conflict for resource '<service>-migrate':       （イメージ）
version '...' was specified but current version is '...'.

Both workflows intended to write identical content; the overlapping timing alone broke the run. The operation itself is idempotent, so rerunning gets through.

A retry that picks up only conflicts

The first measure was a retry at the command level. Serializing the workflows through concurrency was a rejected option: the setup keeps running in parallel and retries only the failed command.

The wrapper behaves as follows:

• Retries only when stderr matches ABORTED and Conflict for resource
• Backs off exponentially, stretching waits through 5, 15, and 30 seconds, three attempts at most
• Fails fast on any other error; no blanket retry
• Reads stderr to decide, and streams it to the caller unchanged, so logging keeps flowing
• Lets environment variables override the attempt count and wait durations

Only jobs update gets wrapped. jobs create produces no conflict, so it stays unwrapped. Tests pin the behavior down: passing success straight through, succeeding after two conflicts, dropping immediately on non-conflict errors, and dropping once the attempt ceiling is exceeded.

ABORTED failures decreased after this. The recurring plan diff remained exactly as before.

An owner per attribute

Next we removed the opportunity to conflict. The job resource in the migrate job’s Terraform module carries lifecycle ignore_changes, fixing an owner for each attribute.

Attribute	Owner
client / client_version / containers[0].image / containers[0].env	DB deploy (gcloud)
Job existence, memory / cpu / timeout / max_retries, VPC, Cloud SQL, command / args	Terraform

To keep this list from growing or shrinking unnoticed, a contract test checks exact equality on all four items.

Two effects follow. Apply stops rewriting the job; Terraform touches it only for structural changes such as memory or timeout. And since the writers never overlap, etag conflicts stop occurring. The workflows stay parallel.

plan confirms the diffs are gone: the PR plan CI can verify the migrate job shows no diff.

The migrate job in every environment across both regions shares this module. One change halts the flip-flop across all environments.

Where the authority to update lives

Splitting ownership raises a question: to add an env variable, which side do I edit?

The CD side. Env additions and changes take the task definition that updates the migrate job as authoritative. The env block on the Terraform side functions solely as initial values when the job is created fresh.

A dedicated contract test watches the Prisma versions in command and args for sync with the task definition.

With one owner per attribute, the table answers what changes when you edit what. The attribute table and the contract tests hold that mapping instead of anyone’s memory.

The fate of the stopgap

We closed the retry change after the ownership separation merged. Its PR description labeled it a stopgap, and the root-cause fix superseded it.

Retries excel at failures from idempotent operations briefly overlapping, and ABORTED vanished that way. A diff that every apply regenerates survives any amount of retrying. Ending the structure that produces the diff requires deciding who writes each attribute.