Back to Blog

Language: English

Reworking Terraform Without Moving State

Splitting a six-root Terraform setup organized around a 1,500-line god module into domain-specific subsystems without recreating any production resource, relying on roughly 745 moved entries plus machine verification that plans stay no-change.

I reworked a Terraform setup with six roots (dev / stg / prd across two regions). A 1,500-line god module was in place, resources were written directly in the roots, and environment-specific branching lived inside the module.

There was one constraint. Production resources could not be recreated. The code structure changes entirely, but resources in state either stay untouched or are tracked via moved blocks.

Splitting Into Domain-Specific Subsystems

I carved things into domain-specific subsystems and turned each root into a thin composition root.

Eight of them: network, data, identity, secrets_mgmt, runtime, edge, observability, and data_platform. What remains in each root is only a description of which subsystems to combine and how.

In one region, a roughly 1,500-line module called regional-environment had been holding everything. I deleted it and unified the setup so both sides reference the same subsystems as Japan. The differences are declared as inputs such as connectivity_mode and edge_topology, and the adapter side absorbs them.

I also split the observability module. security-audit had 63 resources and seven or eight responsibilities bundled into one, so I divided it into three: audit_store, slack_relay, and alerting. Dependencies flow one way only, from alerting to the other two.

Machine Verification of No-Change Plans

When changing structure, the hard part is guaranteeing that real resources stay unchanged. For every PR, I set the goal that the plan for the affected roots shows no diff. For dev and dev-b, this is verified mechanically by PR plan CI. I did not settle for a human eyeballing the plan and saying “probably fine.”

Excess or missing moved blocks are likewise checked by cross-referencing the sets of from and to addresses against the resource sets of the old and new structures. Phase 3 of the subsystem split alone contains about 230 moved entries. Counting those by eye is not realistic.

There was also a trap in how indices are handled when moving away from modules with count. Get it wrong — moving module.foo to module.bar[0], or module.foo[0] to module.bar — and the plan shows up as destroy and create. In fact I found exactly one such case and fixed it.

Here is an example showing how they actually differ.

# 間違いやすい書き方(イメージ)
moved {
  from = module.foo
  to   = module.bar[0]
}

moved {
  from = module.foo[0]
  to   = module.bar
}

When a plan looks like destroy and create, the first thing to suspect is which side carries the index.

Making Module Inputs Nullable

Alongside the structural split, I also cleaned up module inputs.

The edge inputs had 30 fields for one region and 9 for the other lined up in the same variable list. Whichever side didn’t use a field got "unused", "", 0, or []. I consolidated these into two nullable objects and enforced agreement with edge_topology via cross-variable validation. Both directions of the condition — if global, the global side is required and the regional side forbidden — are verified.

Eight sentinels checked with != "" were changed to nullable = true plus != null. At that point I confirmed, using truth tables over 6 roots × 7 resources, that no count boolean flipped.

The shape of the replacement is as follows.

# Before: 空文字を未指定の代わりに使う(イメージ)
enabled = var.endpoint != ""

# After: 未指定は null で表す
enabled = var.endpoint != null

The meaning of the check is unchanged; only the representation of “unset” was swapped.

Runtime inputs had the same shape. Only Japan’s dev went through a contract module; the other five roots wired scalars directly, switching between them with ternaries in 53 places. I unified 42 flat variables into three objects — backend / frontend / migrate — so that every environment goes through the same path.

Removing moved / import / removed Blocks

moved, import, and removed blocks can be deleted once the migration completes. If you leave them behind, the next person restructuring the code won’t know which ones are still active.

In the end I removed 745 of them (741 moved, 1 import, 3 removed).

However, the timing of that removal is risky. Delete moved blocks before they have been applied to prd, and prd’s plan appears as “destroy renamed resources and recreate them.”

So this PR stayed in Draft until the applies for prd and prd-b were complete. dev, dev-b, stg, and stg-b had been applied ahead of time, and zero destroys were confirmed.

For the same reason, chains of moved blocks need care too. If the next change lands while prd has not yet applied the Phase 3/4 moved blocks, prd’s apply has to resolve a two-step chain — god module → subsystems → for_each keys — all at once. Since dev and stg digested one step at a time, the very same change behaves differently there.

How Verification Was Assembled

With changes of this scale, passing tests are no comfort at all. Whether resources stayed unchanged can only be seen in the plan, not in tests.

What I did was this combination:

  • validate and fmt across all 7 stacks
  • Module-level tftest
  • sha256 drift checks on SQL files (28 files)
  • Machine verification of no-change plans in PR plan CI
  • Adversarial review by a separate model

The adversarial review actually produced REJECT verdicts. During data platform normalization, it flagged missing regression tests, so I fixed that and re-verified. Phase 3 of the subsystem split drew flags as well: moved indices, contract wiring, and an IAP grant mistakenly given to the development backend.

When you write something and then verify it yourself, you overlook things under the same assumptions. For changes that move structure in a big way, I make sure to insert a reader who does not share those assumptions.