Splitting Production Logs into Lanes and Shrinking What We're Allowed to Emit

Delivery verification was covered in who checks that the audit logs actually arrived?. This article is about governing where logs live and what they may contain.

Splitting telemetry into lanes

Collect all your logs in one place, and both retention and access rights get dragged toward the strictest of them.

So we split sink, retention, and IAM per telemetry lane. Audit logs are kept long with access restricted. Application logs are kept short, within reach of developers.

Here’s the shape after splitting:

lane	retention	access
Audit logs	kept long-term	restricted set only
Application logs	short period	readable by developers

With lanes in place, retention and permissions can be decided independently.

The observability Terraform module was also split along the same lines into audit_store, slack_relay, and alerting. The module split itself is written up in rearchitecting Terraform without touching state.

Containing stdout output

In parallel, we contained the output itself.

We plugged the paths where PII, credentials, queries, and request bodies were ending up on application stdout. The path feeding raw production logs to an external AI analysis service has been shut off in every environment as well. Raw logs can contain user information, so anything flowing to an analysis platform must pass through redaction first.

Redaction lives behind the logger port, on a shared path. If each feature writes its own redact logic, one forgotten spot becomes a leak.

A ratchet for the legacy logger

Migrating every existing logger call in one go isn’t realistic. But if new calls keep piling up during the migration, it never ends.

So we added a ratchet that compares against merge-base and fails CI when the number of legacy logger calls increases. Accept the current count while stopping only the deterioration. The work of reducing the count proceeds at a separate pace.

Written out as the failure condition:

merge-base 時点の legacy logger 呼び出し数: N
現在の呼び出し数: M

M > N なら CI は失敗する

The only failure case is M exceeding N; everything else does nothing.