The Query Took 3.6ms, the API Took 304ms: What Went Outside Was Authentication

The stats API was slow. Opening a single request in Cloud Trace showed the backend span at about 304 milliseconds. Within it, the first DB query took 4.2 milliseconds, and the SELECT that executed took 3.6 milliseconds.

The query is plenty fast. And yet the request as a whole stayed slow. Adding indexes, rewriting the query—those 300 milliseconds wouldn’t budge.

Drilling one level deeper into the trace revealed an outbound POST sitting directly beneath the controller, eating 217 milliseconds. It was authentication.

checkRevoked as a security measure

There’s backstory here. Earlier, an issue had been reported where a Firebase ID token stolen during a login session could be reused after logout until its expiry. ID tokens are valid for up to an hour. Even after logout, a stolen token would keep getting through for that hour.

As a countermeasure, we called revokeRefreshTokens(uid) on logout and switched ordinary requests to verifyIdToken(token, true). Setting the second argument, checkRevoked, to true lets us reject tokens issued before the logout. The requirement itself—making logout the revocation boundary—is a legitimate one.

The problem was that with checkRevoked set to true, the Firebase Admin SDK calls Firebase’s GetAccountInfo (accounts:lookup) on every single request. Signature verification completes locally, but whether a token has been revoked can only be settled by asking Firebase. In the other region’s production environment we also observed the same path taking 280 to 330 milliseconds.

Which means every request from every user was carrying one round of external I/O.

Where the revocation decision lives

Weakening the requirement would make things faster. But we weren’t about to go back to stolen tokens working after logout.

So we moved where the authority to decide revocation resides.

	Before	After
Verification	verifyIdToken(token, true)	verifyIdToken(token, false)
Revocation checked against	Firebase GetAccountInfo	The app’s DB
Ordinary requests	External lookup every time	Local JWT verification only

Signature verification stays local. Revocation from logout or BAN is decided by a monotonic boundary held in the app-side DB. Each user carries a value meaning “tokens issued before this instant are invalid,” and comparing it against the token’s issue time is all that’s needed. That value only moves on logout or BAN, so writes are rare.

As a result, outbound communication vanished from the normal request path. The requirement that logout marks the revocation boundary remains exactly as it was.

Spans in Cloud Trace

What made this problem tricky was that the API’s slowness and the SELECT’s speed coexisted. With the API burning 300 milliseconds, staring at EXPLAIN while suspecting the DB yields nothing. The SELECT finishes in 3.6 milliseconds, after all.

OpenTelemetry spans flow from the frontend and backend Cloud Run services into Cloud Trace, letting us follow service boundaries and processing times within a single trace ID. That’s how we knew the outbound POST directly beneath the controller was taking 217 milliseconds. The decisive factor was being able to see exactly how the request’s time was distributed.

Authentication and authorization run on every request. Put one round of external I/O in there, and it lands equally on every endpoint. Speeding up individual endpoints is a different order of magnitude of impact.

---

Based on material presented at the results presentation on July 31, 2026.