Skip to content

fix(relay): anti-enum tests blew their budget on harness overhead, not a hang - #13

Open
angusbezzina wants to merge 1 commit into
mainfrom
fix/relay-antienum-test-timing
Open

fix(relay): anti-enum tests blew their budget on harness overhead, not a hang#13
angusbezzina wants to merge 1 commit into
mainfrom
fix/relay-antienum-test-timing

Conversation

@angusbezzina

Copy link
Copy Markdown
Collaborator

Fixes the Relay Tests (Ubuntu) job, which has failed on almost every run since 2026-07-27 on the same anti-enumeration cases in relay/test/integration/rate-limit.test.ts.

The recorded diagnosis was wrong

attn-i38g had these down as hanging only under GitHub Actions, on the grounds that they finish in ~350ms locally, and explicitly said not to raise the budgets until the hang was explained — if the limiter can stall a request path, a longer timeout would hide a production bug. That was the right instinct, so the first job was to test it.

There is no hang, and nothing is Actions-specific. The ~350ms figure came from running the file on its own. Run the whole integration suite and the same two cases take 19,660ms and 9,216ms on macOS.

The cost is the SELF.fetch round trip, not anything under test:

request file alone under suite load
GET unknown room 1.6ms ~250ms
GET /health — no DO, no quota, no rate limiter ~217ms

A no-op route is the same price as a limiter-exercising one, so the overhead is per-round-trip in the shared workerd isolate and scales with (sequential calls) x (suite activity). These four cases make the most sequential calls in the repo, because the cap (antiEnumPerFiveMin = 30) can only be exercised by 30+ distinct probes. They were simply first over the line — which is also why they were intermittent rather than uniformly broken.

This clears the production concern. The limiter does not stall anything.

Ruled out by measurement

  • File parallelism--no-file-parallelism changed nothing (19,214ms).
  • Accumulated Durable Objects — 500 preloaded DOs left the tests fast (452ms).
  • One bad neighbour file — conformance, alarms, share-watch, share-lifecycle, quota and v3-admission each pair fine; either half of the integration set pairs fine; only the full set reproduces.
  • Concurrency as the fixPromise.all over 20 probes measured 1.0x against the sequential loop, both loaded and idle. The pool serializes SELF.fetch regardless. This killed the obvious fix before it shipped.

The change

Only lever left is fewer sequential calls.

  1. The existing rooms known-room loop drops 50 hits to 5. It was ~11s of that case's ~19s (50 round trips plus 50 admission-header signings), and the property under test — a known room never calls recordUnknownRoom — is a per-request branch that holds on the first hit or not at all. Five guards against an accumulator as well as fifty did.
  2. All four cases get an explicit 30s budget. Each is structurally 30+ sequential calls and was marginal at the 15s default, so the two that still passed were next in line. Raising the budget here is the measured response to harness overhead, documented in a block comment, not a way to hide a suspected stall.

Results

Full relay suite: 474/474 passing in 206.79s. tsc --noEmit clean.

case before after
existing rooms do NOT count 19,660ms 8,753ms
probes from a DIFFERENT IP 9,216ms 8,515ms
31 distinct unknown roomIds 7,026ms
unsafe legacy blob paths 6,478ms

~3.4x headroom against the budget on the worst case.

Deliberately not done

Making antiEnumPerFiveMin configurable would be the real speed fix (30 probes becomes ~6), but the limiter is a module-level singleton shared across the isolate, and test files other than this one still share the "unknown" IP bucket — a smaller global cap would make unrelated cases 429. Noted in the code and on the bead.

Refs: attn-i38g

🤖 Generated with Claude Code

…t a hang

The two anti-enumeration cases had been failing 'Relay Tests (Ubuntu)' on
almost every run since 2026-07-27. attn-i38g recorded them as hanging only
under GitHub Actions, on the grounds that they finish in ~350ms locally, and
told us not to touch the budgets until the hang was explained: if the limiter
can stall a request path, a longer timeout would hide a production bug.

There is no hang, and nothing is Actions-specific. The ~350ms figure came from
running the file on its own. Run the whole integration suite and the same two
cases take 19,660ms and 9,216ms on macOS.

The cost is the SELF.fetch round trip, not anything under test. An unknown-room
GET costs 1.6ms when this file runs alone and ~250ms when the suite shares the
workerd isolate. A GET /health -- no Durable Object, no quota, no rate limiter
on that path -- costs ~217ms under the same load, the same price. So it scales
with sequential calls times suite activity, and these four cases make the most
sequential calls in the repo: the cap (antiEnumPerFiveMin=30) can only be
exercised by 30+ distinct probes. They were simply first over the line, which
is also why they were intermittent rather than uniformly broken.

That clears the concern the bead raised. The limiter does not stall anything; a
no-op route is equally slow under the same conditions.

Ruled out by measurement: file parallelism (--no-file-parallelism changed
nothing), accumulated Durable Objects (500 preloaded DOs left the tests fast),
a single bad neighbour file (only the full integration set reproduces), and
concurrency as a fix (Promise.all over 20 probes measured 1.0x against the
sequential loop, loaded and idle -- the pool serializes them regardless).

So the only lever is fewer sequential calls. The 'existing rooms' known-room
loop drops from 50 hits to 5: it was ~11s of that case's ~19s, and the property
under test is a per-request branch that holds on the first hit or not at all,
so five guards against an accumulator as well as fifty did. Worst case under
full-integration load falls 19,660ms -> 8,967ms.

All four cases then get an explicit 30s budget. Every one of them is
structurally 30+ sequential calls and was marginal at the 15s default, so the
two that still passed were next. Raising the budget here is the measured
response to harness overhead, documented in the block comment, not a way to
hide a suspected stall.

Making antiEnumPerFiveMin configurable would be the real speed fix -- 30 probes
becomes ~6 -- but the limiter is a module-level singleton shared across the
isolate, and files other than this one still share the "unknown" IP bucket, so
a smaller global cap would make unrelated cases 429. Left alone deliberately.

Refs: attn-i38g

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
attn Ready Ready Preview Aug 10, 2026 6:06pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant