ci: add a manual firestore flake probe for #776 - #780
Open
tyler-reitz wants to merge 1 commit into
Open
Conversation
Every measurement of the FirebaseExtended#776 flake so far has been local, where the failure is a plain waitFor timeout with no gRPC error. In CI it arrives alongside a gRPC framing desync (RESOURCE_EXHAUSTED: Received message larger than max), which raises the possibility that the local repro and the CI failure are not the same bug. That matters, because the @grpc/grpc-js override proposed as the fix was measured only against the local one. This runs the firestore suite N times per arm, across both Node versions and both grpc-js versions, under CI conditions, so the comparison happens where the failure actually occurs. Notes on the design: - workflow_dispatch only. It never runs on a push, a PR or a schedule, so it costs nothing until someone asks for it. - A fresh emulator per iteration, matching how npm test runs in CI. Reusing one emulator across iterations would measure something else. - Failures are classified, not counted. Only the FirebaseExtended#776 assertion signature counts toward the rate; emulator start failures are reported separately, because folding them in previously inflated a local rate estimate by roughly 50%. - The job reports rather than fails. A red run here means the probe broke, not that the flake reproduced. - Inputs reach the script through env rather than interpolation, and iterations is validated before it reaches the loop. Classifier dry-run against synthetic logs covering pass, flake, flake-with-gRPC-error and infra-failure returns the expected counts and excludes infra failures from the rate. zizmor 1.25.2 reports no findings beyond the cache-poisoning rule CI suppresses. Refs FirebaseExtended#776
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #776. This measures the flake; it does not fix it.
Why
Every measurement of the #776 flake so far has been on a laptop. Locally the failure is a plain
waitFortimeout with no gRPC error at all. In CI it arrives alongside a gRPC framing desync:A ~795 MB claimed frame against a 4 MB cap means the reader is interpreting payload bytes as a length header. That is a
@grpc/grpc-js-layer failure, and it does not appear in any local run.So the local repro and the CI failure may not be the same bug. That matters, because the
@grpc/grpc-jsoverride proposed as the fix was measured only against the local one, at p = 0.12. Merging that override would also mean CI never runs 1.9.16 again, so it is a before-and-after with no control.This workflow makes the comparison in CI, where the failure actually happens, before anything is changed.
What it does
Matrixes Node 22 and 24 against
@grpc/grpc-jsbaseline and override, runs the firestore suite N times per arm (default 20), and writes a table per arm to the job summary: resolved grpc-js version, pass / flake / infra counts, how many runs showedRESOURCE_EXHAUSTED, and the resulting rate.Design decisions worth reviewing:
workflow_dispatchonly. It never runs on a push, a PR, a release or a schedule, so it adds nothing to normal CI time.npm testruns in CI. Reusing one emulator across iterations would measure something else.RESOURCE_EXHAUSTEDis counted per arm. This is the one signal no existing measurement has, and it is what would show that CI is hitting a different bug.envrather than being interpolated into the shell, anditerationsis validated before it reaches the loop.Verification
workflow_dispatchfor workflows present on the default branch, so it is unrunnable until this merges. Per the repo's own lesson about treating jobs authored on a branch as unrun, it was verified out of band instead:cache-poisoningrule CI suppresses, which is the severity set the gate reads.package.jsonoverride edit produces the right key (npm pkg setmangles keys containing a slash, hence thenode -e);npx firebaseresolves throughfirebase-tools.The first real run is therefore the first proof it works. If it misbehaves, the fix is another PR, not a rollback of anything consumers see.
Known confound
The override is repo-wide, so the override arm also moves
@grpc/grpc-jsforfirebase-toolsandgoogle-gax, not only the Firestore client. The emulator itself is a Java process and so should not be in the data path, but the two arms differ by slightly more than the client library. Worth remembering when reading the numbers.What this changes for anyone else
Nothing. One new file, no existing workflow touched, no runtime or packaging change, and no effect on normal CI.