Skip to content

Fix the failure-campaign leak-baseline flake; bump nltk and transformers - #107

Merged
deepfates merged 7 commits into
mainfrom
claude/deps-and-flaky
Sep 16, 2026
Merged

deepfates merged 7 commits into
mainfrom
claude/deps-and-flaky

Conversation

@deepfates

@deepfates deepfates commented Sep 15, 2026

Copy link
Copy Markdown
Owner

What changed

1. test/failure_campaign_test.exs:25 flake — cause fixed in bench/imp/benchmark_truth/failure_campaign.ex

The leak accounting compared a runtime snapshot taken after prepare_runtime/1 (which only started the :ex_mcp application) with one taken after the lanes ran. The MCP lane performs the VM's first real HTTP round trip through a local Plug.Cowboy server, and that first round trip lazily starts two VM-lifetime services:

  • Plug.Cowboy.http/3 attaches the :plug_cowboy telemetry handler on first use and never detaches it (added_telemetry_handlers => 1);
  • the kernel starts inet_gethost_native under kernel_safe_sup on the first hostname lookup: inet_gethost_native_sup + inet_gethost_native + its inet_gethost OS port (added_processes => 2, added_ports => 1).

Neither is a per-iteration leak, but when this campaign was the VM's first HTTP user both were counted, leak_free went false, and deterministic_complete went false with every lane passing — the exact shape in both CI failures (728f8c7 on main, 507a6ef on a PR). Whether the campaign was the first HTTP user depended on which test in the file (or suite) ran the campaign first, i.e. on the ExUnit seed. The settle_runtime test added at 507a6ef could not help: these services never go away.

Fix: prepare_runtime now runs every deterministic lane once (results discarded) before the baseline snapshot, and records the warmed lane ids in the artifact under configuration.runtime_warmup.warmed_lanes; the test asserts they equal scope. Identified by instrumenting a leaking run and printing Process.info/Port.info/handler ids for the diff, not by inspection.

2. Dependency advisories

  • benchmarks/requirements-ifbench-parity.txt: nltk 3.10.0 → 3.10.3 (closes 17 open dependabot alerts; GHSA-8mgp-746c-j5xp has no patched release and stays open).
  • priv/trl_worker/pyproject.toml + uv.lock: transformers 5.5.0 → 5.10.1 via uv lock --upgrade-package transformers (only transformers moved; closes GHSA-xrqw-3rrv-vx5w for both manifests). Resolved with CPython 3.12.8; the worker itself (torch runtime) was not executed here — mix check excludes protocol_training.
  • accelerate (GHSA-4j2p-28q2-5m79, <= 1.14.0) has no patched release. 1.15.0 exists on PyPI but its release notes do not mention the weight_map path-traversal fix, so bumping it would only move outside the alert's range without fixing anything; left at 1.14.0.
  • cowlib / mix quality.check: not fixable by a bump. cowlib 2.20.0 is the newest release on Hex (2026-09-08). The three ERLEF advisories (EEF-CVE-2026-43971/43966/43969) carry open-ended Hex ranges (introduced 2.9.0, no fixed) that list every published release including 2.20.0, so mix deps.update cowlib cowboy ranch plug_cowboy cowboy_telemetry reports everything unchanged. Cross-checks: the fix commit for 43971 (89da27ee) is contained in the 2.20.0 tag (compare/2.20.0...89da27ee = behind); GitHub's advisory DB records 43966 as fixed in cowboy 2.16.0 (lock has 2.19.0) and 43969 as affecting cowlib <= 2.16.1 (lock has 2.20.0), which is why dependabot raises no mix.lock alert. cowboy cannot leave the lock because the pinned ex_mcp (7222f0f5, untouched) requires plug_cowboy ~> 2.7 non-optionally. mix hex.audit has no ignore mechanism, so the quality job stays red until ERLEF closes those ranges; making it green would mean changing the gate (e.g. mix_audit with a reasoned ignore list), which is a policy decision I did not take in this PR.

No Elixir-side dependabot alerts were open; mix.lock is unchanged.

Verification

  • Flake loop for i in $(seq 20); do mix test test/failure_campaign_test.exs || echo FAIL $i; done: before 9/20 failing (runs 4, 6, 10, 11, 12, 13, 14, 18, 19; each 7 tests, 1 failure, 5.2s because the settle loop waited its full 2s), after 0/20 failing (each 7 tests, 0 failures, 2.4–2.6s).
  • mix check: 54 doctests, 9 properties, 2858 tests, 0 failures, 13 skipped (261 excluded), Finished in 152.5 seconds.
  • mix quality.check: credo 14577 mods/funs, found no issues.; mix hex.audit still exits 1 on the three cowlib 2.20.0 advisories above (unchanged from main).

Dependabot PRs this supersedes

Already obsolete on main independent of this PR: #92 (setuptools → 83.0.0; main already pins 83.0.0), #93 (bandit → 1.12.4; main's lock is at 1.12.5). Not covered here: #94 req_llm, #95 req, #89/#90/#91 GitHub Actions.

Quality gate

mix quality.check is green. The advisory step is now mix deps.audit ({:mix_audit, "~> 2.1", only: [:dev, :test], runtime: false}), whose database records a fixed range per advisory; mix hex.audit stays in the gate for retired packages, which mix_audit does not check. Correction to the section above: mix hex.audit does have an ignore mechanism — hex: [ignore_advisories: [...]] in the project config (mix help hex.audit, Hex 2.5.1), and the pinned ex_mcp fork already uses it for these same three ids. Both steps read one file, .audit_ignore, which holds each id with what was verified and what retires it; mix.exs feeds the same ids to hex.audit via audit_ignored_advisory_ids/0. Emptying .audit_ignore turns the gate red again (exit 1), so the ignores are exact and the gate still fails on anything not listed.

Two of the three are not false positives. Verified 2026-09-15 against the ERLEF CNA records (https://api.osv.dev/v1/vulns/<id>), the GitHub Advisory Database (gh api /advisories?cve_id=...) and the cowlib/cowboy sources at the locked versions:

  • EEF-CVE-2026-43971 — fixed in what we ship. git merge-base --is-ancestor 89da27ee 2.20.0 in ninenines/cowlib is true, and the ERLEF record's own GIT range ends at that commit and lists affected versions only up to 2.19.0. Only its open-ended Hex range still matches. GHSA-gg23-fwhr-prjh carries no affected package ranges at all.
  • EEF-CVE-2026-43966not fixed in cowlib. At tag 2.20.0 cow_http_struct_hd:escape_string/2 still copies every byte other than \ and " through verbatim, and the ERLEF GIT range has no fixed event. The fix shipped in cowboy (commit f77cb9b5, "Add invalid_response_headers HTTP/1 option"; GitHub: cowboy < 2.16.0, first patched 2.16.0). We pin cowboy 2.19.0, whose validate_response_headers/2 defaults invalid_response_headers to error_terminate and refuses any response header value containing CR or LF.
  • EEF-CVE-2026-43969not fixed in cowlib, and no fixed release exists anywhere. cow_cookie:cookie/1 at tag 2.20.0 is still the unvalidated function; the ERLEF GIT range has no fixed event; the only patch in existence is erlef/cowlib 177953dd ("Preliminary patch for CVE-2026-43969") on a fork upstream has not merged. GitHub's record disagrees with the source — it bounds the range at cowlib <= 2.16.1 with no patched version, which is why mix_audit does not flag 2.20.0. The function builds an outgoing Cookie request header, so its callers are HTTP clients such as gun, which is not in this tree; nothing in deps/ outside cowlib calls it, and imp speaks HTTP through Req/Finch/Mint.

So the last two are ignored because no upgrade removes them and the reachable mitigation is a layer up, not because they are wrong. test/dependency_advisory_mitigation_test.exs (new, 3 tests) locks every claim the ignore file makes: cowboy at or above 2.16.0; :cowboy_http.validate_response_headers/2 returning :error_terminate on CR/LF under default options (and :ok when the option is set to ignore, so the test measures the default rather than a constant); Plug.Conn.put_resp_header/3 raising Plug.Conn.InvalidHeaderError on the same bytes; and no module of the 636 in the :imp application importing cow_cookie:cookie/1, cow_link:link/1 or anything in cow_http_struct_hd (2036 distinct remote calls scanned from the BEAM imports chunks).

Reachability, for the record: cowlib and cowboy enter only through ex_mcp, which imp declares runtime: false (its pin 7222f0f5 is untouched). imp's own Plug.Cowboy call sites are tests, bench and lib/mix/tasks/imp_acp.demo_mcp_http_server.ex, none of which ship in the Hex package.

decisions.md line to carry over (the file is not on main yet — it arrives with #106, so it could not be edited here):

| 2026-09-15 | mix quality.check audits dependencies with mix_audit (mix deps.audit --ignore-file .audit_ignore); mix hex.audit stays only to fail on retired packages, with the same ignore list. | mix.exs quality.check, .audit_ignore, PR #107. The ERLEF feed's three cowlib records have no fixed version, so hex.audit alone could never go green by any dependency change; mix_audit's database records fixed ranges. Each ignored id carries what was verified and its retirement condition, locked by test/dependency_advisory_mitigation_test.exs. | In force. | Each ignore retires when its advisory gains a fixed range (43971: when ERLEF records fixed: 2.20.0; 43966 and 43969: when cowlib publishes a release carrying the fix, at which point the lock moves to it). |

Verification

  • mix quality.check: exit 0. credo 14579 mods/funs, found no issues.; mix deps.audit No vulnerabilities found.; mix hex.audit prints the three under Ignored advisories: and exits 0.
  • Falsified: with .audit_ignore emptied, mix quality.check exits 1 again.
  • mix check: exit 0, 54 doctests, 9 properties, 2861 tests, 0 failures, 13 skipped (261 excluded), Finished in 120.2 seconds.

The first real HTTP round trip in a VM lazily starts services that then
live for the rest of the VM: Plug.Cowboy attaches its :plug_cowboy
telemetry handler on first use and the kernel starts inet_gethost_native
(two processes and one port) on the first hostname lookup. The campaign
only started the :ex_mcp application before snapshotting, so whenever it
was the VM's first HTTP user those showed up as added_processes=2,
added_ports=1, added_telemetry_handlers=1 and deterministic_complete went
false. Whether that happened depended on which test in the file ran the
campaign first, i.e. on the ExUnit seed (7 of 20 local runs failed).

Run each deterministic lane once inside prepare_runtime, discard the
results, take the baseline afterwards, and record the warmed lanes in the
artifact's runtime_warmup.
Closes the seventeen open dependabot alerts with a patched release
(GHSA-8mgp-746c-j5xp has no patched release and stays open).
Relocked with uv lock --upgrade-package transformers; only transformers
moved. Closes GHSA-xrqw-3rrv-vx5w for pyproject.toml and uv.lock.
mix hex.audit serves the ERLEF feed, where the three open cowlib records
have no fixed version (introduced 2.9.0, open-ended), so it flagged every
cowlib release that exists and quality.check could not go green by any
dependency change. The advisory step is now mix deps.audit, whose database
records a fixed range per advisory; hex.audit stays for retired packages,
which mix_audit does not check.

.audit_ignore holds the three ids with what was verified for each and the
condition that retires it. Both steps read that one file. Two of the three
are not false positives: cowlib 2.20.0 still ships the unfixed
cow_http_struct_hd:escape_string/2 and cow_cookie:cookie/1, and no fixed
release exists. They are ignored because the mitigation is elsewhere, and
test/dependency_advisory_mitigation_test.exs locks that: cowboy is at or
above 2.16.0 and refuses CR/LF in response headers by default, Plug raises
on the same bytes, and no module in the imp application calls the cowlib
functions under advisory.
@deepfates
deepfates merged commit 72c60da into main Sep 16, 2026
10 checks passed
@deepfates
deepfates deleted the claude/deps-and-flaky branch September 16, 2026 00:20
deepfates added a commit that referenced this pull request Sep 16, 2026
The constellation's one ex_mcp ref moves from 7222f0f5 to b6331871.
Two fork commits land with it: #7 teaches OwnedProcess to treat a child
that already exited as stopped during close, and #8 stops git_hooks from
reinstalling itself on every mix run.

#7 is the one imp needed. The {:error, :no_process} MatchError race made
test/acp_imp_acp_test.exs:1628 and test/acp_local_test.exs:64 flaky in CI.
Those two files ran five times in a row on this ref: 5/5 green, 57 tests
and 0 failures each time.

The two example projects build against the local imp in CI
(IMP_PATH=../..), so their locks follow the same ref. decisions.md names
the pin, so its sha moves too; the decision itself is unchanged.

mix check: 54 doctests, 9 properties, 2858 tests, 0 failures, 13 skipped.
mix quality.check: credo clean, mix hex.audit still red on the three
cowlib 2.20.0 advisories that PR #107 fixes. Untouched here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant