Fix the failure-campaign leak-baseline flake; bump nltk and transformers - #107
Merged
Merged
Conversation
The first real HTTP round trip in a VM lazily starts services that then live for the rest of the VM: Plug.Cowboy attaches its :plug_cowboy telemetry handler on first use and the kernel starts inet_gethost_native (two processes and one port) on the first hostname lookup. The campaign only started the :ex_mcp application before snapshotting, so whenever it was the VM's first HTTP user those showed up as added_processes=2, added_ports=1, added_telemetry_handlers=1 and deterministic_complete went false. Whether that happened depended on which test in the file ran the campaign first, i.e. on the ExUnit seed (7 of 20 local runs failed). Run each deterministic lane once inside prepare_runtime, discard the results, take the baseline afterwards, and record the warmed lanes in the artifact's runtime_warmup.
Closes the seventeen open dependabot alerts with a patched release (GHSA-8mgp-746c-j5xp has no patched release and stays open).
Relocked with uv lock --upgrade-package transformers; only transformers moved. Closes GHSA-xrqw-3rrv-vx5w for pyproject.toml and uv.lock.
mix hex.audit serves the ERLEF feed, where the three open cowlib records have no fixed version (introduced 2.9.0, open-ended), so it flagged every cowlib release that exists and quality.check could not go green by any dependency change. The advisory step is now mix deps.audit, whose database records a fixed range per advisory; hex.audit stays for retired packages, which mix_audit does not check. .audit_ignore holds the three ids with what was verified for each and the condition that retires it. Both steps read that one file. Two of the three are not false positives: cowlib 2.20.0 still ships the unfixed cow_http_struct_hd:escape_string/2 and cow_cookie:cookie/1, and no fixed release exists. They are ignored because the mitigation is elsewhere, and test/dependency_advisory_mitigation_test.exs locks that: cowboy is at or above 2.16.0 and refuses CR/LF in response headers by default, Plug raises on the same bytes, and no module in the imp application calls the cowlib functions under advisory.
deepfates
added a commit
that referenced
this pull request
Sep 16, 2026
The constellation's one ex_mcp ref moves from 7222f0f5 to b6331871. Two fork commits land with it: #7 teaches OwnedProcess to treat a child that already exited as stopped during close, and #8 stops git_hooks from reinstalling itself on every mix run. #7 is the one imp needed. The {:error, :no_process} MatchError race made test/acp_imp_acp_test.exs:1628 and test/acp_local_test.exs:64 flaky in CI. Those two files ran five times in a row on this ref: 5/5 green, 57 tests and 0 failures each time. The two example projects build against the local imp in CI (IMP_PATH=../..), so their locks follow the same ref. decisions.md names the pin, so its sha moves too; the decision itself is unchanged. mix check: 54 doctests, 9 properties, 2858 tests, 0 failures, 13 skipped. mix quality.check: credo clean, mix hex.audit still red on the three cowlib 2.20.0 advisories that PR #107 fixes. Untouched here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
1.
test/failure_campaign_test.exs:25flake — cause fixed inbench/imp/benchmark_truth/failure_campaign.exThe leak accounting compared a runtime snapshot taken after
prepare_runtime/1(which only started the:ex_mcpapplication) with one taken after the lanes ran. The MCP lane performs the VM's first real HTTP round trip through a localPlug.Cowboyserver, and that first round trip lazily starts two VM-lifetime services:Plug.Cowboy.http/3attaches the:plug_cowboytelemetry handler on first use and never detaches it (added_telemetry_handlers => 1);inet_gethost_nativeunderkernel_safe_supon the first hostname lookup:inet_gethost_native_sup+inet_gethost_native+ itsinet_gethostOS port (added_processes => 2,added_ports => 1).Neither is a per-iteration leak, but when this campaign was the VM's first HTTP user both were counted,
leak_freewent false, anddeterministic_completewent false with every lane passing — the exact shape in both CI failures (728f8c7 on main, 507a6ef on a PR). Whether the campaign was the first HTTP user depended on which test in the file (or suite) ran the campaign first, i.e. on the ExUnit seed. Thesettle_runtimetest added at 507a6ef could not help: these services never go away.Fix:
prepare_runtimenow runs every deterministic lane once (results discarded) before the baseline snapshot, and records the warmed lane ids in the artifact underconfiguration.runtime_warmup.warmed_lanes; the test asserts they equalscope. Identified by instrumenting a leaking run and printingProcess.info/Port.info/handler ids for the diff, not by inspection.2. Dependency advisories
benchmarks/requirements-ifbench-parity.txt: nltk 3.10.0 → 3.10.3 (closes 17 open dependabot alerts; GHSA-8mgp-746c-j5xp has no patched release and stays open).priv/trl_worker/pyproject.toml+uv.lock: transformers 5.5.0 → 5.10.1 viauv lock --upgrade-package transformers(only transformers moved; closes GHSA-xrqw-3rrv-vx5w for both manifests). Resolved with CPython 3.12.8; the worker itself (torch runtime) was not executed here —mix checkexcludesprotocol_training.<= 1.14.0) has no patched release. 1.15.0 exists on PyPI but its release notes do not mention the weight_map path-traversal fix, so bumping it would only move outside the alert's range without fixing anything; left at 1.14.0.mix quality.check: not fixable by a bump. cowlib 2.20.0 is the newest release on Hex (2026-09-08). The three ERLEF advisories (EEF-CVE-2026-43971/43966/43969) carry open-ended Hex ranges (introduced 2.9.0, nofixed) that list every published release including 2.20.0, somix deps.update cowlib cowboy ranch plug_cowboy cowboy_telemetryreports everything unchanged. Cross-checks: the fix commit for 43971 (89da27ee) is contained in the 2.20.0 tag (compare/2.20.0...89da27ee= behind); GitHub's advisory DB records 43966 as fixed in cowboy 2.16.0 (lock has 2.19.0) and 43969 as affecting cowlib<= 2.16.1(lock has 2.20.0), which is why dependabot raises no mix.lock alert. cowboy cannot leave the lock because the pinned ex_mcp (7222f0f5, untouched) requiresplug_cowboy ~> 2.7non-optionally.mix hex.audithas no ignore mechanism, so the quality job stays red until ERLEF closes those ranges; making it green would mean changing the gate (e.g.mix_auditwith a reasoned ignore list), which is a policy decision I did not take in this PR.No Elixir-side dependabot alerts were open; mix.lock is unchanged.
Verification
for i in $(seq 20); do mix test test/failure_campaign_test.exs || echo FAIL $i; done: before 9/20 failing (runs 4, 6, 10, 11, 12, 13, 14, 18, 19; each7 tests, 1 failure, 5.2s because the settle loop waited its full 2s), after 0/20 failing (each7 tests, 0 failures, 2.4–2.6s).mix check:54 doctests, 9 properties, 2858 tests, 0 failures, 13 skipped (261 excluded),Finished in 152.5 seconds.mix quality.check: credo14577 mods/funs, found no issues.;mix hex.auditstill exits 1 on the three cowlib 2.20.0 advisories above (unchanged from main).Dependabot PRs this supersedes
Already obsolete on main independent of this PR: #92 (setuptools → 83.0.0; main already pins 83.0.0), #93 (bandit → 1.12.4; main's lock is at 1.12.5). Not covered here: #94 req_llm, #95 req, #89/#90/#91 GitHub Actions.
Quality gate
mix quality.checkis green. The advisory step is nowmix deps.audit({:mix_audit, "~> 2.1", only: [:dev, :test], runtime: false}), whose database records a fixed range per advisory;mix hex.auditstays in the gate for retired packages, whichmix_auditdoes not check. Correction to the section above:mix hex.auditdoes have an ignore mechanism —hex: [ignore_advisories: [...]]in the project config (mix help hex.audit, Hex 2.5.1), and the pinned ex_mcp fork already uses it for these same three ids. Both steps read one file,.audit_ignore, which holds each id with what was verified and what retires it;mix.exsfeeds the same ids tohex.auditviaaudit_ignored_advisory_ids/0. Emptying.audit_ignoreturns the gate red again (exit 1), so the ignores are exact and the gate still fails on anything not listed.Two of the three are not false positives. Verified 2026-09-15 against the ERLEF CNA records (
https://api.osv.dev/v1/vulns/<id>), the GitHub Advisory Database (gh api /advisories?cve_id=...) and the cowlib/cowboy sources at the locked versions:git merge-base --is-ancestor 89da27ee 2.20.0in ninenines/cowlib is true, and the ERLEF record's own GIT range ends at that commit and lists affected versions only up to 2.19.0. Only its open-ended Hex range still matches. GHSA-gg23-fwhr-prjh carries no affected package ranges at all.cow_http_struct_hd:escape_string/2still copies every byte other than\and"through verbatim, and the ERLEF GIT range has no fixed event. The fix shipped in cowboy (commitf77cb9b5, "Add invalid_response_headers HTTP/1 option"; GitHub: cowboy< 2.16.0, first patched 2.16.0). We pin cowboy 2.19.0, whosevalidate_response_headers/2defaultsinvalid_response_headerstoerror_terminateand refuses any response header value containing CR or LF.cow_cookie:cookie/1at tag 2.20.0 is still the unvalidated function; the ERLEF GIT range has no fixed event; the only patch in existence is erlef/cowlib177953dd("Preliminary patch for CVE-2026-43969") on a fork upstream has not merged. GitHub's record disagrees with the source — it bounds the range at cowlib<= 2.16.1with no patched version, which is whymix_auditdoes not flag 2.20.0. The function builds an outgoing Cookie request header, so its callers are HTTP clients such as gun, which is not in this tree; nothing indeps/outside cowlib calls it, and imp speaks HTTP through Req/Finch/Mint.So the last two are ignored because no upgrade removes them and the reachable mitigation is a layer up, not because they are wrong.
test/dependency_advisory_mitigation_test.exs(new, 3 tests) locks every claim the ignore file makes: cowboy at or above 2.16.0;:cowboy_http.validate_response_headers/2returning:error_terminateon CR/LF under default options (and:okwhen the option is set toignore, so the test measures the default rather than a constant);Plug.Conn.put_resp_header/3raisingPlug.Conn.InvalidHeaderErroron the same bytes; and no module of the 636 in the:impapplication importingcow_cookie:cookie/1,cow_link:link/1or anything incow_http_struct_hd(2036 distinct remote calls scanned from the BEAM imports chunks).Reachability, for the record: cowlib and cowboy enter only through ex_mcp, which imp declares
runtime: false(its pin7222f0f5is untouched). imp's ownPlug.Cowboycall sites are tests, bench andlib/mix/tasks/imp_acp.demo_mcp_http_server.ex, none of which ship in the Hex package.decisions.mdline to carry over (the file is not onmainyet — it arrives with #106, so it could not be edited here):| 2026-09-15 |
mix quality.checkaudits dependencies withmix_audit(mix deps.audit --ignore-file .audit_ignore);mix hex.auditstays only to fail on retired packages, with the same ignore list. |mix.exsquality.check,.audit_ignore, PR #107. The ERLEF feed's three cowlib records have no fixed version, sohex.auditalone could never go green by any dependency change;mix_audit's database records fixed ranges. Each ignored id carries what was verified and its retirement condition, locked bytest/dependency_advisory_mitigation_test.exs. | In force. | Each ignore retires when its advisory gains a fixed range (43971: when ERLEF recordsfixed: 2.20.0; 43966 and 43969: when cowlib publishes a release carrying the fix, at which point the lock moves to it). |Verification
mix quality.check: exit 0. credo14579 mods/funs, found no issues.;mix deps.auditNo vulnerabilities found.;mix hex.auditprints the three underIgnored advisories:and exits 0..audit_ignoreemptied,mix quality.checkexits 1 again.mix check: exit 0,54 doctests, 9 properties, 2861 tests, 0 failures, 13 skipped (261 excluded),Finished in 120.2 seconds.