Conversation
Two things in this repo pin the exact req_llm build, and both had to move. `Imp.BenchmarkTruth.MultimodalManifest` carries the campaign's ReqLLM dependency — package hash, upstream revision, version — and `runtime_dependency!/1` refuses to dispatch when the loaded version differs, so the bump failed with `ReqLLM runtime version must equal "1.17.1", got: "1.18.0"`. The pin now names 1.18.0, package SHA-256 d610d4de14c7ef697a2aa4c30eada4e1ae172bec08bb3dea3bbe25f20feb5d40, upstream revision 9aa98a4e02da5083464d2eed268f48272f0d7180 (the commit tagged v1.18.0 in agentjido/req_llm). Both provider manifests carry the same dependency block, so they were updated and their payload checksums recomputed. That changes the manifests' identity. The OpenAI manifest now hashes to 5ab0c61f... instead of 04daa155..., and the Google one to c5ca53e2... instead of 4b4c8049.... The admitted multimodal_live artifact records the 1.17.1 run and keeps its own manifest SHA, so it still validates and the claim still stands; what it no longer is, is reproducible from the manifests in the tree. Admitting a 1.18.0 re-run means a live OpenAI campaign, a new artifact, and a new `multimodal_manifest_sha256/0` in the reproduction validator. MULTIMODAL_FIDELITY.md now says so. `test/campaign_budget_test.exs` asserted that the pinned req_llm overwrites a caller's `max_retries: 0` with its default of 3. 1.18.0 honours the zero, so the assertion is inverted to match and still fails loudly if upstream changes again. The transport guard itself is untouched and the attempt count is unchanged. req stays at 0.6.3 here; dependabot PR #94 moved it to 0.7.2 in the same lock, and that bump needs its own code changes (PR #109). This supersedes dependabot PR #94. mix check: 54 doctests, 9 properties, 2861 tests, 0 failures, 13 skipped (261 excluded). mix quality.check: credo clean, no vulnerabilities found.
Owner
Author
|
Holding this unmerged. The bump changes the multimodal benchmark manifest checksums, so the admitted artifact is no longer reproducible from the tree without a live re-run, and nothing needs req_llm 1.18.0 today. It moves when there is a reason and a budget for the live campaign. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things in this repo pin the exact req_llm build, and both had to move.
Imp.BenchmarkTruth.MultimodalManifestcarries the campaign's ReqLLMdependency — package hash, upstream revision, version — and
runtime_dependency!/1refuses to dispatch when the loaded version differs,so the bump failed with
ReqLLM runtime version must equal "1.17.1", got: "1.18.0". The pin now names 1.18.0, package SHA-256d610d4de14c7ef697a2aa4c30eada4e1ae172bec08bb3dea3bbe25f20feb5d40, upstream
revision 9aa98a4e02da5083464d2eed268f48272f0d7180 (the commit tagged v1.18.0
in agentjido/req_llm). Both provider manifests carry the same dependency
block, so they were updated and their payload checksums recomputed.
That changes the manifests' identity. The OpenAI manifest now hashes to
5ab0c61f... instead of 04daa155..., and the Google one to c5ca53e2...
instead of 4b4c8049.... The admitted multimodal_live artifact records the
1.17.1 run and keeps its own manifest SHA, so it still validates and the
claim still stands; what it no longer is, is reproducible from the manifests
in the tree. Admitting a 1.18.0 re-run means a live OpenAI campaign, a new
artifact, and a new
multimodal_manifest_sha256/0in the reproductionvalidator. MULTIMODAL_FIDELITY.md now says so.
test/campaign_budget_test.exsasserted that the pinned req_llm overwritesa caller's
max_retries: 0with its default of 3. 1.18.0 honours the zero,so the assertion is inverted to match and still fails loudly if upstream
changes again. The transport guard itself is untouched and the attempt count
is unchanged.
req stays at 0.6.3 here; dependabot PR #94 moved it to 0.7.2 in the same
lock, and that bump needs its own code changes (PR #109).
This supersedes dependabot PR #94.
mix check: 54 doctests, 9 properties, 2861 tests, 0 failures, 13 skipped (261 excluded).
mix quality.check: credo clean, no vulnerabilities found.