Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/benchmark-calibration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,7 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
run: |
bun run bench:verify-generations -- \
--screen-profile ../provisional-models.json \
--result "${RUNNER_TEMP}/verified-slots/01/report.json" \
--receipt "${RUNNER_TEMP}/verified-slots/01/receipt.json" \
--result "${RUNNER_TEMP}/verified-slots/02/report.json" \
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,7 @@ jobs:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
run: |
bun run bench:verify-generations -- \
--screen-profile ../provisional-models.json \
--result "${RUNNER_TEMP}/bench-live-reports/slots/01/report.json" \
--receipt "${RUNNER_TEMP}/bench-live-reports/slots/01/receipt.json" \
--result "${RUNNER_TEMP}/bench-live-reports/slots/02/report.json" \
Expand Down
9 changes: 5 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,10 +210,11 @@ acquire diff --> parse supported lockfiles --> parse + index --> bounded evidenc
An operator can activate the embedded provisional roster with
`POSTIL_PROVISIONAL_HOSTED_ROSTER=1` while the formal admission manifest is
empty. `provisional-models.json` fixes the managed endpoint, upstream provider,
model chains, consensus, and price ceilings. The release verifier requires that
profile to match `config.toml`, and the runtime applies the same provider pin,
privacy policy, response-identity checks, and operation cost cap used by an
admitted roster. Removing the flag restores the attested-profile requirement.
model chains, canonical provider generation identities, consensus, and price
ceilings. The release verifier requires that profile to match `config.toml`,
and the runtime applies the same provider pin, privacy policy,
response-identity checks, and operation cost cap used by an admitted roster.
Removing the flag restores the attested-profile requirement.

## Prompt-injected policy sources

Expand Down
7 changes: 4 additions & 3 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -468,9 +468,10 @@ canonical slot directory and attests the running receipt before inference,
then attests its completed report and receipt together. Every accepted provider
response contributes its OpenRouter generation ID to the report. The fan-in job
verifies globally distinct generation IDs against OpenRouter's authenticated
generation API, including the exact model, provider, token totals, and cost,
then verifies every subject against the exact repository, release workflow,
source commit, tag ref, OIDC issuer, and GitHub-hosted runner before parsing it.
generation API, including the exact canonical provider model pinned for each
logical profile model, provider, token totals, and cost. It also verifies every
subject against the exact repository, release workflow, source commit, tag ref,
OIDC issuer, and GitHub-hosted runner before parsing it.
Only the unique first workflow run for
the version tag is authoritative. Tag-scoped concurrency, an existing-release
check, and duplicate-run rejection prevent a second publisher path. A failed
Expand Down
2 changes: 1 addition & 1 deletion bench/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"schemaVersion": 2,
"corpus": {
"fixtureCorpusSha256": "8e4c2cb9ad5a7efdfe6a875566d20133e905155b6f693a873595adf6c069e065",
"evaluatorSha256": "00b424fcc4fb13a1c4bd4010ef79a992d0692d59ceb70eae0669ea9823e6645e"
"evaluatorSha256": "b3c6d4085f173ef3abf280b38d4e4221bd39b7752cfbe210e5c19706b700b7e2"
},
"profiles": {
"z-ai/glm-5.2": {
Expand Down
115 changes: 113 additions & 2 deletions bench/src/generation-evidence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ function sample(
runId: "fixture-run",
ranAt: "2026-08-26T12:00:30.000Z",
model: "openai/gpt-5.6-luna",
scorerMode: "disabled",
scorerModel: null,
screeningProfileSha256: "c".repeat(64),
upstreamProviderIdentity: "Azure",
totalTokens: { prompt: 30, completion: 12, total: 42 },
observedProviderCostUsdDecimal: "0.0042",
Expand Down Expand Up @@ -55,7 +58,7 @@ const records = {
"gen-one": {
id: "gen-one",
created_at: "2026-08-26T12:00:10.000Z",
model: "openai/gpt-5.6-luna",
model: "openai/gpt-5.6-luna-20260709",
provider_name: "Azure",
tokens_prompt: 10,
tokens_completion: 5,
Expand All @@ -64,18 +67,26 @@ const records = {
"gen-two": {
id: "gen-two",
created_at: "2026-08-26T12:00:20.000Z",
model: "openai/gpt-5.6-luna",
model: "openai/gpt-5.6-luna-20260709",
provider_name: "Azure",
tokens_prompt: 20,
tokens_completion: 7,
total_cost: 0.0027,
},
};

const profile = {
sha256: "c".repeat(64),
providerGenerationModels: {
"openai/gpt-5.6-luna": "openai/gpt-5.6-luna-20260709",
},
};

describe("provider generation evidence", () => {
test("verifies distinct generation identity, route, tokens, and cost", async () => {
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).resolves.toBe(2);
});
Expand All @@ -86,6 +97,7 @@ describe("provider generation evidence", () => {
sample(["gen-one", "gen-two"]),
], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).rejects.toThrow("duplicate provider generation IDs");
});
Expand All @@ -95,20 +107,116 @@ describe("provider generation evidence", () => {
sample(["gen-one", "gen-two"], { totalTokens: { prompt: 31, completion: 12, total: 43 } }),
], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).rejects.toThrow("token totals do not match provider generations");
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch({
...records,
"gen-two": { ...records["gen-two"], provider_name: "Other" },
}),
})).rejects.toThrow("generation from another provider");
});

test("binds the logical alias to the exact provider generation model", async () => {
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch({
...records,
"gen-two": { ...records["gen-two"], model: "openai/gpt-5.6-luna-20260801" },
}),
})).rejects.toThrow("generation for another model");

await expect(verifyGenerationEvidence([
sample(["gen-one", "gen-two"], { screeningProfileSha256: "d".repeat(64) }),
], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).rejects.toThrow("does not match its screening profile");
});

test("binds each report to only its generator and scorer identities", async () => {
const multipleModels = {
sha256: "c".repeat(64),
providerGenerationModels: {
"logical/generator": "provider/generator-20260801",
"logical/other": "provider/other-20260801",
"logical/scorer": "provider/scorer-20260801",
},
};
const generatorRecord = {
...records["gen-one"],
model: "provider/generator-20260801",
};
const otherRecord = {
...records["gen-two"],
model: "provider/other-20260801",
};
const generatorReport = sample(["gen-one", "gen-two"], {
model: "logical/generator",
});
await expect(verifyGenerationEvidence([generatorReport], {
apiKey: "fixture",
profile: multipleModels,
fetchImpl: generationFetch({
"gen-one": generatorRecord,
"gen-two": otherRecord,
}),
})).rejects.toThrow("generation for another model");

const scorerRecord = {
...records["gen-two"],
model: "provider/scorer-20260801",
};
await expect(verifyGenerationEvidence([
sample(["gen-one", "gen-two"], {
model: "logical/generator",
scorerMode: "enabled",
scorerModel: "logical/scorer",
}),
], {
apiKey: "fixture",
profile: multipleModels,
fetchImpl: generationFetch({
"gen-one": generatorRecord,
"gen-two": scorerRecord,
}),
})).resolves.toBe(2);
});

test("rejects ambiguous provider identity maps", async () => {
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile: {
sha256: "c".repeat(64),
providerGenerationModels: {
"logical/one": "provider/shared-20260801",
"logical/two": "provider/shared-20260801",
},
},
fetchImpl: generationFetch(records),
})).rejects.toThrow("must not repeat canonical models");

await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile: {
sha256: "c".repeat(64),
providerGenerationModels: {
"openai/gpt-5.6-luna": "openai/gpt-5.6-luna",
},
},
fetchImpl: generationFetch(records),
})).rejects.toThrow("must be distinct from logical model IDs");
});

test("rejects a lookup whose returned generation identity differs", async () => {
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch({
...records,
"gen-two": { ...records["gen-two"], id: "gen-one" },
Expand All @@ -119,6 +227,7 @@ describe("provider generation evidence", () => {
test("binds every generation to the attested receipt interval", async () => {
await expect(verifyGenerationEvidence([sample(["gen-one", "gen-two"])], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch({
...records,
"gen-two": { ...records["gen-two"], created_at: "2024-01-01T00:00:00.000Z" },
Expand All @@ -131,13 +240,15 @@ describe("provider generation evidence", () => {
sample(["gen-one", "gen-two"], {}, { reportRawSha256: "b".repeat(64) }),
], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).rejects.toThrow("does not match its receipt digest");

await expect(verifyGenerationEvidence([
sample(["gen-one", "gen-two"], {}, { runId: "another-run" }),
], {
apiKey: "fixture",
profile,
fetchImpl: generationFetch(records),
})).rejects.toThrow("does not match its receipt run identity");
});
Expand Down
Loading