Publish an order topic this project has, not one this file invented - #79
Merged
Merged
Conversation
The 21 September weekly failed. One agent subscribed its destination to `*`,
and the scorer's wildcard branch published the literal `orders.created` —
a topic this project does not have. `GET /topics` returns `orders`, `retries`,
`order.created`, `order.shipped`, `order.updated`, `order.cancelled`. Outpost
answered `422 {"message":"validation error","data":["topic is invalid"]}`, the
scorer threw, and the cell errored instead of scoring.
The consequence was larger than one cell. An errored eval fails its matrix job,
`publish-results` needs every matrix job, so seventy-five cells that had run
fine published nothing. Filed separately.
The topic now comes from the project. A wildcard subscription says nothing
about which topic to publish, so asking is the only correct answer — and the
list is configuration that can change, which a string compiled into a scorer
cannot follow. Prefers a specific `order.*` topic over the bare `orders`,
because a destination subscribed to everything receives it either way and this
scenario is about an order event reaching a customer.
Verified with score-only on the case that broke the run: `SOLUTION.ts`
temporarily set to `topics: '*'` reproduces the wildcard path, which now
passes. The ordinary solution path passes unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
Reaching the no-topic branch has two causes and only one of them is the agent's: a destination subscribed to nothing order-shaped, or a project with no order topic to publish. The note claimed the first for both, so a harness misconfiguration would have been published as an agent failure — which is the mistake this scorer has now made twice, once by counting a delivery attempt as a delivery and once by inventing a topic the project does not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the 21 September weekly failed.
What happened
One agent —
codex-gpt-5.4-mini-no-skills— subscribed its destination to*. The scorer's wildcard branch then published the literal stringorders.created:That topic does not exist.
GET /topicson this project returns:So Outpost answered
422 {"status":422,"message":"validation error","data":["topic is invalid"]}, the scorer threw, and the cell errored rather than scoring. Pluralorders.createdagainst singularorder.created— the seeds in every other Outpost scenario use the singular.The blast radius was the whole run
An errored eval fails its matrix job.
publish-resultshasneeds: [prepare, run-evals], so it was skipped. Seventy-five cells that ran fine published nothing, because of one wildcard subscription. Filed as a separate issue.The fix
The topic comes from the project. A wildcard subscription says nothing about which topic to publish, so asking is the only correct answer, and the list is configuration that can change while a string compiled into a scorer cannot follow it. A specific
order.*topic is preferred over the bareorders: a destination subscribed to everything receives either, and this scenario is about an order event reaching a customer.Verified against the case that broke it
score-onlywithSOLUTION.tstemporarily set totopics: '*', which reproduces the wildcard path exactly — passes. The ordinary solution path passes unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK