Conversation
`finding` held both product facts and defects in our own instrument, near enough evenly, which is the conflation the release-notes rules already guard against. It is now `product`, and the three instrument defects it carried moved to `harness`. A `product` issue carries an `Owned by:` line naming the repository and the title the issue would take there, because the label says the fix is elsewhere and only that line says where. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine issues carry `product` and two of them — #2 and #27 — have neither an `Owned by:` line nor a fix in another repository, so the rule as written reported two issues as drift on the day it was written. Both shapes are real and worth naming: a question about our own skills does not know its repository until it has an answer, and a mapping issue already links the change it is waiting to measure. The definition also claimed a `product` issue is the record that the benchmark found the thing, which #43 opens by denying, and the history sentence recorded three issues moved rather than the five the triage found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2 was closed as its premise: the weak model is not worse with skills on any run since 13 August, and the live half of the question is now #61 and #83. It was one of the two worked examples for a `product` issue with no `Owned by:` line, so the paragraph now describes that shape without citing it and keeps #27, which is still open and still waiting on a run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The task
Triage every open issue on
hookdeck/evalsinto work that belongs here and workthat belongs in another repository. That analysis is done; this PR is the part of
it that touches the repo.
What changed
One block in
AGENTS.md, under What to work on next, defining the two labelsthat say who fixes a thing, the
Owned by:line aproductissue carries, andthe two shapes that carry
productwithout one.What changed outside this PR
The label and issue edits leave no diff, so they are listed here rather than
being invisible:
findingrenamed toproduct(gh label edit, so it stayed applied to alleleven issues). Of those eleven, six had a fix in another repository, three were
measurements of our own instrument and two are product-subject with the next step
here.
productnow reads "A fact aboutHookdeck, its docs or its skills; the fix lands in another repository", and
harnessgained "— work that can be done in this repo". That string is whatGitHub shows in the label picker, so it is the definition sitting next to this
one.
harness. Documentation access differing by arm, amerged snapshot not saying it is merged, and confirming the docs-suppression
mechanism are all measurements of our own instrument.
productadded to Hookdeck's webhook notification config cannot be read back, and the natural way to set it returns 200 while doing nothing #43, which had onlydocumentationandscenario.Owned by:line added to Agents wire a placeholder verification secret into a live source, and nothing surfaces it #75, The Outpost skill never says you need a key for an Outpost project #40, The Outpost API answers "Not Found" when your key belongs to a non-Outpost project #39, Outpost docs give the wrong env var name for every operator events setting #32, No signal for when a configuration change is in force #25, The operator events API is undocumented: absent from both OpenAPI definitions and from the docs #34 and Hookdeck's webhook notification config cannot be read back, and the natural way to set it returns 200 while doing nothing #43, naming therepository and the title the issue would take there.
Nothing was filed, closed or commented on, here or on any other repository.
Two ownership questions that needed settling
The triage could not name the owner for #39 and #34 from the issues alone. Both
were checked against
hookdeck/outpostatorigin/main(6d0c391d, fetched21 September):
hookdeck/core, nothookdeck/outpost. The observed body is{"code":"NOT_FOUND","status":404,"message":"Not Found","data":{"team_id":"tm_..."}}.Outpost's
ErrorResponseserialises as{"status","message","data"}—Codeisan
inttaggedjson:"-", so no stringcodereaches the wire — its not-foundmessages are lowercase and resource-named, and
team_idappears nowhere in thatrepository. The 404 is emitted above Outpost, by the layer that resolves a project
API key.
internal/apirouter/router.goregisters no/operator-events/*route and
internal/opevents/is emit-only, so the implementation is managed-layerand item 3 belongs to
hookdeck/core. The definition is still Outpost's:docs/apis/openapi.yamllistshttps://api.outpost.hookdeck.com/2025-07-01as itsfirst server and already documents the managed-only
/configendpoints.While there: #32 understates itself. Three files on
outpost@maincarryOPERATION_EVENTS_*—features/operator-events.mdoc(16),self-hosting/changelog/upgrade-v0.16.mdoc(9) andupgrade-v1.1.mdoc(2) — and theupgrade page has moved under
changelog/since the issue was written. ItsOwned by:line records this.
Review findings, and what each one got (
d159412)productissues carry noOwned by:line (major). Confirmed:--label productreturned nine, and Why do our skills make the weak model worse? #2 and Does documenting hookdeck ci stop agents giving up on CLI authentication? #27 had no line and no fix elsewhere.Fixed in the file rather than on the issues — a new paragraph names both shapes.
A mapping issue links the change it is waiting to measure, which is what the
Owned by:line becomes once the issue exists (Does documenting hookdeck ci stop agents giving up on CLI authentication? #27 links the mergedhookdeck/agent-skillspull request in its first line); an open question aboutour own skills or docs cannot name a repository until it has an answer. Anything
else with
productand no line is drift, and the paragraph says so.productwhile its own body says no run produced it (minor).Confirmed. "the issue here is the record that the benchmark found it" is gone.
A new sentence says the label is about where the fix goes, not where the finding
came from, and uses Hookdeck's webhook notification config cannot be read back, and the natural way to set it returns 200 while doing nothing #43 as the case.
of the eleven were measurements of our own instrument, three moved to
harness,and the other two are the exception. Also takes the review's wording for Confirm the docs-suppression mechanism rather than inferring it #10 —
"measurements of our own instrument" rather than "defects".
only
scenarioappears on both; all threedocumentationissues areproductand both
publishingissues are harness-side. Reworded as a statement about whatthe labels mean, not about the current board.
Since the review (
4eee3fe)#2 was closed, which took out one of the two worked examples. It was the open
question about our own skills, and its premise — that skills make the weak model
worse — is not supported by any run since 13 August: recomputed on the three
September runs where both arms ran the same day, the weak model is +4 with
skills, all of it on Outpost. The live half of that question is now #61 and #83.
So the paragraph keeps #27 and describes the other shape without citing an issue,
and the history sentence no longer forward-references it.
--label productnowreturns eight, and #27 is the only one without a line.
Not changed: the block cites #43 and #34 by number and those sentences dangle
when the issues close. Worked examples are worth the risk here — the
Owned by:rule is hard to follow from an abstract description, and the alternative is the
per-scenario status table this file already warns against. The label description
for
productstill reads "the fix lands in another repository" with no exception;a one-line picker string cannot carry the exception, and the file it points at now
does.
Verified
Re-run after
4eee3fe:Nothing was already failing.
pnpm installneeded/opt/homebrew/binahead of theasdf shims on
PATH, because the repo has no.tool-versionsand the shim has nopnpmversion set — unrelated to this change, and worth knowing before running thechecks locally.
Not done, deliberately
block now describes the shape in prose. Worth revisiting when there is a second.
AGENTS.md; nothing here is executable.adding
Owned by:lines to Why do our skills make the weak model worse? #2 and Does documenting hookdeck ci stop agents giving up on CLI authentication? #27 instead. Neither has an owner elsewhere toname, so the line would have been invented; the exception belongs in the file.
hookdeck/outpost,hookdeck/coreorhookdeck/agent-skills.Each
Owned by:line says what to file and where, and the filing is a human decision.Could not check
Whether the managed operator-events endpoints live in
hookdeck/coreor in a privatemanaged service. The public repo settles that they are not in open-source Outpost; it
cannot say which private repo has them, so #34's second line names
hookdeck/coreasthe best available answer rather than a verified one.