Skip to content

Separate issues we fix from issues we file elsewhere - #82

Draft
leggetter wants to merge 3 commits into
mainfrom
feature/evals-triage-proxy-i-od2
Draft

leggetter wants to merge 3 commits into
mainfrom
feature/evals-triage-proxy-i-od2

Conversation

@leggetter

@leggetter leggetter commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

The task

Triage every open issue on hookdeck/evals into work that belongs here and work
that belongs in another repository. That analysis is done; this PR is the part of
it that touches the repo.

What changed

One block in AGENTS.md, under What to work on next, defining the two labels
that say who fixes a thing, the Owned by: line a product issue carries, and
the two shapes that carry product without one.

What changed outside this PR

The label and issue edits leave no diff, so they are listed here rather than
being invisible:

Nothing was filed, closed or commented on, here or on any other repository.

Two ownership questions that needed settling

The triage could not name the owner for #39 and #34 from the issues alone. Both
were checked against hookdeck/outpost at origin/main (6d0c391d, fetched
21 September):

  • The Outpost API answers "Not Found" when your key belongs to a non-Outpost project #39 is hookdeck/core, not hookdeck/outpost. The observed body is
    {"code":"NOT_FOUND","status":404,"message":"Not Found","data":{"team_id":"tm_..."}}.
    Outpost's ErrorResponse serialises as {"status","message","data"}Code is
    an int tagged json:"-", so no string code reaches the wire — its not-found
    messages are lowercase and resource-named, and team_id appears nowhere in that
    repository. The 404 is emitted above Outpost, by the layer that resolves a project
    API key.
  • The operator events API is undocumented: absent from both OpenAPI definitions and from the docs #34 splits. internal/apirouter/router.go registers no /operator-events/*
    route and internal/opevents/ is emit-only, so the implementation is managed-layer
    and item 3 belongs to hookdeck/core. The definition is still Outpost's:
    docs/apis/openapi.yaml lists https://api.outpost.hookdeck.com/2025-07-01 as its
    first server and already documents the managed-only /config endpoints.

While there: #32 understates itself. Three files on outpost@main carry
OPERATION_EVENTS_*features/operator-events.mdoc (16),
self-hosting/changelog/upgrade-v0.16.mdoc (9) and upgrade-v1.1.mdoc (2) — and the
upgrade page has moved under changelog/ since the issue was written. Its Owned by:
line records this.

Review findings, and what each one got (d159412)

Since the review (4eee3fe)

#2 was closed, which took out one of the two worked examples. It was the open
question about our own skills, and its premise — that skills make the weak model
worse — is not supported by any run since 13 August: recomputed on the three
September runs where both arms ran the same day, the weak model is +4 with
skills
, all of it on Outpost. The live half of that question is now #61 and #83.

So the paragraph keeps #27 and describes the other shape without citing an issue,
and the history sentence no longer forward-references it. --label product now
returns eight, and #27 is the only one without a line.

Not changed: the block cites #43 and #34 by number and those sentences dangle
when the issues close. Worked examples are worth the risk here — the Owned by:
rule is hard to follow from an abstract description, and the alternative is the
per-scenario status table this file already warns against. The label description
for product still reads "the fix lands in another repository" with no exception;
a one-line picker string cannot carry the exception, and the file it points at now
does.

Verified

Re-run after 4eee3fe:

pnpm format:check   biome: checked 148 files, no fixes applied; prettier: all matched files
pnpm typecheck      framework + web, clean
pnpm check          20 test files, 227 tests, all passed
pnpm -r build       done

Nothing was already failing. pnpm install needed /opt/homebrew/bin ahead of the
asdf shims on PATH, because the repo has no .tool-versions and the shim has no
pnpm version set — unrelated to this change, and worth knowing before running the
checks locally.

Not done, deliberately

Could not check

Whether the managed operator-events endpoints live in hookdeck/core or in a private
managed service. The public repo settles that they are not in open-source Outpost; it
cannot say which private repo has them, so #34's second line names hookdeck/core as
the best available answer rather than a verified one.

leggetter and others added 3 commits September 21, 2026 19:26
`finding` held both product facts and defects in our own instrument, near
enough evenly, which is the conflation the release-notes rules already
guard against. It is now `product`, and the three instrument defects it
carried moved to `harness`.

A `product` issue carries an `Owned by:` line naming the repository and
the title the issue would take there, because the label says the fix is
elsewhere and only that line says where.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine issues carry `product` and two of them — #2 and #27 — have neither an
`Owned by:` line nor a fix in another repository, so the rule as written
reported two issues as drift on the day it was written. Both shapes are
real and worth naming: a question about our own skills does not know its
repository until it has an answer, and a mapping issue already links the
change it is waiting to measure.

The definition also claimed a `product` issue is the record that the
benchmark found the thing, which #43 opens by denying, and the history
sentence recorded three issues moved rather than the five the triage found.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#2 was closed as its premise: the weak model is not worse with skills on any
run since 13 August, and the live half of the question is now #61 and #83. It
was one of the two worked examples for a `product` issue with no `Owned by:`
line, so the paragraph now describes that shape without citing it and keeps
#27, which is still open and still waiting on a run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant