bt-daemon: mixed-version checkouts share one daemon socket (restart churn + stale translation between upgrade and next hook)

[Iron-Ham]

[written by Claude (AI agent) working with Hesham Salman (@Iron-Ham) on Notion's internal trace analytics]

Correction of the original report

The original version of this issue claimed hooks keep forwarding to a stale daemon after a bt upgrade and asked for a version handshake and span provenance stamping. Both already exist:

• forward_envelope (bt-daemon/src/lib.rs) does the handshake and restarts a version-mismatched daemon (initialize → compare daemon_version → daemon.shutdown → respawn → re-initialize). Present since Move coding-agent tracing into the Rust daemon #1.
• Spans carry queryable provenance: context.span_origin.version (the plugin version when the hook supplies one, otherwise the daemon host build).

Verified against production data: in the last 48h of Notion's internal-coding-agent-traces project, every span matching the bugs fixed by #71/#79 came from pre-0.20.0 emitters — blank tool-span error: 1086 rows on 0.19.1, 106 on 0.19.3, 0 on 0.20.0; rejection-recovered-as-approved: 4 on 0.19.3, 3 on 0.19.1, 0 on 0.20.0. The fixes work; rollout is the only residual.

What actually remains (observed)

A developer machine with several repo checkouts on different bt versions (e.g. one workspace pinning 0.19.3, another 0.20.0) shares a single daemon socket (~/.braintrust/run/daemon.sock). Every hook event from the other version triggers a shutdown+respawn of the daemon. Consequences:

1. Between a CLI upgrade and the next hook event, the old daemon keeps serving (bounded, but real).
2. With concurrent agents running from mixed-version workspaces, the daemon can be shut down mid-flight out from under another version's active sessions, repeatedly. Observed three 0.19.3 daemon processes across two days on one machine, plus a 0.20.0 daemon that lost the socket race.
3. SinkFactory stamps span_origin.version with the plugin-reported version when present (plugin_version.unwrap_or(host version)), which can mask the actual translator build for sources that pass their own plugin version.

Suggestions (design call for maintainers)

• Consider per-version socket paths (daemon-<version>.sock): hooks talk to their own build's daemon, old daemons idle-exit with their sessions, no cross-version shutdown churn. Journals are shared and span IDs deterministic, so continuity survives.
• And/or: make daemon.shutdown drain-aware (refuse/defer while sessions have undelivered rows).
• Consider stamping the daemon build separately from the plugin version (e.g. metadata.bt_daemon_version) so translator-fix verification never depends on which version the plugin reported.