From 6505222e6745a812e2321900bc0ae4b64f7d35ff Mon Sep 17 00:00:00 2001 From: Abhijeet Prasad Date: Mon, 14 Sep 2026 15:53:46 -0400 Subject: [PATCH] docs: clean up readmes across the repository Update setup commands, configuration examples, and test instructions to match current integrations. Add an agent capability matrix and remove stale daemon and package references. Simplify the prose and preserve the generated-repository warnings. --- README.md | 95 +++++++++-- bt-daemon/README.md | 114 ++++++------- bt-daemon/tests/support/README.md | 131 +++++++-------- bt-daemon/tests/support/inference/README.md | 52 +++--- src/plugins/antigravity/content/README.md | 44 +++-- src/plugins/claude/content/README.md | 51 ++++-- src/plugins/claude/content/evals/README.md | 50 ++++-- .../plugins/trace-claude-code/README.md | 14 ++ src/plugins/codex/content/README.md | 47 ++++-- .../content/plugins/trace-codex/README.md | 20 ++- src/plugins/grok/content/README.md | 40 +++-- src/plugins/opencode/content/README.md | 158 ++++++++++-------- src/plugins/pi/content/README.md | 157 +++++++++-------- 13 files changed, 575 insertions(+), 398 deletions(-) diff --git a/README.md b/README.md index 3ec77fc..51810f6 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,88 @@ # Braintrust coding-agent plugins -A monorepo of various [Braintrust](https://braintrust.dev) coding agent plugins. +Trace coding-agent sessions in [Braintrust](https://www.braintrust.dev). +This monorepo contains the plugins and the shared Rust tracing daemon embedded +in the Braintrust CLI (`bt`). Plugins forward native events locally; the daemon +builds traces, journals events for recovery, and delivers spans to Braintrust. -For further instructions, see the instructions for your desired coding agent +## Get started -| Agent | distribution repository | -|-------------|-------------------------| -| Google Antigravity | [braintrustdata/braintrust-antigravity-plugin](https://github.com/braintrustdata/braintrust-antigravity-plugin) | -| Claude Code | [braintrustdata/braintrust-claude-plugin](https://github.com/braintrustdata/braintrust-claude-plugin) | -| Codex | [braintrustdata/braintrust-codex-plugin](https://github.com/braintrustdata/braintrust-codex-plugin) | -| Grok | [braintrustdata/braintrust-grok-plugin](https://github.com/braintrustdata/braintrust-grok-plugin) | -| OpenCode | npm: [`@braintrust/trace-opencode`](https://www.npmjs.com/package/@braintrust/trace-opencode) | -| Pi | npm: [`@braintrust/pi-extension`](https://www.npmjs.com/package/@braintrust/pi-extension) | +Install your coding agent and the +[Braintrust CLI](https://www.braintrust.dev/docs/reference/cli/quickstart), +then authenticate and enable tracing. For example: -## Development & releasing +```bash +bt login +bt trace enable claude --project my-coding-agent +``` -See [AGENTS.md](./AGENTS.md) for the repo structure, the build/deploy model, the -per-plugin versioning, and the manual release workflows. +Replace `claude` with your agent's CLI name below. Restart the agent after +setup and follow any plugin or hook trust prompts. + +## Supported agents + +All six support `bt trace enable`, `update`, `disable`, and `doctor`. + +| Agent and setup guide | CLI name | One-off traced run | Import / live attach | Distribution | +|---|---|---|---|---| +| [Google Antigravity](src/plugins/antigravity/content/README.md) | `antigravity` | — | Yes | [GitHub](https://github.com/braintrustdata/braintrust-antigravity-plugin) | +| [Claude Code](src/plugins/claude/content/README.md) | `claude` | Yes | Yes | [GitHub](https://github.com/braintrustdata/braintrust-claude-plugin) | +| [Codex](src/plugins/codex/content/README.md) | `codex` | Yes | Yes | [GitHub](https://github.com/braintrustdata/braintrust-codex-plugin) | +| [Grok](src/plugins/grok/content/README.md) | `grok` | — | — | [GitHub](https://github.com/braintrustdata/braintrust-grok-plugin) | +| [OpenCode](src/plugins/opencode/content/README.md) | `opencode` | Yes | — | [npm](https://www.npmjs.com/package/@braintrust/trace-opencode) | +| [Pi](src/plugins/pi/content/README.md) | `pi` | Yes | — | [npm](https://www.npmjs.com/package/@braintrust/pi-extension) | + +Use `run` to trace one invocation, `import` to trace a saved session, or +`import --attach` to follow an active transcript: + +```bash +bt trace run --project my-coding-agent claude -- -p "summarize this repository" +bt trace import claude SESSION_ID +bt trace import claude SESSION_ID --attach +``` + +See the agent guides for limitations. Antigravity setup requires a +Unix-compatible shell. + +## Manage tracing + +```bash +bt trace doctor claude +bt trace status +bt trace update claude +bt trace disable claude +``` + +`doctor` shows which configuration is in use. `status` reports trace delivery. +`update` keeps your settings; `disable` removes the plugin and its settings. + +## Development + +```text +src/plugins//content/ Plugin source copied into distribution builds +src/runtime/js-daemon-client/ Shared JavaScript event-forwarding client +src/skills/ Shared skills included in plugin builds +bt-daemon/ Rust daemon, translators, and integration tests +scripts/ Shared build and publishing tools +.github/workflows/ CI and release automation +``` + +Use Bash and Python 3 for plugin scripts, Node.js 24 and pnpm 11.21.0 for the +JavaScript packages, and stable Rust for the daemon. Run from the monorepo root: + +```bash +make build # Build all plugins into dist/ +make build-codex # Build one plugin +make test # Build and validate plugin packages and forwarders +cargo test --manifest-path bt-daemon/Cargo.toml --all-features --locked +``` + +Real-agent tests require installed agents and are run separately; see the +[test harness guide](bt-daemon/tests/support/README.md). See the +[daemon README](bt-daemon/README.md) for architecture and local debugging. + +## Releases and contributions + +Edit plugin sources here. The per-agent GitHub distribution repositories and +npm packages are generated artifacts. See [AGENTS.md](AGENTS.md) for the release +PR and approval workflow, sandbox publishing, and npm release workflows. diff --git a/bt-daemon/README.md b/bt-daemon/README.md index aee0f40..c20b429 100644 --- a/bt-daemon/README.md +++ b/bt-daemon/README.md @@ -1,32 +1,26 @@ # bt-daemon -Shared Rust project for Braintrust coding-agent tracing plugins. A local, -stateful daemon that plugin **hook shims** forward events to; it owns the -event→trace state machine and sends spans to Braintrust out-of-band. See -[`docs/protocol.md`](docs/protocol.md) for the wire contract. - -> **Placeholder name** — the real name is TBD. The subcommand framing -> (`serve` / `hook` / `status` / `import` / `run`) should survive a rename. +The Rust tracing daemon embedded in `bt`. Plugins send it native agent events; +it builds spans, journals events for recovery, and uploads traces to Braintrust. +See [the protocol](docs/protocol.md) for the wire contract. ## Layout -One self-contained Cargo crate, liftable to its own repo by copying -`bt-daemon/` verbatim: +The daemon is one self-contained Cargo crate: - `src/wire` — the wire protocol module: envelope types + JSON-RPC framing. - `src/translate` and `src/sink` — agent state machines and Braintrust output. -- `src/lib.rs` — the embeddable library: clap `Args` + async entry points +- `src/lib.rs` — command arguments and async entry points, including + `run_serve`, `run_hook`, `run_status`, `run_import`, and `run_traced`. - `src/trace_command.rs`, `src/trace_runtime.rs`, and `src/setup.rs` — the complete mounted `bt trace` command schema, dispatch, daemon lifecycle, and agent-specific persistent setup behavior. Hosts supply only credential and destination-resolution services. - (`run_serve`, `run_hook`, `run_status`, `run_import`, `run_traced`). This is what `bt` - depends on. - `src/main.rs` — the standalone **`bt-daemon` binary**, compiled only with the `cli` feature for isolated testing/development. Env/flag static-token auth only; not an end-user artifact. -## Dual consumption and authentication +## Authentication Hooks send only a non-secret `SessionRoute`: an optional profile and organization selection plus the trace destination. The long-lived daemon asks @@ -50,13 +44,14 @@ Each coding agent reads an independent non-credential `braintrust.json` file: `~/.config/opencode/braintrust.json` - Pi: `~/.pi/agent/braintrust.json` - Grok: `~/.grok/braintrust.json` +- Antigravity: `~/.gemini/config/braintrust.json` `BT_DAEMON_CONFIG` can override the path for isolated tests and managed hosts. See [`config.json.example`](config.json.example). `trace_to_braintrust` controls enablement and `route` stores the selected profile, organization, typed -destination, flush mode, and metadata. Omitting `route.auth.profile` selects -the default `bt` profile. Credentials and backend URLs are never stored here; +destination, flush mode, and metadata. Setup saves a stable `route.auth.profile_id`; older files can use +`route.auth.profile`. If neither is set, `bt` resolves the default profile. Credentials and backend URLs are never stored here; production resolves and refreshes them through `bt`. `bt trace run` supplies a process-local settings overlay and never changes any of these files. @@ -108,29 +103,31 @@ Use `bt trace disable ` to remove the installed tracing plugin and its Braintrust settings. `bt trace setup ` remains an alias for `bt trace enable ` for backwards compatibility. -## Build / test +## Build and test ```bash cd bt-daemon -cargo test # library + pipeline tests -cargo test --features cli # also compile/test the CLI -cargo build --features cli --bin bt-daemon # standalone test binary +cargo test --locked # Library and pipeline tests +cargo test --all-features --locked # Include the standalone CLI +cargo build --features cli --locked --bin bt-daemon ``` CI runs the all-feature build, test suite, and Clippy on Linux, macOS, and Windows. The pipeline integration tests use Unix-domain sockets on Unix and -real Windows named pipes on Windows. +real Windows named pipes on Windows. Real-agent tests are separate from the +default suite; see the [test harness guide](tests/support/README.md). + +## Local debugging -## Try it (standalone, debug sink) +From the monorepo root, in a Unix shell: ```bash export BT_DAEMON_SOCKET=/tmp/btd.sock BT_DAEMON_DATA_DIR=/tmp/btd -cargo build --features cli --bin bt-daemon -echo '{"session_id":"s1","hook_event_name":"SessionStart"}' | ./target/debug/bt-daemon hook --source debug -echo '{"session_id":"s1","hook_event_name":"Stop"}' | ./target/debug/bt-daemon hook --source debug -./target/debug/bt-daemon status -# journaled events: $BT_DAEMON_DATA_DIR/journal/s1.ndjson -# emitted span rows: $BT_DAEMON_DATA_DIR/spans/s1.ndjson +cargo build --manifest-path bt-daemon/Cargo.toml --features cli --locked --bin bt-daemon +echo '{"session_id":"s1","hook_event_name":"SessionStart"}' | ./bt-daemon/target/debug/bt-daemon hook --source debug +echo '{"session_id":"s1","hook_event_name":"Stop"}' | ./bt-daemon/target/debug/bt-daemon hook --source debug +./bt-daemon/target/debug/bt-daemon status +# Inspect journal/ and spans/ under $BT_DAEMON_DATA_DIR ``` The first `hook` spawns the daemon detached; it idles out after 5 minutes. @@ -143,8 +140,8 @@ create a trace for the past session. Hook-only facts absent from a native transcript are not invented. Add `--attach` to keep following an active Codex, Claude, or Antigravity transcript until -Ctrl-C. `run [ARGS...]` launches the selected agent with -inherited stdio and injects live Braintrust hooks for that invocation, so it +Ctrl-C. `run [ARGS...]` launches the selected agent with +inherited stdio and injects Braintrust hooks or an adapter for that invocation, so it does not depend on the tracing plugin being installed or enabled. Managed runs suppress inherited Braintrust plugin hooks to avoid logging the same session twice; the injected hooks still use the normal daemon translator and sink. @@ -157,33 +154,32 @@ tracing for that invocation and override the persistent setup route without rewriting it, so ordinary agent sessions and concurrent managed runs may use different profiles, organizations, projects, experiments, or parent spans. -## Status - -Phases 0–5 are implemented: protocol, daemon lifecycle, Braintrust sink, -Codex, Claude, and Grok translators, `bt daemon` integration, and thin hook shims for -all shipped plugins. Every coding-agent capture request returns after the raw -event is flushed to its journal; authentication, correlation, translation, and -reporting run on daemon-owned workers. Restart recovery replays the redacted -journal with deterministic span ids, so resubmitted rows merge into the same -spans instead of creating duplicates. Claude and Codex lifecycle entries -reference a daemon-owned transcript mirror; Grok records independent bounded -updates and events mirrors. Recovery therefore does not depend on mutable -external paths or copy a full transcript into every event. Explicit -turn/session-end flushes are bounded, and sessions can target project logs or -an experiment. - -Memory is bounded end to end, while on-disk records stay complete: the daemon -never holds a transcript or a whole journal in memory, mirroring and replay -both stream, session queues apply backpressure, and sessions that go quiet are -retired and rebuilt from their journal on the next event. Nothing on disk — -journal, mirror, or conversation content — is capped or truncated; only -in-memory caches are bounded, and each is re-derivable from disk. - -Windows named-pipe transport, detached spawning, lifecycle handover, and -cross-platform pipeline tests are implemented. The remaining host follow-ups -are OpenCode and pi, which are not present in this monorepo. - -- The Braintrust sink pins `braintrust-sdk-rust` commit `d33e806`, which adds - deterministic span ids, `span_origin`/`span_attributes` passthrough, and - per-session credential isolation. This follows the same exact-revision Git - dependency policy as `bt`. +## Recovery and storage + +Capture requests return after the event is flushed to its journal. Daemon +workers handle authentication, translation, and delivery. Restart recovery +replays journaled events with deterministic span IDs, so repeated delivery +updates the same spans. + +Claude and Codex events reference daemon-owned transcript mirrors. Grok uses +separate updates and events mirrors. Replay does not depend on the original +transcript staying at its old path. + +The daemon streams journals and transcripts, applies queue backpressure, and +retires idle sessions. Retired sessions are rebuilt from their journals when +another event arrives. Journals and mirrors preserve conversation content; +credentials are excluded from journaled routing data. + +Recovery files are removed after seven days without modification. Cleanup runs +at startup and hourly while the daemon is running. This retention period is +currently fixed in `src/server.rs`. + +## Platform and integration coverage + +The daemon supports Linux, macOS, and Windows, using Unix sockets or Windows +named pipes. Translators exist for Antigravity, Claude Code, Codex, Grok, +OpenCode, and Pi. CI runs packaged Claude Code, Codex, OpenCode, and Pi against +mock inference and mock Braintrust ingest on all three platforms. + +The Rust SDK is pinned to an exact Git revision in [Cargo.toml](Cargo.toml). +Use `--locked` for reproducible builds. diff --git a/bt-daemon/tests/support/README.md b/bt-daemon/tests/support/README.md index c00fc2d..d26838c 100644 --- a/bt-daemon/tests/support/README.md +++ b/bt-daemon/tests/support/README.md @@ -1,68 +1,67 @@ # Agent integration test architecture -The test infrastructure has three independent layers: - -- `server` is a generic container that binds any Axum `Router` to an - ephemeral address and owns its lifecycle. -- `inference` contains OpenAI Responses and Anthropic Messages protocol logic, - programmable scenarios, and captured inference requests. Each mock exports - an Axum router and can be hosted or embedded by any caller. -- `ingest` contains the mock Braintrust API and captured trace rows. It also - exports an Axum router and has no dependency on the server container. Its - scenario builder matches named row shapes as an ordered subsequence, - independent of HTTP batching and unrelated SDK update rows. - -`agent_process` is the Braintrust-specific orchestration layer. It hosts the -ingest router, starts the daemon, and provides the environment shared by agent -processes. - -`agents` contains reusable adapters for real coding-agent CLIs. Each adapter -owns only agent installation and isolated configuration state. The daemon -world is passed to each run as its execution context, avoiding any lifetime or -ownership coupling between the two layers. Adapters provide standard -invocation flags, mock-inference routing, and process output. Runs remain -configurable with additional arguments and environment variables so scenarios -can add inputs such as attachment paths without duplicating CLI setup. - -OpenCode and Pi integration runs set `OPENCODE_PLUGIN` and `PI_EXTENSION_PATH` -to entrypoints from isolated installations of their ephemeral npm packages. -This exercises the same peer-dependency resolution and published file -allowlists as registry installs, rather than loading monorepo build trees -directly. - -The integration test composes those pieces: it hosts an inference router, -starts the daemon world, runs an agent, and evaluates the ingest scenario. This -keeps both protocol mocks usable without coding agents, keeps the generic -server unaware of either protocol, and lets new end-to-end scenarios focus on -model behavior and expected trace shapes. - -The world controls inference and ingest independently: - -- `BT_AGENT_INFERENCE_MODE=mock|live` selects deterministic mock inference or - the agent's normal provider. -- `BT_AGENT_INGEST_MODE=mock|live` selects captured local ingest or the normal - Braintrust backend. - -Mock ingest launches the feature-gated standalone daemon with test -credentials. Live ingest instead launches the profile-aware daemon embedded in -`bt`, selected by: - -- `BT_AGENT_BT_BIN` — `bt` executable to test (defaults to `bt` on `PATH`); -- `BT_AGENT_PROFILE` — optional saved OAuth or API-key profile; -- `BT_AGENT_ORG` — optional organization constraint; -- `BT_AGENT_PROJECT` — destination project name (defaults to `agent-e2e`). - -Only those non-secret selections are written to the harness route. The `bt` -daemon host resolves credentials and refreshes OAuth leases internally. - -This allows deterministic inference to drive real Braintrust ingest without -paying for model inference. Every test uses ordinary assertions for stable -process behavior and trace delivery regardless of mode. When ingest is mocked, -the captured rows are also available for ordinary assertions over stable -metadata. With live ingest, the daemon must report emitted spans and no sink -errors. - -`IngestScenario` is exclusively for the additional deterministic expectations -when both inference and ingest are mocked. Provider request sequences, exact -model output, injected provider failures, and ordered trace shapes are layered -on top of the always-run assertions. +Run these tests from the monorepo root. They launch real agents against mock +model endpoints and mock Braintrust ingest by default. + +## Run the tests + +Install the agents you want to test. The harness finds `codex`, `claude`, +`opencode`, and `pi` on `PATH`; `CODEX_BIN`, `CLAUDE_BIN`, `OPENCODE_BIN`, and +`PI_BIN` override their executable paths. + +OpenCode and Pi also require installed plugin packages. Set `OPENCODE_PLUGIN` +and `PI_EXTENSION_PATH` to absolute paths to their installed `dist/index.mjs` +files. Use packed npm artifacts with their peers installed, as shown in the +[CI workflow](../../../.github/workflows/ci.yml). + +```bash +cargo test --manifest-path bt-daemon/Cargo.toml --all-features --locked \ + --test agent_integration -- --ignored --nocapture --test-threads=1 +``` + +These tests are ignored by a normal `cargo test` because they require agent +executables. To run one agent, add a test-name filter before `--` (for example, +`codex`). The ordinary Rust suite covers translators and daemon behavior +without installing agents. + +## Test layers + +- `server` binds an Axum router to an ephemeral address and manages its lifetime. +- `inference` implements mock OpenAI Responses and Anthropic Messages endpoints. + Scenarios control responses and record incoming requests. +- `ingest` implements a mock Braintrust API and records span rows. Scenarios + match ordered row shapes independently of HTTP batching. +- `agent_process` starts the daemon and ingest server and supplies the shared + process environment. +- `agents` configures and runs each CLI in an isolated workspace. + +An integration test starts an inference mock and daemon, runs an agent, and +checks the resulting trace. Agent arguments and environment variables can be +customized per scenario. + +## Mock and live backends + +Inference and ingest are configured independently: + +| Variable | Values | Default | +|---|---|---| +| `BT_AGENT_INFERENCE_MODE` | `mock`, `live` | `mock` | +| `BT_AGENT_INGEST_MODE` | `mock`, `live` | `mock` | + +Mock ingest uses the standalone daemon with test credentials. Live ingest uses +the daemon embedded in `bt`, with these settings: + +| Variable | Purpose | Default | +|---|---|---| +| `BT_AGENT_BT_BIN` | `bt` executable | `bt` on `PATH` | +| `BT_AGENT_PROFILE` | Saved profile | `BRAINTRUST_PROFILE` | +| `BT_AGENT_ORG` | Organization | `BRAINTRUST_ORG_NAME` | +| `BT_AGENT_PROJECT` | Destination project | `agent-e2e` | + +The harness writes profile and destination selections to the route; `bt` +resolves credentials and refreshes OAuth tokens. + +All modes check process success and trace delivery. Mock ingest also allows +assertions on captured rows. `IngestScenario` adds exact output, failure, and +ordering checks when both backends are mocked. See the +[inference guide](inference/README.md) for examples of live-backend runs. diff --git a/bt-daemon/tests/support/inference/README.md b/bt-daemon/tests/support/inference/README.md index a03e643..46fabce 100644 --- a/bt-daemon/tests/support/inference/README.md +++ b/bt-daemon/tests/support/inference/README.md @@ -1,18 +1,15 @@ # Deterministic inference test support -This directory is a self-contained mock-inference component with two -protocol-faithful servers: +Two mocks implement the model API endpoints used by the integration tests: -- `OpenAiMock` implements the OpenAI Responses API surface used by Codex. +- `OpenAiMock` implements the OpenAI Responses API used by Codex, OpenCode, + and Pi tests. - `AnthropicMock` implements the Anthropic Messages API surface used by Claude Code. -Each public mock owns its protocol routes, scenario closure, and captured -requests, and exports an Axum `Router`. Callers can bind that router with the -shared ephemeral test server or embed it in another Axum application. The two -providers share request indexing and transport outcomes. Request and response -types remain provider-specific so a test cannot accidentally hide a -wire-protocol incompatibility behind a common model abstraction. +Each mock exports an Axum router and records incoming requests. Request and +response types are provider-specific; request indexing and transport outcomes +are shared. Both mocks accept a thread-safe closure: @@ -38,20 +35,16 @@ response bodies for malformed or truncated stream tests. Typed turn builders generate deterministic ids, token usage, and valid provider SSE sequences. Every inference request is captured for later assertions. -The component does not depend on `bt-daemon`, the coding-agent runner, the -ingest mock, or a particular listener implementation. The higher-level -`support::agent_process` harness composes with it only from the integration -test. This boundary is deliberate so the whole mock-inference component can -later move into a reusable crate and serve any client that can target an -OpenAI Responses or Anthropic Messages endpoint. - -`agent_integration.rs` runs real Codex and Claude Code processes against these -mocks. -The tests are ignored in a plain Rust run because they require agent -executables. The core cross-platform CI matrix installs the latest release of -each agent and runs them in the default `mock` mode on every host. This is -intentionally unpinned so upstream compatibility breaks are visible -immediately. +The mocks can run under the shared test server or another Axum application. +They do not depend on the daemon, ingest mock, or agent runner. + +`agent_integration.rs` runs Codex, Claude Code, OpenCode, and Pi against these +mocks. CI installs the latest agents and runs the tests on Linux, macOS, and +Windows. See the [harness guide](../README.md) for executables and package setup. + +## Live backends + +Run the following commands from the monorepo root after preparing the agents. The same agent tests can run without mock inference while continuing to use captured local ingest: @@ -59,12 +52,11 @@ captured local ingest: ```console BT_AGENT_INFERENCE_MODE=live BT_AGENT_INGEST_MODE=mock \ cargo test --manifest-path bt-daemon/Cargo.toml \ - --all-features --test agent_integration -- --ignored --test-threads=1 + --all-features --locked --test agent_integration -- --ignored --test-threads=1 ``` Live inference uses the normal provider endpoint/model and the agent's normal -login or provider credentials. It validates only stable integration invariants -such as trace delivery and origin metadata. Mock inference additionally +login or provider credentials. Tests check trace delivery and origin metadata. Mock inference additionally validates exact request sequences, tool results, output content, and injected failures. @@ -76,10 +68,8 @@ BT_AGENT_INFERENCE_MODE=mock BT_AGENT_INGEST_MODE=live \ BT_AGENT_BT_BIN=/path/to/bt BT_AGENT_PROFILE=work \ BT_AGENT_PROJECT=agent-e2e \ cargo test --manifest-path bt-daemon/Cargo.toml \ - --all-features --test agent_integration -- --ignored --test-threads=1 + --all-features --locked --test agent_integration -- --ignored --test-threads=1 ``` -The live-ingest path starts the daemon through `bt`, so OAuth access-token -refresh and profile resolution remain inside the long-lived CLI host. The -standalone daemon is intentionally limited to mock-ingest and explicit API-key -development scenarios. +Live ingest starts the daemon through `bt`, which handles profiles and OAuth +refresh. The standalone test daemon uses explicit API-key authentication. diff --git a/src/plugins/antigravity/content/README.md b/src/plugins/antigravity/content/README.md index d6f98d0..ec2ac6d 100644 --- a/src/plugins/antigravity/content/README.md +++ b/src/plugins/antigravity/content/README.md @@ -11,22 +11,20 @@ Trace Google Antigravity coding sessions in Braintrust. Prerequisites: -- The latest [Google Antigravity](https://antigravity.google/) -- The latest [Braintrust CLI (`bt`)](https://www.braintrust.dev/docs/reference/cli/quickstart) +- [Google Antigravity](https://antigravity.google/) +- A Unix-compatible `sh` (persistent setup is not supported on Windows) +- The [Braintrust CLI (`bt`)](https://www.braintrust.dev/docs/reference/cli/quickstart) Install the plugin and choose where traces are sent: ```bash -bt login --profile myprofile -bt trace --profile myprofile -p my-coding-agent-project enable antigravity +bt login +bt trace enable antigravity --project my-coding-agent ``` -This causes Google Antigravity sessions to report to your configured project. -You can disable the plugin later with: - -```bash -bt trace disable antigravity -``` +Setup installs the hooks and saves non-secret routing settings in +`~/.gemini/config/braintrust.json`. Use `--profile` or `--org` to select a +Braintrust profile or organization. Restart Antigravity after setup. ## What is captured @@ -36,8 +34,28 @@ Each traced session includes: - a turn span for each user request and visible assistant response; - LLM spans with available model inputs, outputs, and token usage; - tool spans with observable inputs, outputs, duration, outcome, and errors; -- useful session metadata such as the Antigravity version, model, workspace, +- session metadata such as the Antigravity version, model, workspace, and native conversation ID. -The plugin forwards events only to the local Braintrust daemon. It does not -contain Braintrust credentials or send traces directly to Braintrust. +The plugin sends events to the local `bt` daemon, which handles credentials +and uploads traces. + +## Transcript import + +```bash +bt trace import antigravity CONVERSATION_ID +bt trace import antigravity CONVERSATION_ID --attach +``` + +Import reconstructs a trace from a saved conversation; `--attach` follows an +active conversation until Ctrl-C. Only information present in the transcript +can be recovered. `bt trace run antigravity` is not supported. + +## Manage tracing + +```bash +bt trace doctor antigravity +bt trace status +bt trace update antigravity +bt trace disable antigravity +``` diff --git a/src/plugins/claude/content/README.md b/src/plugins/claude/content/README.md index a20e8c5..49153f0 100644 --- a/src/plugins/claude/content/README.md +++ b/src/plugins/claude/content/README.md @@ -1,4 +1,4 @@ -# Braintrust Claude Code Marketplace +# Braintrust tracing for Claude Code > **This repository is generated.** It is built from > [braintrustdata/braintrust-coding-agent-plugins](https://github.com/braintrustdata/braintrust-coding-agent-plugins). @@ -10,7 +10,8 @@ A Claude Code plugin marketplace for tracing Claude Code sessions to [Braintrust ## Prerequisites - A [Braintrust account](https://braintrust.dev) -- The `bt` CLI, authenticated with `bt login` +- Claude Code +- The [Braintrust CLI](https://www.braintrust.dev/docs/reference/cli/quickstart) ## Supported Claude surfaces @@ -22,24 +23,19 @@ configuration, or environment variables. In Cowork, use the Braintrust connector provided through Claude for MCP access. Automatic Cowork session tracing is not currently supported. -## Installation +## Quickstart -Add the marketplace: - -```bash -claude plugin marketplace add braintrustdata/braintrust-claude-plugin -``` - -Then enable tracing: - -Automatically traces Claude Code conversations to Braintrust through the shared -Braintrust daemon. The plugin contains only a fail-open hook forwarder; `bt` -owns authentication, trace construction, and delivery. +Authenticate and install the published tracing plugin: ```bash +bt login bt trace enable claude --project my-coding-agent ``` +Setup adds the marketplace and installs or enables `trace-claude-code`. +The plugin forwards lifecycle events locally; `bt` owns authentication, +trace construction, and delivery. + Use `--profile` or `--org` when needed. Setup stores only non-secret routing settings under `~/.claude/braintrust.json`. Restart Claude Code after setup. @@ -50,7 +46,7 @@ failures never fail a Claude Code turn. This marketplace does not install or configure the Braintrust MCP server. Use your agent's native connector or MCP configuration when you want MCP access. -#### Additional root metadata +## Additional root metadata For a persistent route, pass a JSON object to `bt trace enable claude --additional-metadata ''` to tag the root span of every Claude Code @@ -61,7 +57,7 @@ For one invocation without changing the persistent configuration, use or set `BRAINTRUST_ADDITIONAL_METADATA` before that command (`bt trace run` still accepts it; a launched `claude` session's live hooks do not). -#### Root-span tags +## Root-span tags Use repeatable `--tag` options to apply filterable tags to every root span in a route. Tags can be persisted with setup, supplied to one invocation, or added @@ -79,3 +75,26 @@ invoking setup, a run, or an import: ```bash BRAINTRUST_TAGS=ci,release-validation bt trace run claude -- "review this change" ``` + +## One-off runs and transcript import + +```bash +bt trace run --project my-coding-agent claude -- -p "summarize this repository" +bt trace import claude SESSION_ID +bt trace import claude SESSION_ID --attach +``` + +`run` leaves your saved settings alone. `import` reads a saved transcript; +`--attach` follows it until Ctrl-C. Imports include only what the agent recorded. + +## Manage tracing + +```bash +bt trace doctor claude +bt trace status +bt trace update claude +bt trace disable claude +``` + +See the [tracing plugin guide](plugins/trace-claude-code/README.md) for the +capture architecture and supported surfaces. diff --git a/src/plugins/claude/content/evals/README.md b/src/plugins/claude/content/evals/README.md index 4086dbd..323e324 100644 --- a/src/plugins/claude/content/evals/README.md +++ b/src/plugins/claude/content/evals/README.md @@ -1,32 +1,48 @@ -# Braintrust Skill Evals +# Braintrust skill evaluations -Evaluation suite for Braintrust skills. +Python evaluations for Braintrust documentation answers, data workflows, and +MCP tool use. These are separate from the daemon's tracing integration tests. ## Setup -### Prerequisites - -- Python 3.12+ -- [uv](https://docs.astral.sh/uv/) package manager - -### Installation +Use Python 3.12+ and [uv](https://docs.astral.sh/uv/). From the monorepo root: ```bash -cd evals -uv sync +cd src/plugins/claude/content/evals +uv sync --locked ``` -## Running Evals +In the generated Claude distribution repository, use `cd evals` instead. +Set `ANTHROPIC_API_KEY` for model calls and `BRAINTRUST_API_KEY` for reporting. +These evaluations use live services and write results to Braintrust. -```bash -cd evals +## Run evaluations -# Run a specific eval -uv run python eval_docs_search.py +From the `evals` directory: -# Run all evals -uv run python eval_datasets.py +```bash uv run python eval_docs_search.py +uv run python eval_datasets.py uv run python eval_experiments.py uv run python eval_log_querying.py ``` + +These scripts score answers against criteria using an LLM judge. +The end-to-end scripts also run Claude agents and create test data in Braintrust: + +```bash +uv run --with claude-agent-sdk python eval_e2e_log_fetch.py +uv run --with claude-agent-sdk python eval_e2e_eval_improve.py +``` + +The agent SDK is an extra dependency for these two scripts. Their MCP calls +use `BRAINTRUST_API_KEY`. + +## API helper tests + +```bash +uv run --with pytest pytest test_braintrust_api.py -v +``` + +These tests call the live Braintrust API and create test projects and data. +They are skipped when `BRAINTRUST_API_KEY` is unset. diff --git a/src/plugins/claude/content/plugins/trace-claude-code/README.md b/src/plugins/claude/content/plugins/trace-claude-code/README.md index 2ec5f8d..1cba9c8 100644 --- a/src/plugins/claude/content/plugins/trace-claude-code/README.md +++ b/src/plugins/claude/content/plugins/trace-claude-code/README.md @@ -25,3 +25,17 @@ To add fields to each root trace span, pass a JSON object (as `bt trace run claude` for one invocation. The hook itself never reads that environment variable — only `bt trace enable`, `bt trace run`, and `bt trace import` do. + +## Tags and diagnostics + +Use repeatable `--tag` flags for filterable root-span tags. Inspect the effective +configuration with `doctor` and delivery state with `status`: + +```bash +bt trace enable claude --tag coding-agent --tag development +bt trace doctor claude +bt trace status +``` + +See the [distribution guide](../../README.md) for installation, one-off runs, +transcript import, updates, and disablement. diff --git a/src/plugins/codex/content/README.md b/src/plugins/codex/content/README.md index 3e87ba8..de130ae 100644 --- a/src/plugins/codex/content/README.md +++ b/src/plugins/codex/content/README.md @@ -1,34 +1,53 @@ -# Braintrust Codex Plugin +# Braintrust tracing for Codex > **This repository is generated.** It is built from > [braintrustdata/braintrust-coding-agent-plugins](https://github.com/braintrustdata/braintrust-coding-agent-plugins). > Don't edit files here — make changes and file issues in that repository, and they > will be rebuilt into this one. -Daemon-backed Braintrust session tracing for [Codex](https://developers.openai.com/codex/plugins). +Trace Codex sessions in Braintrust with the `trace-codex` plugin. ## Quickstart -Add this repo as a Codex plugin marketplace: +Install Codex and the +[Braintrust CLI](https://www.braintrust.dev/docs/reference/cli/quickstart), then run: ```bash -codex plugin marketplace add braintrustdata/braintrust-codex-plugin -codex plugin add trace-codex@braintrust-codex-plugins +bt login +bt trace enable codex --project my-coding-agent ``` -The recommended tracing setup is: +Setup adds the published marketplace, installs the plugin, and saves routing +settings in `~/.codex/braintrust.json`. Use `--profile` or `--org` to select a +Braintrust profile or organization. Restart Codex after setup and approve the +Braintrust hook through `/hooks` when prompted. + +## What is captured + +The daemon builds session, turn, LLM, and tool spans from Codex hooks and +transcripts, including available usage, compaction, and subagent activity. +The plugin forwards events locally; `bt` owns authentication and delivery. + +## One-off runs and transcript import ```bash -bt trace enable codex --project my-coding-agent +bt trace run --project my-coding-agent codex -- exec "summarize this repository" +bt trace import codex SESSION_ID +bt trace import codex SESSION_ID --attach ``` -This installs the tracing plugin and stores only non-secret routing settings. -The `bt` CLI owns authentication and forwards hook events through the shared -daemon. Restart Codex after setup. +`run` leaves your saved settings alone. `import` reads a saved transcript; +`--attach` follows it until Ctrl-C. Imports include only what the agent recorded. -## trace codex plugin +## Manage tracing -See the plugin's [README](/plugins/trace-codex/README.md) for details. +```bash +bt trace doctor codex +bt trace status +bt trace update codex +bt trace disable codex +``` -This marketplace does not install or configure the Braintrust MCP server. Use -Codex's native connector or MCP configuration when you want MCP access. +See the [tracing plugin guide](plugins/trace-codex/README.md) for hook behavior, +metadata, and tags. This marketplace does not configure the Braintrust MCP +server; use the agent's connector or MCP configuration for MCP access. diff --git a/src/plugins/codex/content/plugins/trace-codex/README.md b/src/plugins/codex/content/plugins/trace-codex/README.md index 73ced29..c28b564 100644 --- a/src/plugins/codex/content/plugins/trace-codex/README.md +++ b/src/plugins/codex/content/plugins/trace-codex/README.md @@ -25,15 +25,15 @@ Use `--profile` or `--org` when needed. Restart Codex after setup so it loads the plugin. Codex will apply its normal hook-review flow; approve the stable Braintrust hook definition through `/hooks` when prompted. -To verify the daemon path is available: +To inspect tracing configuration and daemon status: ```bash -bt trace hook --help +bt trace doctor codex bt trace status ``` Hook setup or forwarding never fails a Codex turn. If installation fails or the -daemon cannot accept an event, the launcher reports a bounded diagnostic and +daemon cannot accept an event, the launcher reports the failure and exits successfully. ## Additional root metadata @@ -46,3 +46,17 @@ For one invocation without changing the persistent configuration, use `bt trace run --additional-metadata '{"ci":true,"run_id":"abc-123"}' codex`, or set `BRAINTRUST_ADDITIONAL_METADATA` before that command (`bt trace run` still accepts it; a launched `codex` session's live hooks do not). + +## Tags and diagnostics + +Use repeatable `--tag` flags for filterable root-span tags. Inspect the effective +configuration with `doctor` and delivery state with `status`: + +```bash +bt trace enable codex --tag coding-agent --tag development +bt trace doctor codex +bt trace status +``` + +See the [distribution guide](../../README.md) for installation, one-off runs, +transcript import, updates, and disablement. diff --git a/src/plugins/grok/content/README.md b/src/plugins/grok/content/README.md index 606e623..d9ac457 100644 --- a/src/plugins/grok/content/README.md +++ b/src/plugins/grok/content/README.md @@ -11,24 +11,24 @@ Trace Grok coding sessions in Braintrust. Prerequisites: -- The latest [Grok CLI](https://grok.com/build) -- The latest [Braintrust CLI (`bt`)](https://www.braintrust.dev/docs/reference/cli/quickstart) +- The [Grok CLI](https://grok.com/build) +- The [Braintrust CLI (`bt`)](https://www.braintrust.dev/docs/reference/cli/quickstart) Install: ```bash -grok plugin install braintrustdata/braintrust-grok-plugin --trust -grok plugin enable trace-grok -bt login --profile myprofile -bt trace --profile myprofile -p my-coding-agent-project enable grok +bt login +bt trace enable grok --project my-coding-agent ``` -This causes `grok` sessions to report to your configured project. +Setup installs and enables the published plugin and saves non-secret routing +settings in `~/.grok/braintrust.json`. Use `--profile` or `--org` to select a +Braintrust profile or organization. Restart Grok after setup. -> **Note:** Due to a bug in the Grok CLI, you must run `/reload-plugins` at the -> start of each session for traces to be reported. You can use -> `alias grok="grok /reload-plugins"` until the -> [issue is resolved](https://github.com/xai-org/plugin-marketplace/issues/236). +If the plugin is enabled but hooks do not fire, try `/reload-plugins` at the +start of the session. Grok 1.0.3 has a reported plugin-hook dispatch issue; +see [the upstream report](https://github.com/xai-org/plugin-marketplace/issues/236) +for details. ## What is captured @@ -41,11 +41,11 @@ Each traced session includes: - per-turn token, cache, reasoning, model-call, API-duration, and raw cost metrics when Grok records them; - the system prompt and first user message on the first LLM span when available; -- useful session metadata such as Grok and plugin versions, working directory, +- session metadata such as Grok and plugin versions, working directory, workspace, and native session ID. -The plugin forwards events only to the local Braintrust daemon. It does not -contain Braintrust credentials or send traces directly to Braintrust. +The plugin sends events to the local `bt` daemon, which handles credentials +and uploads traces. ## Caveats @@ -64,5 +64,13 @@ native boundaries for every model call. As a result: - managed `bt trace run grok` and historical `bt trace import grok` workflows are not currently supported. -These limitations affect trace detail, not Grok execution. Tracing is fail-open: -if the plugin or local daemon is unavailable, Grok continues normally. +Grok keeps running if tracing fails. + +## Manage tracing + +```bash +bt trace doctor grok +bt trace status +bt trace update grok +bt trace disable grok +``` diff --git a/src/plugins/opencode/content/README.md b/src/plugins/opencode/content/README.md index cf7de22..ef28518 100644 --- a/src/plugins/opencode/content/README.md +++ b/src/plugins/opencode/content/README.md @@ -1,40 +1,52 @@ # @braintrust/trace-opencode -Braintrust tracing plugin for [OpenCode](https://opencode.ai). The JavaScript -adapter forwards native OpenCode events to the installed `bt` daemon, which -constructs and delivers the trace. - -Version 1 requires a current `bt` CLI with the OpenCode daemon translator. If -`bt` or the translator is unavailable, tracing fails open and OpenCode keeps -running. +Trace [OpenCode](https://opencode.ai) sessions in Braintrust. The plugin sends +events to the local `bt` daemon, which builds and uploads traces. OpenCode keeps +running if tracing fails. - **Session spans**: Root span for each OpenCode session with metadata (workspace, hostname, etc.) - **Turn spans**: Captures each user-assistant interaction +- **LLM spans**: Records completed model messages with available inputs, outputs, and usage - **Tool spans**: Records individual tool executions with inputs and outputs -## Quick Start +## Quickstart + +Install OpenCode and the +[Braintrust CLI](https://www.braintrust.dev/docs/reference/cli/quickstart), then run: ```bash -bt auth login -bt trace enable opencode +bt login +bt trace enable opencode --project my-coding-agent opencode ``` -For one invocation without changing OpenCode's global tracing configuration, -use `bt trace run --project opencode -- [OPENCODE_ARGS...]`. +This registers the npm plugin and saves its configuration. Use `--profile` or +`--org` to choose a profile or organization. Restart OpenCode if it is already open. -## Configuration +For one invocation without changing global tracing configuration: -You can configure the plugin using a config file. `braintrust.json` is the -only source of persistent tracing configuration; routing and enablement are -never read directly from the environment. +```bash +bt trace run --project my-coding-agent opencode -- run "summarize this repository" +``` -### Config File +Historical import and live attach are not supported for OpenCode. -Create a `braintrust.json` file in one of these locations: +## Compatibility -- `.opencode/braintrust.json` - Project-level config -- `~/.config/opencode/braintrust.json` - Global config +The package declares `@opencode-ai/plugin` and `@opencode-ai/sdk` peers at +`>=1.2.25`. CI checks package installation at that minimum and at `latest`, +and runs real-agent integration tests with the latest OpenCode CLI. + +## Configuration + +Settings load from these files, with project settings overriding global settings: + +- Global: `$XDG_CONFIG_HOME/opencode/braintrust.json`, or + `~/.config/opencode/braintrust.json` if `XDG_CONFIG_HOME` is unset +- Project: `.opencode/braintrust.json` + +`bt trace run` overrides tracing settings for one invocation. Files are merged +at the top level: a project `route` replaces the global `route`. ```json { @@ -48,34 +60,28 @@ Create a `braintrust.json` file in one of these locations: } ``` -### Config Options - -| Config Key | Env Var | Type | Default | Description | -|------------|---------|------|---------|-------------| -| `trace_to_braintrust` | — | boolean | `false` | Enable/disable tracing | -| `enable_tools` | `BRAINTRUST_OPENCODE_ENABLE_TOOLS` | boolean | `true` | Register Braintrust tools in OpenCode | -| `profile` | — | string | current `bt` profile | Select the `bt` auth profile used by tracing and tools | -| `project` | — | string | `"opencode"` | Project name for traces and project-scoped tools | -| `debug` | `BRAINTRUST_DEBUG` | boolean | `false` | Enable debug logging | -| `org_name` | — | string | profile default | Organization selected within the tracing profile and for tools | -| `additional_metadata` | — | | | JSON object of additional metadata to attach to the root span. Standard metadata keys take precedence on conflict. | +### Settings -`enable_tools` and `debug` control local plugin behavior and can still be set -from the environment. Tracing routing and enablement (`trace_to_braintrust`, -`profile`, `project`, `org_name`, `additional_metadata`) come only from -`braintrust.json` and `bt trace run`. +| Config key | Default | Purpose | +|---|---|---| +| `trace_to_braintrust` | `false` | Enable tracing | +| `enable_tools` | `true` | Register Braintrust tools; override with `BRAINTRUST_OPENCODE_ENABLE_TOOLS` | +| `route.auth.profile_id` | unset | Saved profile ID written by `bt` setup | +| `route.auth.profile` | current `bt` profile | Select a profile by name | +| `route.auth.org_name` | profile default | Select an organization | +| `route.destination` | project logs in `opencode` | Select the trace destination | +| `route.additional_metadata` | unset | Add root-span metadata | +| `route.flush_mode` | `fire_and_forget` in the default route | Control delivery flushing | +| `debug` | `false` | Enable debug logging; override with `BRAINTRUST_DEBUG` | -### Precedence +Older files can still use top-level `profile`, `org_name`, `project`, and +`additional_metadata`. For new files, use `bt trace enable` or the nested +`route` format above. Include a destination when supplying a route. -Configuration is loaded with the following precedence (later overrides earlier): +Only `enable_tools` and `debug` read environment variables directly. Tracing +settings come from these files or `bt trace run`. -1. Default values -2. `~/.config/opencode/braintrust.json` (global config) -3. `.opencode/braintrust.json` (project config) -4. `bt trace run` invocation settings (tracing only, per-invocation, highest - priority; never written back to the config files) - -## Disabling Braintrust Tools +## Disable Braintrust tools Set `enable_tools` to `false` to trace OpenCode sessions without registering Braintrust-branded tools (`braintrust_query_logs`, `braintrust_list_projects`, `braintrust_log_data`, `braintrust_get_experiments`): @@ -83,7 +89,9 @@ Set `enable_tools` to `false` to trace OpenCode sessions without registering Bra { "trace_to_braintrust": true, "enable_tools": false, - "project": "my-project" + "route": { + "destination": { "type": "project_logs", "project_name": "my-project" } + } } ``` @@ -93,9 +101,9 @@ Or use the environment variable: BRAINTRUST_OPENCODE_ENABLE_TOOLS=false opencode ``` -## Adding Dynamic Metadata +## Add root metadata -Set `additional_metadata` via the config file to attach custom key-value pairs to the root span. This is useful for tagging traces in CI or linking them back to a specific run. +Set `route.additional_metadata` to add fields to the root span: For one invocation without changing the persistent configuration, use `bt trace run --additional-metadata '{"ci": true, "run_id": "abc-123"}' opencode -- run "do the thing"`, @@ -106,36 +114,50 @@ You can also set it via the config file: ```json { - "additional_metadata": { - "team": "platform" + "route": { + "destination": { "type": "project_logs", "project_name": "my-project" }, + "additional_metadata": { "team": "platform" } } } ``` -The value must be a JSON object. Any keys that conflict with standard root span metadata (`session_id`, `workspace`, `directory`, `hostname`, `username`, `os`) will be overridden by the standard values. +The value must be a JSON object. Built-in session metadata wins if keys conflict. -## Trace Structure +## Trace structure -Sessions are traced with the following hierarchy: +The daemon reconstructs sessions, turns, model messages, and tool executions: +```text +Session (task) +├── Turn 1 (task) +│ ├── Model response (llm) +│ ├── Tool execution (tool) +│ └── Model response (llm) +└── Turn 2 (task) ``` -Session (task span) -├── metadata: session_id, workspace, hostname, username, os -├── Turn 1 (task span) -│ ├── input: "user message" -│ ├── metadata: turn_number, agent, model -│ ├── Tool 1 (tool span) -│ │ ├── input: tool arguments -│ │ └── output: tool result -│ └── Tool 2 (tool span) -├── Turn 2 (task span) -│ └── ... -└── metrics: total_turns, total_tool_calls + +Model usage and metadata are included when OpenCode provides them. Child +sessions are attached to their parent's active turn when that relationship is +available to the translator. + +## Manage tracing + +```bash +bt trace doctor opencode +bt trace status +bt trace update opencode +bt trace disable opencode ``` ## Runtime architecture -The package never calls the Braintrust API from JavaScript. Tracing forwards -native events over local JSON-RPC to `bt-daemon`. The four optional data-access -tools invoke non-interactive `bt` CLI commands. In both cases, `bt` owns profile -selection, credential storage, refresh, backend resolution, and API transport. +Tracing sends events to the local daemon over JSON-RPC. The four optional +Braintrust tools call `bt` commands. Both use `bt` for credentials and API access. + +## Development + +From the monorepo root, run `make validate-opencode` to build, check, and test +the package. See the +[contribution guide](https://github.com/braintrustdata/braintrust-coding-agent-plugins/blob/main/src/plugins/opencode/content/CONTRIBUTING.md) +for development instructions. The npm package contains compiled entrypoints; +source changes belong in the monorepo. diff --git a/src/plugins/pi/content/README.md b/src/plugins/pi/content/README.md index bb9c9c0..37d88ec 100644 --- a/src/plugins/pi/content/README.md +++ b/src/plugins/pi/content/README.md @@ -2,22 +2,16 @@ [![npm version](https://img.shields.io/npm/v/%40braintrust%2Fpi-extension)](https://www.npmjs.com/package/@braintrust/pi-extension) -Braintrust extension for [pi](https://github.com/earendil-works/pi-coding-agent). - -Today this extension automatically traces pi sessions, turns, model calls, and tool executions to Braintrust. -The extension forwards native pi events to the installed `bt` tracing daemon; -all span construction, authentication, recovery, and Braintrust delivery happen -inside the daemon. - -Version 1 requires a current `bt` CLI with the Pi daemon translator. If `bt` or -the translator is unavailable, tracing fails open and Pi keeps running. +Trace [Pi](https://github.com/earendil-works/pi-coding-agent) sessions in +Braintrust. The extension sends events to the local `bt` daemon, which builds +and uploads traces. Pi keeps running if tracing fails. ## What gets traced -- **Session spans**: one root span per pi session that actually produces at least one turn +- **Session spans**: one root span per Pi session with at least one turn - **Turn spans**: one span per user prompt / agent run -- **LLM spans**: one span per model response inside a turn, including canonical token, cache, reasoning, estimated-cost, and time-to-first-token metrics -- **Tool spans**: one span per tool execution, including tools activated through pi's dynamic/deferred tool-loading flow +- **LLM spans**: one span per model response inside a turn, with token usage, cache usage, reasoning tokens, estimated cost, and time to first token +- **Tool spans**: one span per tool execution, including dynamically loaded tools - **Compaction spans**: one span per session compaction, including trigger/retry metadata when available - **Branch summary spans**: one span per summarized `/tree` navigation branch @@ -35,66 +29,49 @@ Session (task) └── Turn 2 (task) ``` -## Install +## Quickstart -### From npm +Install Pi and the +[Braintrust CLI](https://www.braintrust.dev/docs/reference/cli/quickstart), then run: ```bash -pi install npm:@braintrust/pi-extension +bt login +bt trace enable pi --project my-coding-agent +pi ``` -### From this repo +This installs the extension and saves its configuration. Use `--profile` or +`--org` to choose a profile or organization. Restart Pi if it is already open. +The footer shows tracing status and a link to the trace when available. -```bash -pi install . -``` +To install the npm extension separately, use +`pi install npm:@braintrust/pi-extension`, then configure tracing with +`bt trace enable pi`. -Or load it just for one run: +For one invocation without changing global tracing configuration: ```bash -pi -e . +bt trace run --project my-coding-agent pi -- -p "summarize this repository" ``` -## Compatibility - -This package supports the **latest patch release from each of the last five stable pi minor versions**, currently excluding pi versions before `0.65.0`. - -Our GitHub Actions compatibility job automatically resolves and tests that compatibility window, so new pi releases are picked up without manually updating the matrix. +The `bt trace run` routing and metadata flags also accept their matching +`BRAINTRUST_*` environment variables; a plain `pi` session's extension does not. +Historical import and live attach are not supported for Pi. -## Quick start - -```bash -bt auth login -bt trace enable pi -pi -``` - -For one invocation without changing Pi's global tracing configuration, use -`bt trace run --project pi -- [PI_ARGS...]`. `bt trace run`'s flags -(`--profile`, `--org`, `--project`, `--additional-metadata`) also accept the -matching `BRAINTRUST_*` environment variable; a plain `pi` session's -extension does not. +## Compatibility -In interactive mode, the footer shows a `Braintrust` status indicator while tracing is active, and a widget below the editor shows a shortened clickable trace link when available. +CI installs the package against the latest patch from each of the last five +stable Pi release lines. The compatibility job resolves these versions on each +run, including releases from Pi's former npm package name when needed. ## Configuration -You can configure the extension with JSON config files. `braintrust.json` is -the only source of persistent tracing configuration; routing and enablement -are never read directly from the environment. - -Config precedence is: +Settings load in this order, with later values taking precedence: -1. defaults +1. Defaults 2. `~/.pi/agent/braintrust.json` -3. `.pi/braintrust.json` -4. `bt trace run` invocation settings (tracing only, per-invocation, highest - priority; never written back to the config files) - -### Config file locations - -- Global: `~/.pi/agent/braintrust.json` -- Project: `.pi/braintrust.json` +3. `.pi/braintrust.json` in the project (or Pi's configured project config directory) +4. `bt trace run` settings for that invocation Example: @@ -109,33 +86,49 @@ Example: } ``` -## Supported settings +### Settings -| Config key | Env var | Default | +| Config key | Default | Purpose | |---|---|---| -| `trace_to_braintrust` | — | `false` | -| `org_name` | — | unset | -| `profile` | — | default `bt` profile | -| `project` | — | `pi` | -| `additional_metadata` | — | `{}` | -| `show_ui` | `BRAINTRUST_SHOW_UI` | `true` | -| `show_trace_link` | `BRAINTRUST_SHOW_TRACE_LINK` | `true` | - -`show_ui` and `show_trace_link` control local display behavior only and can -still be set from the environment. Tracing routing and enablement come only -from `braintrust.json` and `bt trace run`. - -## Notes - -- Project config overrides global config. -- Project config follows pi's configured project config directory, which defaults to `.pi`. -- The extension does not persist local span state; recovery and incomplete-operation cleanup are owned by the daemon journal. -- Span construction and Braintrust delivery run in the installed `bt` tracing daemon. -- The extension never reads or stores Braintrust credentials. Profile selection is - non-secret, optional, and resolved by the daemon through `bt` authentication. -- Provider request tracing is allowlisted to effective model, thinking, output-limit, and tool-count settings; full provider payloads and thinking signatures are never logged. -- If Braintrust is unavailable, pi should continue working normally. - -## Contributing - -See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup, validation, and repository conventions. +| `trace_to_braintrust` | `false` | Enable tracing | +| `route.auth.profile_id` | unset | Saved profile ID written by `bt` setup | +| `route.auth.profile` | default `bt` profile | Select a profile by name | +| `route.auth.org_name` | profile default | Select an organization | +| `route.destination` | project logs in `pi` | Select the trace destination | +| `route.additional_metadata` | unset | Add root-span metadata | +| `route.flush_mode` | `flush_on_turn_end` in the default route | Control delivery flushing | +| `show_ui` | `true` | Show the status indicator; override with `BRAINTRUST_SHOW_UI` | +| `show_trace_link` | `true` | Show the trace link; override with `BRAINTRUST_SHOW_TRACE_LINK` | + +Older files can still use top-level `profile`, `org_name`, `project`, and +`additional_metadata`. For new files, use `bt trace enable` or the nested +`route` format above. Include a destination when supplying a route. + +Only the display settings read environment variables directly. Tracing settings +come from these files or `bt trace run`. Credentials are managed by `bt`. + +Provider request metadata is limited to model, thinking, output-limit, and +tool-count settings. Full provider payloads and thinking signatures are omitted. + +## Manage tracing + +```bash +bt trace doctor pi +bt trace status +bt trace update pi +bt trace disable pi +``` + +## Local development + +From the monorepo root, build the extension before loading it locally: + +```bash +make build-pi +pi -e ./dist/pi/dist/index.mjs +``` + +This loads the extension for one run; it still needs an enabled Braintrust +configuration. Run `make validate-pi` for package checks. See the +[contribution guide](https://github.com/braintrustdata/braintrust-coding-agent-plugins/blob/main/src/plugins/pi/content/CONTRIBUTING.md) +for source development instructions.