Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .agents/skills/instrumentation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ Map the change before editing:
- Maintain isomorphic behavior. Node and browser/bundled paths must use compatible channel implementations and avoid channel-registry mismatches.
- Setup, teardown, and patching must be idempotent. Enabling twice, disabling twice, or applying a patch twice should remain safe.
- Promise/stream behavior must be preserved. Patches need to keep subclass/helper semantics intact.
- Contain instrumentation failures. Extraction/logging bugs should be logged or ignored as appropriate, but must not break the user call path.
- Treat Braintrust logging and span-finalization methods, including `span.log()` and `span.end()`, as non-throwing.
Do not wrap these methods in defensive `try`/`catch` blocks or add fallback logic for hypothetical logging exceptions.
- Use the SDK `debugLogger` for SDK instrumentation diagnostics. Do not call `console.*` directly from instrumentation code; direct console use should stay inside the debug logger implementation or another explicitly justified exception.
- Pass `Error` objects directly as `span.log({ error })` values. The SDK serializes errors correctly, so do not add local helpers that manually turn errors into message/stack strings unless an external API requires a non-`Error` representation.
- Log only the useful surface. Prefer narrow, stable payloads over dumping full request/response objects; exclude redundant or overly large data when possible.
Expand Down
5 changes: 5 additions & 0 deletions .changeset/add-typesafe-instrumentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"braintrust": minor
---

feat: Add instrumentation for `@typesafe-ai/sdk`
9 changes: 5 additions & 4 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ Provider credentials are only required when recording or explicitly running live
- `HUGGINGFACE_API_KEY`
- `COHERE_API_KEY`
- `GROQ_API_KEY`
- `TYPESAFE_API_KEY`

`claude-agent-sdk-instrumentation` also uses `ANTHROPIC_API_KEY`, because it runs the real Claude Agent SDK against Anthropic in the same style as the existing live Anthropic wrapper coverage.

Expand Down Expand Up @@ -142,7 +143,7 @@ The mock Braintrust server captures **outbound** SDK→Braintrust traffic and sn

The cassette layer is backed by the internal `@braintrust/seinfeld` workspace package. For e2e tests, the harness starts a local cassette HTTP server and points provider SDK base URLs at that server.

- **Layer:** `withScenarioHarness(...)` starts a `createCassetteServer()` instance for cassette-enabled scenario runs. Provider base URL env vars (for OpenAI, Anthropic, Anthropic Bedrock, Google, Cohere, Cursor, Groq, HuggingFace, Mistral, OpenRouter, and the GitHub Copilot SDK's OpenAI provider mode) point at route prefixes on that server, so subprocesses and SDK-launched binaries can be captured too.
- **Layer:** `withScenarioHarness(...)` starts a `createCassetteServer()` instance for cassette-enabled scenario runs. Provider base URL env vars (for OpenAI, Anthropic, Anthropic Bedrock, Google, Cohere, Cursor, Groq, HuggingFace, Mistral, OpenRouter, TypeSafe, and the GitHub Copilot SDK's OpenAI provider mode) point at route prefixes on that server, so subprocesses and SDK-launched binaries can be captured too.
- **Auto-engage:** the harness automatically engages the cassette layer when (a) a scenario run has `runContext.variantKey`, (b) a cassette JSON exists for the scenario+variant on disk, OR (c) `BRAINTRUST_E2E_CASSETTE_MODE` is `record` / `record-missing`. Scenarios just need to thread `runContext: { variantKey, originalScenarioDir }` into their runner calls — no other code change required.
- **Provider replay failures:** provider scenarios are not skipped when a cassette is missing. In replay mode, the cassette server still starts from `runContext.variantKey`, injects placeholder provider keys, and fails on cassette misses instead of silently calling a live provider.
- **Mode** is set by `BRAINTRUST_E2E_CASSETTE_MODE`:
Expand All @@ -169,14 +170,14 @@ ANTHROPIC_API_KEY=... AWS_BEARER_TOKEN_BEDROCK=... \
OPENAI_API_KEY=... GEMINI_API_KEY=... \
COHERE_API_KEY=... GROQ_API_KEY=... HUGGINGFACE_API_KEY=... \
MISTRAL_API_KEY=... OLLAMA_API_KEY=... OPENROUTER_API_KEY=... \
CURSOR_API_KEY=... \
CURSOR_API_KEY=... TYPESAFE_API_KEY=... \
pnpm --filter=@braintrust/js-e2e-tests run test:e2e:record
```

After recording, run again **without any provider keys** to confirm the cassette is sufficient:

```bash
unset ANTHROPIC_API_KEY AWS_BEARER_TOKEN_BEDROCK OPENAI_API_KEY GEMINI_API_KEY GOOGLE_API_KEY GOOGLE_GENAI_API_KEY COHERE_API_KEY GROQ_API_KEY HUGGINGFACE_API_KEY MISTRAL_API_KEY OLLAMA_API_KEY OPENROUTER_API_KEY CURSOR_API_KEY
unset ANTHROPIC_API_KEY AWS_BEARER_TOKEN_BEDROCK OPENAI_API_KEY GEMINI_API_KEY GOOGLE_API_KEY GOOGLE_GENAI_API_KEY COHERE_API_KEY GROQ_API_KEY HUGGINGFACE_API_KEY MISTRAL_API_KEY OLLAMA_API_KEY OPENROUTER_API_KEY CURSOR_API_KEY TYPESAFE_API_KEY
pnpm --filter=@braintrust/js-e2e-tests run test:e2e
```

Expand All @@ -188,7 +189,7 @@ After any successful record run, stale cassette variants are cleaned only inside

These scenarios have cassette wiring in place and will use cassettes once they're recorded:

`anthropic-bedrock-instrumentation`, `anthropic-instrumentation`, `openai-instrumentation`, `openai-codex-instrumentation`, `ai-sdk-instrumentation`, `ai-sdk-otel-export`, `claude-agent-sdk-instrumentation`, `cohere-instrumentation`, `cursor-sdk-instrumentation`, `github-copilot-instrumentation`, `google-adk-instrumentation`, `google-genai-instrumentation`, `groq-instrumentation`, `huggingface-instrumentation`, `mistral-instrumentation`, `ollama-instrumentation`, `openrouter-agent-instrumentation`, `openrouter-instrumentation`, `wrap-langchain-js-traces`.
`anthropic-bedrock-instrumentation`, `anthropic-instrumentation`, `openai-instrumentation`, `openai-codex-instrumentation`, `ai-sdk-instrumentation`, `ai-sdk-otel-export`, `claude-agent-sdk-instrumentation`, `cohere-instrumentation`, `cursor-sdk-instrumentation`, `github-copilot-instrumentation`, `google-adk-instrumentation`, `google-genai-instrumentation`, `groq-instrumentation`, `huggingface-instrumentation`, `mistral-instrumentation`, `ollama-instrumentation`, `openrouter-agent-instrumentation`, `openrouter-instrumentation`, `typesafe-instrumentation`, `wrap-langchain-js-traces`.

### Cassette format

Expand Down
15 changes: 15 additions & 0 deletions e2e/config/pr-comment-scenarios.json
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,21 @@
}
]
},
{
"scenarioDirName": "typesafe-instrumentation",
"label": "TypeSafe Instrumentation",
"metadataScenario": "typesafe-instrumentation",
"variants": [
{
"variantKey": "typesafe-v0",
"label": "v0 pinned"
},
{
"variantKey": "typesafe-v0-latest",
"label": "v0 latest"
}
]
},
{
"scenarioDirName": "openrouter-instrumentation",
"label": "OpenRouter Instrumentation",
Expand Down
3 changes: 3 additions & 0 deletions e2e/helpers/scenario-harness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,7 @@ function getCassetteServerRoutes(): CassetteServerRoute[] {
{ prefix: "/ollama", upstreamOrigin: "https://ollama.com" },
{ prefix: "/openai", upstreamOrigin: "https://api.openai.com" },
{ prefix: "/openrouter", upstreamOrigin: "https://openrouter.ai" },
{ prefix: "/typesafe", upstreamOrigin: "https://api.typesafe.ai" },
];
}

Expand Down Expand Up @@ -356,6 +357,7 @@ function getCassetteEnv(wiring: ActiveCassetteWiring): Record<string, string> {
OLLAMA_HOST: `${serverUrl}/ollama`,
OPENAI_BASE_URL: `${serverUrl}/openai/v1`,
OPENROUTER_BASE_URL: `${serverUrl}/openrouter/api/v1`,
TYPESAFE_BASE_URL: `${serverUrl}/typesafe`,
};
}

Expand Down Expand Up @@ -410,6 +412,7 @@ const CASSETTE_PROVIDER_KEYS: Array<{
envVars: ["OPENROUTER_API_KEY"],
placeholder: "sk-or-cassette-placeholder",
},
{ envVars: ["TYPESAFE_API_KEY"], placeholder: "cassette-placeholder" },
];

function getProviderKeyPlaceholders(): Record<string, string> {
Expand Down
1 change: 1 addition & 0 deletions e2e/helpers/scenario-installer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const INSTALL_SECRET_ENV_VARS = [
"OPENAI_API_KEY",
"OPENROUTER_API_KEY",
"MISTRAL_API_KEY",
"TYPESAFE_API_KEY",
] as const;

const cleanupDirs = new Set<string>();
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading