Skip to content

[Needs review] Update openai provider types - #451

Draft
github-actions[bot] wants to merge 1 commit into
mainfrom
update-openai-provider-types-25a170bd-34873536812
Draft

github-actions[bot] wants to merge 1 commit into
mainfrom
update-openai-provider-types-25a170bd-34873536812

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Automated update of Lingua provider types.

Provider: openai

Publication mode: needs-review draft

Continuation required

This draft preserves the automation's partial work. One or more validation or advisory checks did not complete successfully; continue from this branch instead of restarting the provider update.

Feedback: comment /bt good or /bt bad to log review feedback to the Braintrust trace.

Human decisions required

The agent completed every unblocked item. These decisions require human input before the update can be finished:

  • model enum member gpt-5.5 (specs/openai/openapi.yml:62019 and Beta mirror :96172) (openai-gpt-5-5-reasoning-effort-family)
    • Question: Which reasoning-effort values does gpt-5.5 accept — i.e. which EffortFamily arm should reasoning_effort_family_for_model return for point release 5? Concretely: does gpt-5.5 support xhigh (matching gpt-5.4) and does it support max (introduced with gpt-5.6)? Secondary: should the same change also close the adjacent hole for non-codex gpt-5.3, which falls through identically today?
    • Evidence: The specification diff adds gpt-5.5 to two model enums (specs/openai/openapi.yml:62019 and :96172) and nothing else — it contains no per-model reasoning-effort enum, so the supported set is not derivable from this update. In crates/lingua/src/providers/openai/capabilities.rs:188-217, reasoning_effort_family_for_model dispatches on the point-release digit from gpt_5_point_release_suffix (:182-186) and has arms for 6, 4, 2, 3-codex, 1-codex, and 1, but none for 5. The fallback at :208-212 cannot catch it because model == "gpt-5" is false and model.starts_with("gpt-5-") fails on the . separator. The predicate therefore returns None, and clamp_reasoning_effort_for_model at :223-226 early-returns the effort unclamped, so a synthesized ReasoningEffort::Max reaches adapter.rs:573 and responses_adapter.rs:1045 verbatim. The omission is invisible because the clamp test table at :519-570 enumerates 5.1, 5.2, 5.2-codex, 5.3-codex, 5.4, 5.6-terra, 5.6-luna, 5, 5-mini, and 5-nano but not 5.5. Sibling arms bracket the answer without settling it: 5.1 -> NoneLowMediumHigh, 5.2 and 5.4 -> NoneLowMediumHighXhigh, 5.6 -> NoneLowMediumHighXhighMax. Corroborating signal that max arrived at 5.6 and not earlier: supports_prompt_cache_breakpoint (:78-95) gates on major.minor >= (5,6). Note gpt-5.5 is already referenced elsewhere in the tree for unrelated predicates — capabilities.rs:465 and crates/braintrust-llm-router/src/catalog/spec.rs:91,96 — so only the effort family is missing.
    • Recommended option: Return EffortFamily::NoneLowMediumHighXhigh for point release 5, matching gpt-5.4. Rationale: the enum orders gpt-5.5 immediately below the 5.6 family, xhigh has been present continuously since 5.2, and max appears to be a 5.6-and-later capability on the same boundary that supports_prompt_cache_breakpoint already encodes as >= (5,6). Implement it by extending the existing starts_with('4') || starts_with('2') arm at :194-196 to include '5' so the dotted and databricks-gpt-5-5 dashed forms are both covered by the shared suffix helper. Please confirm against the OpenAI model documentation before merging — this recommendation is an inference from sibling versions, not a fact stated by the specification.
    • Alternatives:
      • EffortFamily::NoneLowMediumHighXhighMax (treat 5.5 like 5.6). Tradeoff: safe if gpt-5.5 does support max, because nothing is silently downgraded; but if it does not, Lingua emits reasoning.effort: "max" and the request fails at the API with a 400 that surfaces to the caller as a provider error rather than as a Lingua mapping decision. Fails loudly rather than silently, which is the better failure mode of the two wrong answers.
      • EffortFamily::NoneLowMediumHighXhigh (treat 5.5 like 5.4) — the recommendation. Tradeoff: if gpt-5.5 actually supports max, a caller requesting max silently receives xhigh. This is a silent semantic downgrade, which AGENTS.md:19 disfavours, and it is invisible in the response.
      • Leave gpt-5.5 unmatched, i.e. accept today's behavior and document it. Tradeoff: zero code change and exact user-supplied efforts pass through untouched, but it is inconsistent with every other supported point release, and a synthesized cross-provider effort (for example Anthropic thinking mapped to max) would reach the API unvalidated. This also leaves the identical non-codex gpt-5.3 hole in place.
      • Replace the digit-dispatch chain with an explicit model-id-to-family table plus a test that fails when a known id has no entry. Tradeoff: eliminates this whole class of silent fallthrough permanently and would have caught both the 5.3 and 5.5 holes, but it is a larger refactor than a provider type update should carry and still requires the same product fact for gpt-5.5. Recommended as a follow-up, not as part of this change.
    • Likely files:
      • crates/lingua/src/providers/openai/capabilities.rs
      • crates/lingua/src/providers/openai/adapter.rs
      • crates/lingua/src/providers/openai/responses_adapter.rs
      • crates/braintrust-llm-router/src/catalog/spec.rs
    • Validation commands:
      • cargo test -p lingua --lib providers::openai::capabilities::tests::test_clamp_reasoning_effort_for_model
      • cargo test -p lingua --lib providers::openai::capabilities::tests
      • cargo test -p lingua --lib providers::openai
      • cargo test -p braintrust-llm-router --lib catalog::spec::tests
      • make test-payloads
      • make typed-boundary-check

Provider-only changes

These native wire features are accepted and passed through unchanged, but cross-provider transformation is intentionally unsupported:

  • FunctionToolCallOutput.namespace / FunctionCallOutputItemParam.namespace (and Beta mirrors) (openai-function-call-output-namespace)
    • Native contract: Generated and native wire types continue to accept namespace unchanged (generated.rs:6094), and native request/response validation and format detection recognize a function_call_output item carrying it. An unmodified same-format OpenAI request or response containing namespace takes the byte-preserving passthrough path at crates/lingua/src/processing/transform.rs:462-464 and is returned byte-identical. A cross-provider transform of a function_call_output item carrying a non-empty namespace returns an explicit ConvertError::UnsupportedMapping naming the field and item kind, raised before the field can be dropped or coerced. No universal field, no provider-options marker, no opaque replay carrier, and no cross-provider mapping is added.
    • Scope decision: namespace is a provider-defined toolset/registry scope, not portable message content. Three independent signals: the specification groups it with caller / ToolCallCallerParam, explicitly described as the execution context that produced the tool call; the param-side constraint ^[a-zA-Z0-9_-]+$ with maxLength 64 describes an opaque provider-side registry key rather than user-meaningful content; and no other provider has an equivalent scope with compatible semantics. This is provider-defined harness state inside the provider-only boundary. Note the contrast with name in the sibling change: generic caller-defined function-tool identity remains portable and is mapped — only the provider-scoped namespace is provider-only, and it is provider-only because of what it means, not because OpenAI uses a particular wire envelope. Provider-only is a completed scope decision, so this item is unblocked and is not listed as a human blocker.

Validation

  • ./pipelines/generate-provider-types.sh openai: success
  • make generate-types PROVIDER=openai: success
  • Braintrust workflow trace: success
  • Claude integration plan: success
  • Structured plan validation: success
  • Human design blockers: true
  • Claude focused implementation: failure
  • Immutable plan revalidation: success
  • Initial post-implementation Rust regeneration: success
  • Initial provider update path policy: success
  • Initial provider semantic policy: failure
  • Initial post-implementation TypeScript regeneration: success
  • Initial formatting: success
  • Initial focused provider tests: success
  • Initial conditional generator tests: success
  • Initial build: success
  • Initial clippy: failure
  • Bounded Claude mechanical repair: success
  • Effective mechanical validation source: post-repair
  • Effective mechanical validation: success
  • make lingua-wasm: success
  • Unblocked payload capture cases: responsesFunctionCallOutputToolNameParam
  • Live capture (OpenAI): success
  • Live capture (Anthropic): skipped
  • Live capture (Google): skipped
  • Payload cross-provider transform capture: success
  • Payload fixture sync: success
  • make test-payloads: failure
  • make typed-boundary-check: success
  • cargo test -p coverage-report --test cross_provider_test cross_provider_transformations_have_no_unexpected_failures: failure
  • Claude read-only verification: success
  • Verification report validation: success
  • Verification verdict: fail
  • Recoverable binary patch archive: success
  • Patch artifact upload: success

Only safely scoped patches with a locally archived binary diff are published. Any incomplete AI phase or failed deterministic check produces a needs-review draft so work can continue from the PR branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant