Complete OpenAI Responses support for GPT-5.6 prompt caching and remote compaction

[521ox]

TL;DR

BitFun's cache-first architecture performs well with DeepSeek, but my
GPT-5.6 Responses workloads showed only 7.50%-17.29% cache-read coverage.
The current Responses adapter also lacks first-class explicit caching,
cache-write accounting, remote compaction, and durable checkpoint replay.

I have a local OpenCode-derived reference implementation and would like to
confirm whether the maintainers would accept a phased architecture
contribution, starting with observability only.

Problem / opportunity

BitFun already has a strong cache-first architecture, and its published
DeepSeek benchmark demonstrates that stable prompt construction can achieve
very high KV-cache reuse.

However, while testing BitFun v0.2.17 with OpenAI Responses-compatible
GPT-5.6 deployments, I observed a significant protocol-specific gap:

• DeepSeek requests achieved approximately 95.78% cache-read coverage.
• Two GPT-5.6 Responses deployments achieved approximately 17.29% and 7.50%.
• The aggregate cache-read rate was approximately 68.93%, which was close to
  the roughly 65% rate shown by the upstream provider dashboard.

This does not necessarily contradict BitFun's existing cache benchmark.
Instead, it suggests that the existing cache-first work is effective for some
providers, while the OpenAI Responses path currently lacks protocol features
required by newer GPT models.

This proposal covers two related but separately implementable areas:

1. GPT-5.6 explicit prompt caching and cache-write accounting.
2. OpenAI Responses remote/server-side compaction and checkpoint lifecycle.

Related issue:

• perf(core): Prefix-Cache Optimization Roadmap — Cache-First Architecture for DeepSeek/Claude/Gemini #857

I believe this should be treated as a follow-up to #857 rather than a duplicate.
#857 focused primarily on deterministic prefixes for DeepSeek, Claude, and
Gemini. This proposal concerns newer OpenAI Responses protocol semantics and
the state lifecycle required by remote compaction.

---

Reproduction evidence

The following values were collected from BitFun's own local token-usage
statistics. No prompts, API keys, or user content are included.

Model path	Input tokens	Cache-read tokens	Cache-read ratio
DeepSeek deployment	26,608,952	25,485,312	95.78%
GPT-5.6 Responses deployment A	12,351,052	2,135,552	17.29%
GPT-5.6 Responses deployment B	1,248,544	93,696	7.50%
Combined	40,208,548	27,714,560	68.93%

A representative GPT-5.6 request reported:

• Input tokens: 330,895
• Cache-read tokens: 25,088
• Effective cache-read ratio: approximately 7.58%

Adjacent requests in the same long-running session reported cache-read token
counts similar to:

0
25,088
0
178,688
0
25,088

The visible request characteristics remained stable across these requests:

• Same model
• Same reasoning setting
• Same system instructions
• Same ordered tool-name list
• Same visible reminder blocks
• store: false

I do not have enough evidence to claim that every serialized tool schema byte
was identical, and upstream TTL or routing policy may also contribute.
Therefore, the first proposed contribution is observability, rather than an
assumption that every miss is caused by BitFun.

Current implementation observations

Relevant source areas include:

• src/crates/adapters/ai-adapters/src/providers/openai/responses.rs
• src/crates/assembly/core/src/agentic/execution/execution_engine.rs
• src/crates/assembly/core/src/agentic/execution/round_executor.rs
• src/crates/assembly/core/src/agentic/session/compression/
• src/crates/execution/agent-runtime/src/prompt.rs
• src/crates/execution/agent-stream/src/unified.rs

1. OpenAI Responses request options are not first-class protocol fields

The current Responses request builder owns fields such as:

• model
• input
• instructions
• stream
• max_output_tokens
• reasoning
• tools

Arbitrary fields can be injected through the custom request-body mechanism,
but fields such as the following do not appear to be modeled as typed,
runtime-owned protocol capabilities:

• prompt_cache_key
• prompt_cache_options
• explicit cache breakpoint or TTL configuration
• previous_response_id
• context_management
• compact_threshold

Custom request-body injection is useful for simple stateless fields, but it
cannot implement response-side state management, durable checkpoints,
continuation, recovery, or replay.

2. Cache-read and cache-write accounting are not equivalent

GPT-5.6 explicit prompt caching can expose both:

• cache-read tokens
• cache-write tokens

Cache writes can have different billing semantics from uncached input and
cache reads.

BitFun already has provider-neutral cache usage structures, but the OpenAI
Responses path should explicitly parse and surface GPT-5.6 cache-write usage
rather than treating cache behavior as only a cache-hit/read metric.

For example, the UI and persisted usage records should be able to distinguish:

uncached_input_tokens
cache_read_tokens
cache_write_tokens
output_tokens
reasoning_tokens

A zero cache-write value should mean that the provider reported zero, rather
than that BitFun did not parse the field.

3. Prompt stability needs protocol-specific observability

BitFun already contains comments warning that changing tool definitions between
turns changes the request prefix and causes provider KV-cache misses.

Potential prefix boundaries include:

• system instructions
• prepended reminders
• deferred-tool listing
• skill listing
• agent listing
• runtime context
• user/workspace context
• ordered and serialized tool schemas
• historical messages

Because the reminder block is inserted before conversation history, a small
change in this block can invalidate cache reuse for the entire subsequent
history.

I suggest adding privacy-preserving request-prefix diagnostics:

• Hash of canonical system instructions
• Hash of canonical reminder block
• Hash of ordered serialized tool schemas
• Hash of the stable history prefix
• Byte length or token length of each section
• First section whose hash changed from the previous request
• Provider-reported cache-read and cache-write tokens

The diagnostics should never persist prompt contents by default.

This would let users distinguish among:

• BitFun prefix instability
• intentional prompt/tool changes
• local compaction
• provider TTL expiration
• provider-side routing differences
• unsupported explicit caching fields

4. Local compaction currently creates a new prefix

BitFun's current local compaction replaces older history with a newly generated
summary and invalidates the local prompt cache.

That behavior is valid for local summarization, but it inevitably creates a
new provider prefix and temporarily reduces remote KV-cache reuse.

Local summarization and OpenAI remote compaction should therefore be treated as
two distinct strategies:

Local compaction
    -> BitFun generates a summary
    -> historical bytes change
    -> provider prefix changes

Remote Responses compaction
    -> provider emits an opaque compaction/checkpoint item
    -> BitFun persists and replays that opaque item
    -> provider controls the compacted representation

Requested OpenAI Responses capabilities

A. Explicit prompt caching for GPT-5.6-compatible providers

Please consider first-class support for:

• prompt_cache_key
• provider-supported prompt_cache_options
• explicit caching mode
• provider-supported cache TTL
• explicit cache breakpoints, where supported
• cache_write_tokens usage parsing
• separate cache-read and cache-write cost accounting
• cache capability negotiation for OpenAI-compatible providers

These should preferably be typed provider capabilities rather than arbitrary
static request-body patches.

Static body configuration alone cannot safely determine:

• Which part of a dynamically assembled prompt is stable
• Whether a cache key must change after a tool-schema change
• Whether a session fork should share or rotate a cache identity
• Whether local compaction invalidates the cache identity
• Whether a provider actually supports explicit caching
• How cache writes should be recorded and billed

B. Automatic remote compaction

For compatible Responses endpoints, support request fields such as:

{
  "context_management": [
    {
      "type": "compaction",
      "compact_threshold": 300000
    }
  ]
}

The threshold should be computed dynamically from:

• model context limit
• reserved output tokens
• configured safety reserve
• provider capabilities

This should not be implemented only as a static custom request-body field,
because the response can contain protocol items that require runtime handling.

C. Manual remote compaction

Support the provider's Responses compaction endpoint, where available:

POST /responses/compact

The returned compaction output should be validated and stored as provider-owned
opaque state.

D. Streaming compaction checkpoints

The Responses stream may emit compaction-related output items.

BitFun would need to:

1. Recognize the beginning of a remote compaction boundary.
2. Capture the ordered opaque output items.
3. Persist them transactionally.
4. Materialize incomplete streamed output after interruption.
5. Replay the exact items in the next request.
6. Reject malformed or incompatible checkpoint items.
7. Avoid converting opaque provider items into lossy text summaries.

E. Remote-state binding

A remote checkpoint should be bound to the execution identity that created it,
for example:

• provider identity
• model identity
• protocol/SDK identity
• base URL
• account or credential reference
• organization/project, when applicable
• transport type
• configuration revision

If this identity changes, BitFun should not silently replay the old opaque
checkpoint to a different route.

The product can either:

• fail closed and request an explicit user action, or
• fall back to local compaction after clearly recording the boundary

but it should not silently mix incompatible remote state.

F. Session lifecycle integration

Remote checkpoint state should have defined behavior for:

• application restart
• interrupted stream
• retry
• session restore
• session fork
• undo/revert
• session deletion
• project relocation
• export/share
• provider or model change

Opaque or encrypted checkpoint data should be redacted from exported/shared
transcripts unless the user explicitly requests a private diagnostic export.

G. Optional stored Responses continuation

This can be a separate later phase.

For providers that support it, BitFun could optionally implement:

• store: true
• previous_response_id
• bootstrap versus active continuation states
• cursor persistence
• invalid/expired cursor recovery
• provider retention semantics

Stored continuation should not be required for remote compaction, and it should
remain independently configurable because it has different privacy and
retention implications.

Reference implementation available

We have a local, unpublished implementation derived from OpenCode v1.18.9.

This is not a claim about upstream OpenCode functionality. It is a private
implementation that can be used as a design and test reference.

It currently includes:

• Typed OpenAI Responses request options
• prompt_cache_key
• GPT reasoning and encrypted reasoning-item replay
• Dynamic context_management and compact_threshold
• Manual /responses/compact
• Ordered remote compaction-item capture
• Durable checkpoint persistence
• Crash/interruption recovery
• Provider/model/config route binding
• Exact opaque-item replay
• Export redaction
• Optional store and previous_response_id continuation
• Protocol and session lifecycle tests

If the BitFun maintainers are interested, we can contribute the relevant design
notes and test scenarios without copying BitFun-incompatible architecture
directly.

Proposed architecture direction

Rather than extending custom_request_body until it becomes an implicit
protocol implementation, would the maintainers be open to introducing a small
OpenAI Responses protocol boundary?

One possible responsibility split would be:

ai-adapters

Own provider wire semantics:

• typed request fields
• request lowering
• stream event parsing
• usage parsing
• provider capability detection
• compaction-item validation

agent-stream

Expose structured execution events such as:

• remote compaction started
• remote checkpoint reset
• remote checkpoint item received
• cache-read usage
• cache-write usage

The event type can remain provider-neutral where the lifecycle semantics are
shared, while retaining opaque provider payloads.

agent-runtime / session contracts

Own session semantics:

• checkpoint identity
• compatibility validation
• replay contract
• continuation state
• fork/revert/delete behavior

Persistence/service layer

Own durable state:

• ordered checkpoint ledger
• atomic append/materialization
• restart recovery
• deletion and cleanup
• export redaction

Execution engine / coordinator

Own policy:

• when to request remote compaction
• threshold calculation
• local-versus-remote compaction selection
• fallback behavior
• provider/model transition behavior

This would keep provider-specific JSON out of the general orchestration layer
while preventing the adapter from becoming the owner of session lifecycle.

Suggested phased contribution plan

To keep review scope bounded, I propose separate PRs.

Phase 1 - Observability only

• Record cache-read and cache-write usage separately
• Add privacy-preserving prompt-section hashes
• Add deterministic tool-schema fingerprinting
• Add cache diagnostics to request traces
• Add tests proving whether adjacent requests preserve the intended prefix

No remote-state behavior would change in this phase.

Phase 2 - Typed explicit prompt caching

• Add typed Responses cache options
• Add provider capability negotiation
• Add session/cache identity policy
• Parse cache_write_tokens
• Add cache cost accounting
• Preserve backwards compatibility with existing custom request-body settings

Phase 3 - Remote compaction protocol

• Add typed context_management
• Add manual /responses/compact
• Parse streamed compaction/checkpoint items
• Introduce checkpoint events and validation
• Initially guard the behavior behind an experimental capability flag

Phase 4 - Durable checkpoint lifecycle

• Persist ordered opaque checkpoint items
• Recover interrupted writes
• Replay exact checkpoint items
• Bind checkpoints to route identity
• Integrate restore/fork/revert/delete/export behavior

Phase 5 - Optional stored continuation

• Add store
• Add previous_response_id
• Persist continuation cursors
• Handle expiration and invalid cursors
• Document privacy and retention behavior

Proposed acceptance criteria

• Existing Chat Completions, Anthropic, Gemini, and DeepSeek paths do not regress.
• Providers that do not advertise the new capability retain current behavior.
• Unsupported remote compaction fails closed or uses an explicitly documented
  local fallback.
• Cache-read and cache-write usage are reported separately.
• Stable request sections have deterministic fingerprints.
• Tool ordering and serialization are deterministic.
• Opaque checkpoint items survive application restart without transformation.
• Checkpoints are not replayed after an incompatible provider/model/config
  change.
• Session fork, revert, delete, and export behavior is covered by tests.
• No prompt content or API credentials are recorded by cache diagnostics.
• Custom OpenAI-compatible providers can enable capabilities explicitly rather
  than being identified only by hard-coded host names.

Maintainer questions

Before preparing an implementation, could the maintainers clarify:

1. Would an architecture refactor of this scope be acceptable if delivered as
   the phased PRs above?
2. Should remote checkpoint state live in the existing session persistence
   model, or would a separate protocol-state store be preferred?
3. Should OpenAI-compatible providers opt into these capabilities through
   explicit model/provider configuration?
4. Would you prefer an ADR/design document before any code PR?
5. Should the first contribution be limited to observability and cache-write
   accounting?
6. Is optional stored continuation considered in scope, or should the initial
   design support only stateless Responses plus remote compaction?
7. Are maintainers interested in reviewing the tests and architecture notes
   from the local OpenCode-derived implementation?

I would be willing to contribute this work, but I would like to align on the
ownership boundaries and migration strategy before changing the execution and
session-persistence architecture.

References

• Existing BitFun cache-first roadmap:
  perf(core): Prefix-Cache Optimization Roadmap — Cache-First Architecture for DeepSeek/Claude/Gemini #857
• OpenAI latest model guide:
  https://developers.openai.com/api/docs/guides/latest-model
• OpenAI Responses API:
  https://platform.openai.com/docs/api-reference/responses

[wsp1911]

Thanks for the detailed investigation and for providing the measurements and
architecture proposal. We agree that the Responses caching and compaction gaps
are real areas worth addressing.

After reviewing the current implementation, we do not think this should be
implemented as an independent cross-layer PR at this stage. The work would
touch the Responses adapter, stream event handling, provider-neutral usage
contracts, execution/compression policy, and durable session state. In
particular, remote compaction introduces opaque provider-owned checkpoint items
that must be persisted, replayed, recovered, and validated against the provider,
model, and configuration that created them.

We are going to take ownership of the core implementation and align the
ownership and migration boundaries first. Our target is to land the fix before
Monday, August 17. Please hold off on preparing a cross-layer code PR against
the current architecture for now.

Thanks again for the careful analysis and for bringing this to our attention.