server: fix prompt-cache miss on whitespace-only thinking channels - #856
Open
maxness wants to merge 1 commit into
Open
server: fix prompt-cache miss on whitespace-only thinking channels#856maxness wants to merge 1 commit into
maxness wants to merge 1 commit into
Conversation
DeepSeek thinking-mode clients pad an empty reasoning channel with a single space so the echo of reasoning_content is not rejected as an empty string. On replay the prompt is re-rendered as " thinking<space> response", but the model actually samples the opening tag immediately followed by the closing one with no space. That extra space token desyncs the byte-prefix match and invalidates the whole live KV prefix, forcing a full re-prefill on every turn. Render a whitespace-only thinking channel back as empty so the reconstituted prompt matches the clean tag boundary the live KV was sampled from. Add a regression test. While here, drop the verbose --trace cache-miss diagnostics block: the trace files it produced were not enough to diagnose these misses and the live server warning already reports the relevant numbers. Also remove the mistaken AGENT.md (superseded by local, gitignored AGENTS.md) and ignore local agent/KV-investigation/ops cruft.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a prompt-cache miss that hit every turn of a DeepSeek thinking-mode conversation when the assistant produced an empty reasoning channel.
Root cause
Thinking-mode clients pad an empty reasoning channel with a single space so the echoed
reasoning_contentisn't rejected as an empty string. On the next request the conversation is re-rendered as:but the model actually samples the opening tag immediately followed by the closing tag, with no space between them. That extra space token desyncs the byte-prefix match, so the live KV prefix is invalidated and the server falls back to a full re-prefill on every turn.
Fix
Render a whitespace-only thinking channel back as empty, so the reconstituted prompt matches the clean
thinking responseboundary the live KV was sampled from. Added a regression test (test_thinking_whitespace_renders_empty).Also dropped the verbose
--tracecache-miss diagnostics block: the trace files it produced weren't enough to diagnose these misses, and the live server warning already reports the relevant numbers.Testing
make ds4-serverbuilds clean.ds4-server tests: ok.