API: Report prompt cache and speculative decoding counters in usage stats - #452
Open
DenysAshikhin wants to merge 1 commit into
Open
API: Report prompt cache and speculative decoding counters in usage stats#452DenysAshikhin wants to merge 1 commit into
DenysAshikhin wants to merge 1 commit into
Conversation
…tats The exllamav3 backend already puts cached_tokens on every finish chunk, and draft_accept/draft_reject whenever a draft model is active, but get_usage_stats dropped all three. Today they only reach the console log via gen_logging, so an API client has no way to see prompt cache hits or draft acceptance. Surface them on UsageStats using OpenAI's nested detail objects, so existing OpenAI-compatible clients pick them up without changes: - prompt_tokens_details.cached_tokens. Rounded, since exllamav3 reports it fractional. - completion_tokens_details.accepted_prediction_tokens / rejected_prediction_tokens. Speculative decoding is the same accounting OpenAI defines these for under Predicted Outputs: tokens proposed ahead of time that the model either confirmed or discarded. Both detail objects stay None when the backend does not report them, so responses from a non-speculative model are byte-identical to before. aggregate_usage_stats (the n>1 path) sums the draft counters across generations and takes prompt_tokens_details from the first entry, since the generations share one prompt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is your pull request related to a problem? Please describe.
The exllamav3 backend already computes prompt-cache and draft-token counters for
every request, but they never reach the API nor is there any way for a client to get these stats.
_create_responsebuilds a finish chunk withcached_tokens(backends/exllamav3/model.py:1177) and, whenever a draft model is active,draft_accept/draft_reject(:1185-1191); thenget_usage_statsdrops all three.Today their only consumer is
common/gen_logging.py:93-94, which prints them to the console. An API client has no way to see whether a prompt hit the cache or how well speculative decoding is performing, even though the numbers alreadyexist server-side.
backends/exllamav3/model.py:1184carries a matching# TODO: Add extended draft stats in backend.Why should this feature be added?
Prompt cache hit rate and draft acceptance are the two numbers that explain latency on an exllamav3 deployment. Surfacing them lets clients and proxies report effective cost, spot cache thrashing, and tune
draft_num_tokenswithout scraping server logs.Both use OpenAI's existing nested shape, so OpenAI-compatible clients pick them up with no changes:
usage.prompt_tokens_details.cached_tokensusage.completion_tokens_details.accepted_prediction_tokens/rejected_prediction_tokensExamples
Request (usage is gated on
stream_options.include_usage):{ "messages": [{"role": "user", "content": "<~2000 token prompt>"}], "max_tokens": 200, "stream": false, "stream_options": {"include_usage": true} } Sending the same prompt twice against a 27B exl3 model with MTP drafting (draft_mode: mtp, draft_num_tokens: 3): // call 1: cold "usage": { "prompt_tokens": 1985, "prompt_tokens_details": { "cached_tokens": 0 }, "prompt_time": 1.95, "completion_tokens": 73, "completion_tokens_details": { "accepted_prediction_tokens": 48, "rejected_prediction_tokens": 27 }, "total_tokens": 2058 } // call 2: identical prompt, warm prefix cache "usage": { "prompt_tokens": 1985, "prompt_tokens_details": { "cached_tokens": 1792 }, "prompt_time": 0.17, "completion_tokens": 73, "completion_tokens_details": { "accepted_prediction_tokens": 48, "rejected_prediction_tokens": 27 }, "total_tokens": 2058 } Additional context - cached_tokens is rounded, since exllamav3 reports it fractional. - On the naming: OpenAI defines accepted_prediction_tokens /rejected_prediction_tokens for Predicted Outputs, where rejected tokens are still billed. Speculative decoding is the same accounting - tokens proposed ahead of time that the model confirmed or discarded - so the fields are reused rather than inventing tabby-specific names. Happy to rename them inside completion_tokens_details if you'd prefer they stay distinct. - ruff format --diff and ruff check both clean.