dspark: stop skipping after a cold miss; decode 1-token drafts - #833
Open
audreyt wants to merge 1 commit into
Open
dspark: stop skipping after a cold miss; decode 1-token drafts#833audreyt wants to merge 1 commit into
audreyt wants to merge 1 commit into
Conversation
Immediate no-draft skip (default 3, cold 7) was treating the first confidence prune as a failed cycle and throwing away later 5-token accepts. Turn that skip off. Widen the remaining window to 8 with min-avg 2.0 so low-accept prose still pauses. A 1-token proposal that already matches argmax does not need the batched verifier: run ordinary decode. Metal verify cost is nearly width-flat, so this is the cheap miss path. M5 Max Headroom128 + matching DSpark, --temp 0 --nothink n=128: C add 48.63 -> 53.24 t/s; Redis prose 41.61 -> 43.56 t/s; JSON wash.
There was a problem hiding this comment.
Pull request overview
This PR adjusts DSpark scheduling defaults to avoid prematurely pausing after an initial cold miss, and adds a fast-path for 1-token DSpark proposals that match the current argmax so they can be committed via normal decode instead of running the batched verifier.
Changes:
- Update DSpark scheduler defaults (window=8, min-avg=2.0, and disable immediate no-draft skip by default).
- Add a
draft_n == 1fast-path in the DSpark speculative argmax verifier to commit viads4_session_eval_probe_tp(). - Expand CLI help text for
--dsparkto describe the new default scheduling/fast-path behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| ds4.c | Tweaks DSpark scheduler defaults and adds a 1-token decode fast-path inside DSpark speculative verification. |
| ds4_help.c | Updates --dspark help text to mention the new defaults and 1-token behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| opt(fp, c, "--glm-mtp", "Enable integrated greedy GLM MTP speculation."); | ||
| opt(fp, c, "--glm-mtp-timing", "Enable GLM MTP and print acceptance/timing counters."); | ||
| opt(fp, c, "--dspark", "Enable DSpark using the support GGUF passed with --mtp."); | ||
| opt(fp, c, "--dspark", "Enable DSpark using the support GGUF passed with --mtp. Immediate no-draft skip is off; scheduler window=8 min-avg=2.0. One-token proposals use ordinary decode."); |
Comment on lines
+62698
to
+62715
| if (draft_n == 1) { | ||
| if (ds4_session_eval_probe_tp(s, drafts[0], true, err, errlen) != 0) { | ||
| return -1; | ||
| } | ||
| if (n_accept < accepted_cap) accepted[n_accept++] = drafts[0]; | ||
| if (stats_enabled) { | ||
| s->dspark_stats.full_accepts++; | ||
| s->dspark_stats.accepted_draft_tokens += 1; | ||
| ds4_dspark_stats_note_len(s->dspark_stats.accepted_len_hist, 1); | ||
| } | ||
| ds4_session_dspark_scheduler_note(s, 1, false, DS4_DSPARK_SCHED_EXTRA_MS()); | ||
| if (spec_log) { | ||
| fprintf(stderr, "ds4: DSpark spec one-token decode accepted=%d\n", | ||
| n_accept); | ||
| } | ||
| DS4_DSPARK_STATS_FINISH(); | ||
| return n_accept; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Immediate no-draft skip (default 3, plus cold-7) treats the first confidence prune as a failed cycle and throws away later 5-token accepts. This PR turns that skip off and widens the remaining window to 8 with min-avg 2.0 so low-accept prose still pauses.
A 1-token proposal that already matches argmax does not need the batched verifier: run ordinary decode. Metal verify cost is nearly width-flat, so this is the cheap miss path.
Measured (M5 Max 128 GiB, Headroom128 + matching DSpark support,
--temp 0 --nothinkn=128)Clean branch
25f520bvsorigin/main84cc882, same binary pair, one A/B:addEarlier two repeats of the same policy (before isolating onto this branch) were C add 53.39/53.43 vs 50.56/50.68 and prose 44.55/44.23 vs 42.13/42.06. JSON is a wash either way.
./ds4_test --metal-kernels→gtest: ok.Not included: path-select pair walk and
choose_kdepth policy. Both were measured losses or no-ops on this Metal path.