feat(speculation): generator contract and bestfirst impl

[behinddwalls]

Summary

Why?

The standard Speculator needs a ranked stream of candidate speculation paths to fund. Nothing enumerates or ranks the path space today.

What?

Adds submitqueue/extension/speculation/generator — the Generator/Iterator contract — plus the bestfirst implementation and mocks.

The generator is a pure enumerator: Generate takes only the queue’s batches and offers every coherent path, including ones whose builds already ran. Suppressing those belongs to the Allocator, which reconciles candidates against stored path sets by ID. The snapshot is a strict contract: every batch a head’s direct dependencies reference must be present, and a snapshot with a missing dependency, empty or duplicate IDs, or a self or repeated dependency errors instead of opening a stream — any defaulting for a batch that is hard to score belongs to the scorer, not the generator.

bestfirst ranks each path by how likely all its assumptions are to hold and generates lazily in two levels: each head enumerates its own paths through a per-head stream — extend/swap moves over a cheapest-first flip list, worked out only when the head is first pulled — and a global heap holds each head’s current best, so pulling k paths does O(k) work however large the space behind them. Scores are summed log probabilities so wide heads cannot underflow into ties, every score is summed fresh from the head’s best score so the ordering holds in floating point as computed, and the extend/swap tree emits each path exactly once in non-increasing score order. The full algorithm, tie-breaks, derivation, and floating-point constraints live in doc/rfc/submitqueue/speculation-generator-best-first.md; the package README records only operational behavior.

Also clarifies scorer.Scorer’s contract as final success: the probability that the batch ultimately succeeds — reaches its terminal Succeeded state with its changes landed — not merely that its build passes. That is the quantity bestfirst consumes: a path’s assumptions resolve at terminal batch states, and a Merging dependency has already passed its build yet can still fail to land.

Test Plan

✅ ./tool/bazel test //submitqueue/extension/speculation/...
✅ make fmt
✅ make lint
✅ make check-tidy
✅ make check-gazelle
✅ make check-mocks — fails only on the pre-existing storage/mock/request_batch_store_mock.go header drift that exists on main; the new generator/mock is up to date

[sbalabanov]

accepting to unblock, please address comments