Sort submissions by metadata count

[jerbaroo]

This PR makes changes to how the chunk selection query of the PreferDistinct strategy. In the Files changed tab, you can see the impact on the benchmark SVG .

Key changes:

• Includes a test of fairness in the Python test suite.
• We no longer do index walks over all chunks, instead we sort submissions by metadata.
• Already-reserved check is from SQLite via FFI, commit from @ReinierMaas
• Metadata counts are not passed to SQLite via the generated query, but rather checked from SQLite via FFI, commit from @ReinierMaas

[Copilot]

Pull request overview

Moves PreferDistinct selection from chunk scans to metadata-ranked submission selection with indexed chunk lookup.

Changes:

• Ranks submissions using reservation metadata exposed through SQLite FFI.
• Adds submission random-order indexing and simplifies metastate tracking.
• Adds fairness and reservation-filtering tests.

Reviewed changes

Copilot reviewed 12 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
workspace-hack/Cargo.toml	Updates generated dependency features.
opsqueue/src/consumer/strategy.rs	Reworks strategy query generation.
opsqueue/src/consumer/dispatcher/reserver.rs	Exposes reservation lookup.
opsqueue/src/consumer/dispatcher/mod.rs	Registers SQLite FFI callbacks.
opsqueue/src/consumer/dispatcher/metastate.rs	Simplifies counts and adds JSON export.
opsqueue/src/common/submission.rs	Reuses one SQL parameter.
opsqueue/src/common/chunk.rs	Adds signed chunk-index conversion.
opsqueue/migrations/20260803133844_add_random_order_index_to_submissions.up.sql	Adds submission random ordering.
opsqueue/migrations/20260803133844_add_random_order_index_to_submissions.down.sql	Reverts submission random ordering.
opsqueue/Cargo.toml	Adds SQLite FFI dependency.
libs/opsqueue_python/tests/test_roundtrip.py	Tests PreferDistinct fairness.
Cargo.toml	Defines the workspace SQLite dependency.
Cargo.lock	Updates resolved dependencies.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ReinierMaas]

There are some CI failures but the changes in the last fixup don't explain the errors on CI... Are they introduced with the rebase?

[jerbaroo]

There are some CI failures but the changes in the last fixup don't explain the errors on CI... Are they introduced with the rebase?

Somehow I had incorrectly generated the DB migrations. Re-generated opsqueue/opsqueue_example_database_schema.db in d644aa5

[ReinierMaas]

I still have to review: opsqueue/src/consumer/strategy.rs

[ReinierMaas]

LGTM! I left some more changes but I think that covers all.

[jerbaroo]

LGTM! I left some more changes but I think that covers all.

Thanks for the comprehensive review! Educational :)

Comments are addressed and merge conflicts resolved. Will deploy first thing in the morning.

[jerbaroo]

@OpsBotPrime merge and tag

[OpsBotPrime]

Failed to rebase, please rebase manually using

git fetch && git rebase --interactive --autosquash --no-keep-empty origin/master jerbaroo-faster-prefer-distinct

[jerbaroo]

@OpsBotPrime merge and tag

[OpsBotPrime]

Rebased as 1aaa74d, waiting for CI …

[OpsBotPrime]

CI job 🟡 started.

[OpsBotPrime]

@jerbaroo I tagged your PR with v60. Please wait for the build of 1aaa74d to pass and don't forget to deploy it!