Feat/quality based upsampling - #461

Open

rrutmann wants to merge 37 commits into

mainfrom

feat/quality-based-upsampling

Commits on Aug 17, 2026

feat: quality-based document selection and up/downsampling

rrutmann
and
claude
committed
764bdad
perf: make the quality pipeline fast and shardable

rrutmann
and
claude
committed
d1cf08d
docs: runbook and SLURM scripts for the annealing blend

rrutmann
and
claude
committed
71aa068
docs: record the real-data validation run and correct a claim

rrutmann
and
claude
committed
ce203eb
fix: bound the calibration read instead of scaling it with file count

rrutmann
and
claude
committed
d2ac8f2
feat: timed runner for the quality pipeline

rrutmann
and
claude
committed
76cb919
docs: correct the build-sidecar estimate, which ignored the index pass

rrutmann
and
claude
committed
37611b9

Commits on Aug 18, 2026

fix: spread sidecar tasks across nodes instead of packing them

rrutmann
and
claude
committed
f8728e3
docs: record why the sidecar run took 15 h, not 7 h

rrutmann
and
claude
committed
6e6296c
fix: bound bucket-writer memory by total rows, not per bucket

rrutmann
and
claude
committed
6d9e45d
fix: write bucketing metadata atomically to end a read/write race

rrutmann
and
claude
committed
ed3d4b7
feat: env.sh and a guarded bucket reset for the quality runbook

rrutmann
and
claude
committed
6e2672e
fix: batch the join so it stops re-reading the whole annotation split

rrutmann
and
claude
committed
bf5951e

Commits on Aug 19, 2026

fix: resumable join, and a cube stage that fails clearly

rrutmann
and
claude
committed
60b39bb
fix: preview must not silently scan the sidecar

rrutmann
and
claude
committed
48a7108
fix: pin the source file list a sidecar was built against

rrutmann
and
claude
committed
5733fc4
fix: a resumed join reported zero coverage

rrutmann
and
claude
committed
7b888e0
fix: token estimator was 16-19% out and looked stable while wrong

rrutmann
and
claude
committed
f8afe7b
docs: end-to-end validation of the estimator against a real packing run

rrutmann
and
claude
committed
f793b68
perf: resolve the join in Arrow and drop bucket routing

rrutmann
and
claude
committed
729964f
docs: excalidraw diagram of the data preprocessing pipeline

rrutmann
and
claude
committed
65e718f
perf: raise the join memory request to the measured peak

rrutmann
and
claude
committed
db453bc

Commits on Aug 20, 2026

Commits on Aug 24, 2026

fix: three failures found by the first full production run

rrutmann
and
claude
committed
856a91b
fix: bound join memory by fragment count, not batch size

rrutmann
and
claude
committed
27b82b8
perf: pack many configs per process instead of one CLI call each

rrutmann
and
claude
committed
7ea7f6b

Commits on Aug 25, 2026

feat: verification for a real packed blend, and fix a stale-file trap

rrutmann
and
claude
committed
5df1977
style: clean up the verification scripts

rrutmann
and
claude
committed
48442e1
docs: runbook and sbatch wrappers matching the real production run

rrutmann
and
claude
committed
133f123
test: regressions for the three production-only failures

rrutmann
and
claude
committed
f700ca4
fix: Address GPT-5.6 Sol review
rrutmann
committed
eb618fb
feat: Write out text instead of tokenized data
rrutmann
committed
f31e01b

Commits on Aug 26, 2026

test: Run jsonl export pipeline end-to-end
rrutmann
committed
44076f9
feat: Extend registry to full annealing tree, minus duplicate epochs
rrutmann
committed
a00e213