O4d: prepare 0.0.18.0rc4 release

.github/workflows/ci.yml

@@ -301,6 +301,47 @@ jobs:
       - name: Census every test file against the CI roster
         run: python .travis/test-ci-roster.py
 
+  roster-verify-check:
+    needs: install
+    runs-on: ubuntu-latest
+    # Companion to ci-roster-check, and the answer to a gap that job cannot close: it enforces
+    # that every ungated test file carries a REASON, but not that the reason is TRUE.  That was
+    # not hypothetical -- the roster asserted "skips cleanly without them" for two jax_gp files
+    # while one ERRORED without jax and the other had no guard at all (PR #248), and it carried
+    # two OPTDEP entries whose prose said "unverified on a runner" / "belongs in another job"
+    # when both in fact collect and pass completely.  A reason nobody re-checks is prose, which
+    # is the thing this whole census exists to stop being mistaken for coverage.
+    #
+    # So each status now has a falsifiable predicate and this job runs it: LEGACY must fail to
+    # import, HANDRUN must collect nothing, EXPENSIVE must collect but pass nothing without
+    # RIFT_RUN_EXPENSIVE, and OPTDEP must name its dependencies (`needs:<mod>`) which
+    # requirements.txt must not install.  See .travis/test-roster-verify.py for what is
+    # deliberately NOT checked and why.
+    #
+    # SEPARATE from ci-roster-check on purpose: that one is stdlib-only with no `needs: install`
+    # and must stay that way, so it still reports when the install matrix is broken.  This one
+    # imports RIFT.  ~3.5 min measured on CIT (52 entries, one pytest collection each).
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+          cache: 'pip'
+          cache-dependency-path: requirements.txt
+      - name: Enable symlink
+        run: sudo ln -sf $(which python3) /usr/bin/python
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip --break-system-packages
+          python -m pip install -r requirements.txt --break-system-packages
+          python -m pip install coverage pytest --break-system-packages
+          python -m pip install --editable . --break-system-packages
+      - name: Verify each roster reason still holds
+        env:
+          OMP_NUM_THREADS: 1
+        run: python .travis/test-roster-verify.py
+
   core-unit-check:
     needs: install
     runs-on: ubuntu-latest
@@ -320,8 +361,12 @@ jobs:
     # pattern, so the marker's SCOPE_GLOBS check -- the half that makes a marker fail-closed --
     # would have nothing to scope over.  The census job above is what keeps this set honest.
     #
-    # Python 3.10 to match the sibling numpy jobs.  timeout-minutes is a runaway backstop, an
-    # order of magnitude above the 49 s measured on CIT, not a budget.
+    # Python 3.10 to match the sibling numpy jobs.  timeout-minutes is a runaway backstop,
+    # not a budget -- but the margin is no longer large: the gate measures 350 s on CIT
+    # (was 49 s when this comment first claimed 'an order of magnitude'), because the
+    # per-file collection loop spawns one RIFT-importing interpreter per manifest entry and
+    # grows with the manifest.  20 min is ~3.4x that, not 20x.  Re-measure before adding
+    # many more files, and see .travis/test-core-units.sh for the breakdown.
     timeout-minutes: 20
     steps:
       - uses: actions/checkout@v4
@@ -393,6 +438,15 @@ jobs:
   jax-ile-check:
     needs: install
     runs-on: ubuntu-latest
+    # Sharded 2026-09-08: the suite outgrew the 60-minute cap below.  Only the
+    # execute step splits; the FILES manifest, the collection floor against
+    # EXPECTED_TESTS and the deselect-resolution check run in full in every
+    # shard, so a shard cannot go green on a partial view.  Raise `shard` to
+    # add capacity; the split is round-robin by index in test-jax.sh.
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: [1, 2, 3]
     # test/jax/ was run by NOTHING in this workflow until this job landed (the file had
     # zero matches for "jax"), and two real defects survived a month each behind that
     # gap. See .travis/test-jax.sh for why the gate counts tests instead of just
@@ -456,11 +510,46 @@ jobs:
           python -m pip install --upgrade pip --break-system-packages
           python -m pip install -r requirements.txt coverage pytest "jax[cpu]" numpyro --break-system-packages
           python -m pip install --editable . --break-system-packages
+      # WHY THIS EXISTS.  jax is installed UNPINNED above, so this job's stack is
+      # whatever pip resolved the day it ran, and no developer environment
+      # necessarily has that version.  Without a record, a red gate six weeks old
+      # cannot be reproduced -- you cannot even tell which jax produced it.  Two
+      # failures during the #214 landing were chased locally before a
+      # pristine-base control showed they were the environment, not the branch
+      # (issue #292).  The gate prints jax and numpyro; this captures the rest,
+      # jaxlib and lal included.
+      - name: Record the resolved jax stack
+        run: |
+          python -m pip freeze > pip-freeze-jax-ile.txt
+          echo "== resolved stack (full list in the jax-ile-resolved-stack artifact) =="
+          # importlib.metadata, not a grep of pip freeze: freeze reports a URL
+          # rather than a version for anything not installed from PyPI, so the
+          # grep prints a wheel path in exactly the environments someone is most
+          # likely to be comparing against.
+          python -c "
+          import importlib.metadata as md
+          for n in ('jax', 'jaxlib', 'numpyro', 'numpy', 'scipy', 'numba', 'lal'):
+              try:
+                  print('%-10s %s' % (n, md.version(n)))
+              except Exception:
+                  print('%-10s (absent)' % n)
+          "
       - name: Run jax_ile CPU regression gate
         env:
           JAX_PLATFORMS: cpu
           OMP_NUM_THREADS: 1
+          JAX_GATE_SHARDS: 3
+          JAX_GATE_SHARD: ${{ matrix.shard }}
         run: bash .travis/test-jax.sh
+      # always(): a FAILING gate is the case the record is for, so it must upload
+      # after a red run too, not only after a green one.
+      - name: Upload the resolved jax stack
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: jax-ile-resolved-stack
+          path: pip-freeze-jax-ile.txt
+          if-no-files-found: warn
 
   # CPU ONLY.  GitHub runners have no GPU and no cupy, so this lane cannot
   # exercise the cupy branch of RIFT/likelihood/SphericalHarmonics_gpu.py -- the
@@ -715,6 +804,20 @@ jobs:
           bash .travis/test-run.sh
           bash .travis/test-run-alts.sh
           bash .travis/test-build.sh
+      - name: Run ILE executable selection tests (--use-jax-ile / --ile-exe)
+        # Full subprocess DAG builds against the same reference ini/coinc as
+        # test-build.sh, so the pseudo_pipe CLI wiring is what is exercised:
+        # --use-jax-ile and --ile-exe land in ILE.sub/ILE_puff.sub/ILE_extr.sub,
+        # and --use-jax-ile is refused at DAG-build time with --calmarg-envelope-directory.
+        run: |
+          python -m pytest -q MonteCarloMarginalizeCode/Code/test/test_jax_ile_selectable.py
+      - name: Run q-time-pregrid DAG-build test (--internal-ile-q-time-pregrid-factor)
+        # Full subprocess DAG build against the same reference ini/coinc, same reason as
+        # the step above (PR #281 follow-up review, MAJOR #1): asserts
+        # --internal-ile-q-time-pregrid-factor 8 lands in ILE.sub, ILE_extr.sub, and
+        # ILE_puff.sub, not only in helper_ile_args.txt / args_ile.txt.
+        run: |
+          python -m pytest -q MonteCarloMarginalizeCode/Code/test/test_q_time_pregrid_dag.py
       - name: Upload test logs on failure
         if: failure()
         uses: actions/upload-artifact@v4

.travis/PRECOMMIT.md

@@ -0,0 +1,47 @@
+# CI tiers
+
+`.github/workflows/ci.yml` gates every push/PR with ~22 jobs. That is already large, and
+covering every backend and hardware corner in it would grow it without bound: no GitHub
+runner has a GPU, and the JAX suite already costs real time. This tiers the existing suite;
+nothing here removes or shortens a job.
+
+## Tier 1 -- fast CI, every push/PR (unchanged)
+
+Everything in `ci.yml` runs automatically: install, help-check, import-check,
+dependency-compat-check, sim-manager-check, integrator-gate-accounting-check,
+q-window-stencil-check, ci-roster-check, roster-verify-check, core-unit-check, slowrot-check,
+jax-ile-check, lisa-check, calmarg-check, integration-check, asimov-integration,
+rimsky-integration, test-run, container-dep-canary, container-swig-canary, docs. Plus
+`.gitlab-ci.yml`'s default pipeline (coord/integrate/posterior/run/run-alts/build).
+
+RECOMMENDATION for RO'S: `jax-ile-check` runs on every push, and its cost argues for tier 2 or
+a schedule instead. `.travis/test-jax.sh:494` measures the gated 139-test baseline at 13m53s
+(833s), against a 60-minute timeout; the suite has since grown to 577 tests
+(`.travis/test-jax.sh:766`). `test_angle_marg_exact.py` (36 tests, excluded from this gate at
+`.travis/test-jax.sh:484`) does not add to that per-push cost; run it by hand per the comment
+there. Moving `jax-ile-check` to a nightly schedule would return the 833s to every push, since
+nothing else in tier 1 imports jax. This PR leaves it in place: moving a job is a behavior
+change and belongs in its own PR.
+
+## Tier 2 -- strongly recommended before merging into rift_O4d, by hand on a GPU/big node
+
+No GitHub runner has a GPU, so these either never run, or run a cupy-less CPU parity check
+that cannot see a real device. Run `.travis/precommit-recommended.sh` before merging
+anything touching the ILE likelihood, the NoLoop stencils, calibration marginalization, or
+the JAX driver:
+
+- `RIFT/likelihood/test_q_window_interp_gpu.py`, `test_noloop_gpu_stencils.py` (real cupy)
+- `.travis/test-calmarg-gpu.sh` (fused calibration CUDA kernels)
+- `test/jax/test_angle_marg_exact.py`, and the full `.travis/test-jax.sh`
+- `test/expensive_before_merging/` (`RIFT_RUN_EXPENSIVE=1`)
+
+The script exits 1 when cupy/CUDA or the jax stack is absent; it does not skip.
+
+## Tier 3 -- scheduled or manual only
+
+`.gitlab-ci.yml`'s `gpu_integration` job: a real GPU runner, running `test-integrate.sh` +
+`test-calmarg-gpu.sh` + `test-lisa-gpu.sh` under CUDA. Its `rules:` block
+(`.gitlab-ci.yml:180-184`) has two branches: web-triggered `when: manual`, and
+`$CI_PIPELINE_SOURCE == "schedule"` with `when: on_success`. So it also runs automatically on a
+scheduled pipeline. A nightly `jax-ile-check` would belong on the same schedule if RO'S takes
+the tier-1 recommendation above.