Skip to content

Record the host in the run logs, and stop losing warnings, tracebacks and diagnostics - #881

Merged
m-reuter merged 11 commits into
Deep-MI:devfrom
m-reuter:log
Sep 15, 2026
Merged

m-reuter merged 11 commits into
Deep-MI:devfrom
m-reuter:log

Conversation

@m-reuter

@m-reuter m-reuter commented Sep 14, 2026

Copy link
Copy Markdown
Member

Two quicktest runs of the same pipeline agreed bit for bit at 1.0mm but not at 0.8mm, and
nothing in the artifacts said which machine either ran on. PyTorch dispatches its CPU kernels
on the ISA the host reports, so two runners of different generations produce different last
bits, which is enough to flip an argmax at a label boundary. Chasing that turned up that the
logs were missing rather more than the host.

What the logs gain

deep-seg.log and recon-surf.log now open with the machine:

Platform: Linux 6.17.0-1022-azure x86_64
CPU: AMD EPYC 9V74 80-Core Processor
CPU cores: 4
Thread limits: OMP_NUM_THREADS=4, ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4

CPU cores reports affinity and names a cgroup quota separately, because docker run --cpus=2
leaves affinity at every core. The hostname is deliberately absent: uname -a, recon-all.log,
recon-all.env and recon-surf.done already carry it, and in a container it is a random string.

Each network then logs one line once its threads and device are final, so the numbers are the
ones the run used rather than torch's defaults:

[INFO: run_prediction.py: 243]: Torch 2.7.1+cpu, CPU capability AVX512, 4 intra-op and 8 inter-op threads

CPU capability is the field that answers the original question: if two runs disagree on it,
last-bit differences are expected. recon-surf.sh, recon-surfreg.sh and
long_prepare_template.sh report the torch build too, because neuroreg imports torch, so their
registration steps run torch kernels.

Things that were reaching no file at all

Measured on a full run by diffing stdout against the log:

  • The callosum module's entire output. It called setup_logging(None, options.verbose) with
    --verbose defaulting to 0, which maps to WARNING, so "Using device", model loading and slice
    selection were dropped from every run. It now takes --seg_log like the other three networks.
  • quick_qc printed nothing whatsoever. Same pattern, and it runs in every pipeline from
    recon-surf.sh, so the total segmentation volume and the ventricle intersection volume it
    computes never reached recon-surf.log. cc_visualization.py had it too.
  • Every warning. The MPS fallback warning fires four times per run on macOS and appeared
    nowhere. logging.captureWarnings(True) in setup_logging fixes it for all callers.
  • Every traceback. The modules that take a log file are run without a tee, so a crash left
    the log ending at the last record. A sys.excepthook now writes it to the log and still
    prints it to the terminal; Ctrl-C stays a clean interrupt.

Plus the remaining 17 stdout print() calls in CerebNet and CorpusCallosum are now logger
records. After all of it, a full --seg_only run loses nothing but tqdm progress bars.

No first-party script uses a counted verbosity flag any more: all are -v for DEBUG, INFO by
default.

Three bugs found on the way

The callosum networks ignored --threads. fastsurfer_cc.py accepted the flag and
run_fastsurfer.sh passed it, but the module never called torch.set_num_threads;
FastSurferCNN/utils/parallel.py only records the value for FastSurfer's own executors. A
--threads 2 run gave the CC networks 4 threads. This changes CC output on any run where
--threads differs from the core count, so the next quicktest will show it. It is the flag
finally doing what the documentation already says.

14 of 19 error messages in run_fastsurfer.sh could never print. time_it ended with
exit, and a function's exit ends the shell it runs in, which is the caller for a plain call
but only a subshell inside a pipeline. So every unpiped call site had a dead error branch. It
now returns and the caller reports which step failed. One call site had no check at all and
relied on the hard exit; it has one now. RunIt goes through $timecmd directly, so
recon-surf.sh is unaffected.

recon-surfreg.sh never set FASTSURFER_HOME in its fallback branch, unlike
recon-surf.sh:47. Line 274 already built PYTHONPATH from it for the version probe, so a
standalone run without the variable exported silently reported VERSION=unknown. Fixed here
because the new host-info line needs the same variable; it is a pre-existing bug rather than
anything this branch introduced.

Testing

New: test/utils/test_host_info.py (23 cases: both /proc/cpuinfo paths, the ARM fallback,
sysctl, all four cgroup shapes, affinity, the no-sched_getaffinity case, and both sides of the
torch import) and test/shell/test_time_it.py (28 cases across piped/unpiped and timed/untimed).
The static half of the latter, which only reads run_fastsurfer.sh, is a lint and lives in
test/lint/test_time_it_call_sites.py. Both were verified against the old behaviour: exactly the
cases that should fail do.

The test/utils job installs no project, so the torch-dependent assertions skip there and the
ImportError branch is asserted instead; with torch present that one skips and the others run.

98 tests pass, ruff clean, bash -n clean on all shell scripts.

@m-reuter m-reuter changed the title Record the host and the kernels torch selected in the run logs Record the host in the run logs, and stop losing warnings, tracebacks and diagnostics Sep 15, 2026
@m-reuter
m-reuter merged commit 1c09bc0 into Deep-MI:dev Sep 15, 2026
6 checks passed
@m-reuter
m-reuter deleted the log branch September 15, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant