Skip to content

fix: re-raise on test results CSV save failure - #20

Merged
rgutzen merged 2 commits into
Lindsay-Lab:mainfrom
rgutzen:fix/test-model-swallowed-csv-write-error
Sep 9, 2026
Merged

fix: re-raise on test results CSV save failure#20
rgutzen merged 2 commits into
Lindsay-Lab:mainfrom
rgutzen:fix/test-model-swallowed-csv-write-error

Conversation

@rgutzen

@rgutzen rgutzen commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

process_single_test was failing with a confusing pandas.errors.EmptyDataError: No columns to parse from file when reading test_outputs.csv, even though the upstream test rule had reported success.

Root cause

In dynvision/runtime/test_model.py::TestingOrchestrator.save_results(), the block that writes test_outputs.csv catches all exceptions and only logs them:

except Exception as e:
    logger.error(f"Failed to save test results: {e}")

If model.storage.get_dataframe() or to_csv() fails partway (disk error, OOM while formatting, quota, interrupted write, etc.), the exception is swallowed and save_results() returns normally. The shell command then exits 0 with a missing/empty/truncated test_outputs.csv on disk. Snakemake treats the output as successfully produced and never reruns the job — the corrupted file then silently propagates to downstream processing steps, which fail later with an unrelated-looking pandas error.

This is inconsistent with the response-tensor save path immediately below it in the same method, which already re-raises on failure specifically so Snakemake can detect it via a missing output file.

Fix

Re-raise after logging in the CSV-save except block, mirroring the existing response-save error handling, so a failed write causes the Snakemake job itself to fail (and be rerun) rather than leaving a corrupt output silently marked complete.

Testing

  • ast.parse syntax check on the modified file.
  • No existing unit test directly covers save_results(); behavior verified by code inspection and by reproducing the downstream symptom (EmptyDataError from an empty test_outputs.csv) against the described failure mode.

save_results() caught exceptions from writing test_outputs.csv and
only logged them, without re-raising. This let the test rule's shell
command exit 0 with a missing or corrupted (e.g. empty/truncated)
test_outputs.csv on disk. Snakemake then treated the output as
successfully produced and never reran the job, so the corrupted CSV
was silently picked up by downstream processing (process_single_test),
failing later with a confusing EmptyDataError.

Mirrors the existing error handling in the response-tensor save path
below, which already re-raises so Snakemake can detect the failure via
a missing output file.
Copilot AI lite review requested due to automatic review settings September 9, 2026 10:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, consistent with the existing response-save error handling in the same method, and directly addresses the described root cause by ensuring Snakemake observes failed writes as job failures.

Pull request overview

This PR fixes a failure-mode in the testing runtime where writing test_outputs.csv could fail silently, leaving behind a missing/empty/truncated CSV that Snakemake still treats as a successful output, causing confusing downstream errors when the corrupted file is read later.

Changes:

  • Re-raise exceptions encountered while saving the test results CSV (after logging), so the Snakemake job fails and can be rerun rather than producing a corrupt output artifact.
File summaries
File Description
dynvision/runtime/test_model.py Re-raises on test_outputs.csv write failures to prevent silently “successful” runs with missing/corrupt outputs.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@rgutzen
rgutzen merged commit d8291d3 into Lindsay-Lab:main Sep 9, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants