fix: re-raise on test results CSV save failure - #20
Merged
rgutzen merged 2 commits intoSep 9, 2026
Conversation
save_results() caught exceptions from writing test_outputs.csv and only logged them, without re-raising. This let the test rule's shell command exit 0 with a missing or corrupted (e.g. empty/truncated) test_outputs.csv on disk. Snakemake then treated the output as successfully produced and never reran the job, so the corrupted CSV was silently picked up by downstream processing (process_single_test), failing later with a confusing EmptyDataError. Mirrors the existing error handling in the response-tensor save path below, which already re-raises so Snakemake can detect the failure via a missing output file.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, consistent with the existing response-save error handling in the same method, and directly addresses the described root cause by ensuring Snakemake observes failed writes as job failures.
Pull request overview
This PR fixes a failure-mode in the testing runtime where writing test_outputs.csv could fail silently, leaving behind a missing/empty/truncated CSV that Snakemake still treats as a successful output, causing confusing downstream errors when the corrupted file is read later.
Changes:
- Re-raise exceptions encountered while saving the test results CSV (after logging), so the Snakemake job fails and can be rerun rather than producing a corrupt output artifact.
File summaries
| File | Description |
|---|---|
| dynvision/runtime/test_model.py | Re-raises on test_outputs.csv write failures to prevent silently “successful” runs with missing/corrupt outputs. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
process_single_testwas failing with a confusingpandas.errors.EmptyDataError: No columns to parse from filewhen readingtest_outputs.csv, even though the upstream test rule had reported success.Root cause
In
dynvision/runtime/test_model.py::TestingOrchestrator.save_results(), the block that writestest_outputs.csvcatches all exceptions and only logs them:If
model.storage.get_dataframe()orto_csv()fails partway (disk error, OOM while formatting, quota, interrupted write, etc.), the exception is swallowed andsave_results()returns normally. The shell command then exits 0 with a missing/empty/truncatedtest_outputs.csvon disk. Snakemake treats the output as successfully produced and never reruns the job — the corrupted file then silently propagates to downstream processing steps, which fail later with an unrelated-looking pandas error.This is inconsistent with the response-tensor save path immediately below it in the same method, which already re-raises on failure specifically so Snakemake can detect it via a missing output file.
Fix
Re-raise after logging in the CSV-save except block, mirroring the existing response-save error handling, so a failed write causes the Snakemake job itself to fail (and be rerun) rather than leaving a corrupt output silently marked complete.
Testing
ast.parsesyntax check on the modified file.save_results(); behavior verified by code inspection and by reproducing the downstream symptom (EmptyDataError from an empty test_outputs.csv) against the described failure mode.