feat(validation): report an unexpected exception as a validator fault by simontaurus · Pull Request #151 · OO-LD/oold-python

simontaurus · 2026-09-11T02:55:46Z

Closes #145. Unblocks the remaining eleven sites in #127.

The shape chosen

The fault carries the id of the check that raised it, rather than reporting under a new id of its own. So no new check id enters the public interface - the interface change is the new status, which the MCP status field now documents.

That also makes the report say which part of the validator broke using the same identifier that names what it was trying to establish: roundtrip.generated faulting means frame derivation or the RDF round-trip broke, and nothing has to map a generic validator.fault back to a subject.

What it looks like

With roundtrip forced to raise, over oold-schema/examples:

FAIL  ...\oold-schema\examples
      362 ok, 0 failed, 21 warning(s), 0 skipped, 18 VALIDATOR FAULT(S), across 21 target(s)

  FAULT roundtrip.generated    Address.schema.json: RuntimeError: frame derivation broke
  FAULT variants               Contact.schema.json properties/address/anyOf[0]: RuntimeError: ...

Exit 1. 0 failed is the point: nothing was concluded about the documents. All 21 targets still reported - the run did not abort.

Changes

report.py: FAULT status, Check.is_fault, Report.faults(), counts extended. Report.passed requires no fail and no fault. failure_reasons puts faults first, because a finding tells the reader to change their document while a fault tells them part of the answer is missing.
pipeline.py: a _guard context manager binding a check id lexically to the region it guards, applied per check section. Two places take an early return instead, where the section produces state everything downstream needs (schema.refs, generate.satisfiable) - carrying on would report one defect once per section that tripped over it. Plus a last-resort guard per file in validate_directory.
cli.py: FAULT styled magenta, louder than red, because the reader should not go looking for it in their own document. The count appears only when non-zero - a permanent 0 fault(s) would train people to skip the field.
Status column widened from 4 to 5.

Note on #127

context.remote's catch is still broad, so nothing reaches its guard yet; its comment said the narrowing "waits on #145" and now says what actually remains. Narrowing predicates.py to JsonLdError and retargeting test_a_processor_failure_is_not_downgraded_to_a_coverage_warning, as #145 describes, is #127's work and not done here.

Verification

pytest: 587 passed, 9 skipped, including two new tests - one that a raising check faults under its own id and produces no finding, one that a fault fails the run without discarding the other verdicts
pre-commit: clean
oold-schema/examples unchanged at 380 ok / 0 failed

Both checks that call roundtrip fault when it raises, each under its own id. That is the behaviour, not a leak: both genuinely broke.

A check raising something it does not expect had two possible fates and both were wrong: reported as FAIL it named the user's document for a defect of ours, and left to propagate it discarded the verdicts already computed for every other target in the run. - new `fault` status, fatal to the verdict like `fail` but distinct from it: no verdict was produced, so the document is neither condemned nor cleared - the fault keeps the id of the check that raised it, so no new check id enters the public interface - guards per check section, plus a last-resort guard per file, so one broken check costs one check - faults lead `failure_reasons` and are counted separately in the CLI Closes #145

github-actions · 2026-09-11T02:56:11Z

Release preview

Merging this PR would release v0.20.0 (current: v0.19.0).

Changelog preview (truncated)

## v0.20.0 (2026-09-11)

### Features

- **validation**: Report an unexpected exception as a validator fault
  ([`87a3852`](https://github.com/OO-LD/oold-python/commit/87a3852e371b07ca3e639b80a34b0201bbc447ba))

### Testing

- **validation**: Cover genuine remote retrieval and the warm cache
  ([#139](https://github.com/OO-LD/oold-python/pull/139),
  [`7fd050f`](https://github.com/OO-LD/oold-python/commit/7fd050f513dafe715d5fe6dfae6bdf0e1db788a8))

- **validation**: Cover the fault paths the guard-shaped tests missed
  ([`a243160`](https://github.com/OO-LD/oold-python/commit/a24316079d4f40b0ff8c7dd244d66b6b1bb02e80))

_{Preview via python-semantic-release and conventional commits.}

github-actions · 2026-09-11T02:57:02Z

📊 Benchmark Results

Click to see benchmark comparison

📊 Benchmark Comparison (threshold: 1.3x)
============================================================

➖ Unchanged (within threshold):
  ➖ test_simple_dict_document_store: 0.0015s → 0.0015s (-0.2%)
  ➖ test_sqlite_document_store: 0.0017s → 0.0017s (-0.5%)
  ➖ test_local_sparql_store: 0.0338s → 0.0336s (-0.4%)
  ➖ test_oneof_subschema: 0.0535s → 0.0532s (-0.5%)
  ➖ test_enum_docstrings: 0.0445s → 0.0448s (+0.7%)
  ➖ test_subclass_inheritance: 0.0479s → 0.0476s (-0.5%)
  ➖ test_class_hierarchy: 0.0452s → 0.0451s (-0.2%)
  ➖ test_core[v1]: 0.0331s → 0.0344s (+4.1%)
  ➖ test_core[v2]: 0.0385s → 0.0398s (+3.6%)
  ➖ test_schema_generation[v1]: 0.0015s → 0.0015s (+0.5%)
  ➖ test_schema_generation[v2]: 0.0025s → 0.0025s (+1.0%)
  ➖ test_simple_json: 0.0006s → 0.0006s (+1.5%)
  ➖ test_complex_graph: 0.0014s → 0.0014s (+0.6%)

============================================================
Summary: 0 regressions, 0 improvements, 13 unchanged
============================================================

✅ No significant performance regressions

Threshold: 1.3x (30% slower triggers a regression warning)

Note: Benchmarks are informational only and won't fail the build.

_{💡 Tip: Download the benchmark-results artifact for detailed JSON data}

codecov · 2026-09-11T02:59:07Z

Codecov Report

❌ Patch coverage is 87.28814% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/oold/validation/pipeline.py	85.4%	8 Missing and 7 partials ⚠️

📢 Thoughts on this report? Let us know!

codecov reported the patch as covered; measuring the lines showed the two bespoke paths were not. Both sections that return early instead of using _guard had no test, and neither did the CLI rendering. Also drops Check.line(), which had no callers and duplicated the CLI's own rendering closely enough that widening the status column had to be done in both.

github-actions · 2026-09-11T03:36:28Z

📊 Benchmark Results

Click to see benchmark comparison

📊 Benchmark Comparison (threshold: 1.3x)
============================================================

⚠️  Performance Regressions:
  ❌ test_subclass_inheritance: 0.0520s → 0.0795s (+53.0%, ratio: 1.53x)

➖ Unchanged (within threshold):
  ➖ test_simple_dict_document_store: 0.0016s → 0.0016s (-0.7%)
  ➖ test_sqlite_document_store: 0.0017s → 0.0017s (+0.9%)
  ➖ test_local_sparql_store: 0.0372s → 0.0379s (+1.9%)
  ➖ test_oneof_subschema: 0.0577s → 0.0590s (+2.4%)
  ➖ test_enum_docstrings: 0.0544s → 0.0483s (-11.3%)
  ➖ test_class_hierarchy: 0.0486s → 0.0512s (+5.5%)
  ➖ test_core[v1]: 0.0365s → 0.0370s (+1.4%)
  ➖ test_core[v2]: 0.0425s → 0.0420s (-1.2%)
  ➖ test_schema_generation[v1]: 0.0015s → 0.0015s (-0.3%)
  ➖ test_schema_generation[v2]: 0.0026s → 0.0026s (-0.5%)
  ➖ test_simple_json: 0.0006s → 0.0006s (+0.1%)
  ➖ test_complex_graph: 0.0014s → 0.0014s (+0.1%)

============================================================
Summary: 1 regressions, 0 improvements, 12 unchanged
============================================================

⚠️  Regressions detected but not failing build (informational only)

Threshold: 1.3x (30% slower triggers a regression warning)

Note: Benchmarks are informational only and won't fail the build.

_{💡 Tip: Download the benchmark-results artifact for detailed JSON data}

#151 added a status that makes the exit code non-zero and left the how-to saying exit is 0 "only when no check failed", which a fault also breaks.

simontaurus merged commit 70e0102 into main Sep 11, 2026
21 checks passed

simontaurus deleted the feat/validator-fault-status branch September 11, 2026 04:42

simontaurus mentioned this pull request Sep 11, 2026

docs(validation): document the fault status in the how-to #152

Merged

simontaurus added a commit that referenced this pull request Sep 11, 2026

docs(validation): document the fault status in the how-to (#152)

6b31fd3

#151 added a status that makes the exit code non-zero and left the how-to saying exit is 0 "only when no check failed", which a fault also breaks.

Navigation Menu

Uh oh!

feat(validation): report an unexpected exception as a validator fault - #151

feat(validation): report an unexpected exception as a validator fault#151
simontaurus merged 2 commits into
mainfrom
feat/validator-fault-status

simontaurus commented Sep 11, 2026

Uh oh!

github-actions Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

codecov Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

simontaurus commented Sep 11, 2026

The shape chosen

What it looks like

Changes

Note on #127

Verification

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release preview

Uh oh!

github-actions Bot commented Sep 11, 2026

📊 Benchmark Results

Uh oh!

codecov Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 11, 2026

📊 Benchmark Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Sep 11, 2026 •

edited

Loading

codecov Bot commented Sep 11, 2026 •

edited

Loading