ExceptionGraph turns a deceptively difficult question into a reproducible research problem: when policies change, evidence conflicts, and precedents may be wrong, can a system make the safe decision—and know when to abstain?
Each episode is a synthetic operational world with versioned policies, dated evidence, sparse demonstrations, and an explicit decision cutoff. Systems must retrieve the governing evidence, reconstruct the point-in-time chronology, infer the rule, and choose a supported action or abstention. Results are reported by failure mode; no composite score is allowed to hide safety regressions.
Status — pilot-complete research prototype. The generator, scorer, retrieval interfaces, deterministic experiment arms, falsification gates, and E0–E10 runners are implemented. The current end-to-end fixture contains 12 episodes across 4 worlds and is intentionally sized for deterministic pipeline validation—not benchmark conclusions. The core release target is at least 64 worlds and 2,000 scored episodes.
The current corpus is domain-neutral, but its failure modes transfer directly to regulated and financial workflows: an outdated policy is retrieved, a prior approval is mistaken for a rule, a scope exception is missed, or weak evidence is automated instead of escalated. ExceptionGraph makes those tail cases first-class evaluation slices.
| Signal | What the repository demonstrates |
|---|---|
| Quantitative research | Controlled comparisons, pre-specified falsification thresholds, paired resampling, calibration, and risk-coverage analysis |
| Research engineering | Seeded generation, config hashing, manifest-before-execution provenance, family-level splits, and reproducible offline runs |
| Decision safety | Explicit safe-action sets, abstention scoring, false-automation metrics, evidence citations, and adversarial challenge sets |
| Systems design | Interfaces and deterministic offline scaffolding for retrieval, rule induction, verification, multi-agent, and human-approval experiments |
Local verification in this checkout reports:
| Check | Result or scope |
|---|---|
| Offline test suite | 1,651 passed, 4 optional tests deselected |
| Static typing | mypy strict: clean across 86 source files |
| Linting | Ruff: clean |
| CI workflow | Configured for Python 3.11 and 3.12; lint, typing, tests, and coverage |
| Experiment surface | 11 staged runners, E0 through E10 |
The verification proves implementation and deterministic harness behavior. It does not claim model quality, economic value, production readiness, or benchmark-scale results. The linked GitHub Actions workflow provides the corresponding release check on the supported Python versions.
flowchart LR
A[Canonical synthetic world] --> B[Versioned policies and evidence]
B --> C[Seeded noise and adversarial slices]
C --> D[Episode plus gold package]
D --> E{System under test}
E --> E1[Sparse / dense adapters]
E --> E2[Temporal / graph / hybrid adapters]
E --> E3[Induction / verifier / agent harnesses]
E1 --> F[Decision plus cited evidence]
E2 --> F
E3 --> F
F --> G[Slice-aware scorer]
G --> H[Calibration and risk metrics]
G --> I[Paired statistical tests]
H --> J[Pre-specified falsification gates]
I --> J
The central contract is an Episode: a world, query case, decision cutoff, selected
demonstrations, mandatory slice labels, and a gold package containing safe actions,
essential evidence, chronology, contradictions, compatible rules, and abstention reasons.
The experiment sequence is staged so later comparisons cannot silently outrun their prerequisites:
| Stage | Question |
|---|---|
| E0–E1 | Is the generator valid, and how does performance change with sparse or noisy demonstrations? |
| E2–E4 | Which retrieval, evidence, and reasoning choices add measurable signal? |
| E5–E7 | Do agent architecture, temporal provenance, and robustness components survive controlled tests? |
| E8–E9 | How do security challenges and human approval affect unsafe execution? |
| E10 | Do fixes generalize without regressing previously solved exception families? |
Every comparison holds ten controls constant unless one is the independent variable, including model version, token and retrieval budgets, corpus cutoff, action schema, sampling settings, and post-processing. Resampling occurs at the family or world level so near-neighbor cases do not masquerade as independent observations.
Requires Python 3.11+ and uv.
git clone https://github.com/rishixgamer/exceptiongraph.git
cd exceptiongraph
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ".[dev]"
make allGenerate the deterministic smoke corpus:
from exceptiongraph.experiments.toy import toy_corpus
episodes = toy_corpus()
worlds = {episode.world_id for episode in episodes}
print(f"{len(episodes)} episodes across {len(worlds)} worlds")12 episodes across 4 worlds
Validate an experiment contract without running a model:
python -m exceptiongraph.experiments.e0_generator_validity --dry-runThe default suite is offline. Tests marked neo4j, network, or slow are excluded
unless explicitly selected.
E0 is a deterministic generator-validity check: an evidence oracle should solve the toy corpus, while a no-evidence floor should not. A current run produces:
evidence_oracle: 12 episodes, 0 missing
no_evidence_floor: 12 episodes, 0 missing
decision_accuracy: +0.5000 [+0.2500, +0.7500] (95% paired CI)
paired_permutation: p=0.0330
generator validity: PASS — oracle 1.000 vs floor 0.500
This is evidence that the deterministic pipeline and comparison machinery behave as designed. It is not a live-model result or a benchmark-scale finding.
The checked-in E0 deterministic demonstration reproduces this toy-corpus result from the shipped implementation. See the reproduction instructions for the exact command and the artifact's limitations; it demonstrates byte-stable pipeline validation only, not model or benchmark performance.
| Path | Purpose |
|---|---|
src/exceptiongraph/generator/ |
Canonical worlds, policy histories, gold derivation, rendering, noise, and adversarial cases |
src/exceptiongraph/retrieval/ |
Retrieval interfaces and deterministic sparse, dense, temporal, graph, and hybrid adapters |
src/exceptiongraph/scoring/ |
Overall and mandatory-slice score reports with missing-response handling |
src/exceptiongraph/metrics/ |
Decision, evidence, temporal, provenance, calibration, abstention, and security metrics |
src/exceptiongraph/experiments/ |
Controlled E0–E10 runners, budgets, manifests, and paired comparisons |
examples/ |
Checked-in E0 demonstration artifact and deterministic reproduction instructions |
RESEARCH_BUILD_SPEC.md |
Eight machine-readable, pre-specified falsification gates |
EVALUATION.md |
Controls, metrics, slices, statistical plan, and experiment sequence |
EXCEPTIONBENCH.md |
Episode contract, generator rules, splits, and release targets |
docs/DEVELOPMENT_PROCESS.md |
Human and AI development responsibilities, review, verification, and reproducibility boundaries |
docs/THREAT_MODEL.md |
Trust boundary and security attack taxonomy |
- The 12-episode, 4-world toy corpus verifies plumbing; it is too small for research claims.
- Benchmark-scale generation and sealed evaluation have not yet been completed.
- The protocol pre-specifies falsification gates and provides a preregistration template; no sealed evaluation preregistration has been created yet.
- The default dense adapter uses a deterministic hash projection, not a production embedding model; E2 fixes the decision to gold so it can isolate retrieval recall.
- Optional Neo4j and network integrations are not part of the default offline CI gate.
- No results here establish live-model performance, trading alpha, financial return, or suitability for production decision automation.
Benchmark changes must preserve determinism, prevent family leakage, report mandatory
slices, and pass the reviewer checklist in CONTRIBUTING.md.
Architecture decisions are recorded under docs/adr/.
Code is licensed under Apache-2.0. Generated benchmark data is intended for release under CC-BY-4.0.