From 5178f4d35fc974d7a4ccac630473ab4c8dacf0b7 Mon Sep 17 00:00:00 2001 From: Jaak Laineste Date: Wed, 9 Sep 2026 10:41:09 +0000 Subject: [PATCH 1/2] fix(example): give the source cache an age, and a way to discard it `data/source/` is a cache that never expired. Reuse was keyed on the files being present and internally coherent -- counts agreeing with recorded metadata, ownership and active-status predicates holding -- and the log called that "completeness-verified" and "semantics-verified". Both are true statements about the bytes on disk and neither says the data is current. The two are indistinguishable in a log until someone regenerates the committed artifacts from a stale cache. The result is self-consistent, passes validate and verify, and cannot be reproduced by CI, which always starts cold and fetches what the services publish today. The failure then surfaces as the project.qgz currency check, a long way from its cause. That happened on 2026-09-09: a regeneration silently reused two-week-old sources and reported 518 candidates against the current 517. So: - `--refresh` discards the cache and re-fetches every service. Run it before regenerating the committed artifacts; - reuse now logs the cache's age, and reuse beyond CACHE_MAX_AGE_DAYS (7) warns that coherent is not the same as current; - the log wording no longer implies currency. The example is regenerated in the same commit, because the run record hashes pipeline.py and tests/test_verify.py holds it to that. The refreshed sources produce a byte-identical project.qgz, which is independent evidence that this environment matches CI's. Also corrects two manifest prose figures that contradicted the derived `row_count: 79082` beside them, and the same count in the README. Those were stale before this change; the refresh is what exposed them. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DqoLGyKY5opHGjNMSerpHg --- docs/maintainers/debugging.md | 9 +- examples/tartu-development/README.md | 19 ++- examples/tartu-development/dashboard.html | 2 +- examples/tartu-development/pipeline.py | 97 +++++++++++++-- examples/tartu-development/project.yaml | 26 ++-- .../runs/run-20260909-103753.json | 113 ++++++++++++++++++ .../validation/latest-report.json | 6 +- 7 files changed, 241 insertions(+), 31 deletions(-) create mode 100644 examples/tartu-development/runs/run-20260909-103753.json diff --git a/docs/maintainers/debugging.md b/docs/maintainers/debugging.md index b969c8e..6f923ba 100644 --- a/docs/maintainers/debugging.md +++ b/docs/maintainers/debugging.md @@ -136,8 +136,13 @@ identical in the checks list: opening or closing in Tartu rewrites the file. It is advisory on a pull request and a hard failure on the schedule for exactly this reason. -To tell them apart without guessing, read the job's uploaded `worked-example-run` -artifact rather than re-running anything: its run record inventories every +Regenerating the example is what clears the second kind, and it must be done +with `pipeline.py --refresh`: `data/source/` is a cache with no expiry, so a +plain re-run can reproduce the committed artifacts exactly from stale local +data and look like it worked. + +To tell the two kinds apart without guessing, read the job's uploaded +`worked-example-run` artifact rather than re-running anything: its run record inventories every output by SHA-256 and its validation report carries feature counts, both comparable with what the repository ships. diff --git a/examples/tartu-development/README.md b/examples/tartu-development/README.md index 27e4186..13383b3 100644 --- a/examples/tartu-development/README.md +++ b/examples/tartu-development/README.md @@ -136,9 +136,26 @@ The equivalent convenience command is: ../../.e2e-venv/bin/python run_e2e.py ``` +`data/source/` is a cache, and it never expires on its own. A re-run reuses +whatever is there if the files are internally coherent — counts agreeing with +their recorded metadata, ownership and active-status predicates holding — which +establishes that the bytes are sound, not that they still match what the +services publish. Reuse older than seven days logs a warning naming the age. + +**Before regenerating the committed artifacts, discard it:** + +```bash +../../.e2e-venv/bin/python pipeline.py --refresh +``` + +Regenerating from a stale cache produces a project that looks fine and that CI +cannot reproduce, because CI always starts cold and fetches the current data. +That mismatch surfaces as the `project.qgz` currency failure in +`.github/workflows/example.yml`, a long way from its cause. + Outputs include: -- `data/source/Tartu_maakond_KATASTER_GPKG.gpkg` — 79,056 cadastral parcels +- `data/source/Tartu_maakond_KATASTER_GPKG.gpkg` — 79,082 cadastral parcels - `data/source/etak_main_roads.geojson` — completeness-verified ETAK main roads - `data/source/tartu_municipal_education.geojson` — normalized official municipal facilities - `data/derived/final-candidates.gpkg`, `.parquet`, `.json` diff --git a/examples/tartu-development/dashboard.html b/examples/tartu-development/dashboard.html index f14ff19..3c82817 100644 --- a/examples/tartu-development/dashboard.html +++ b/examples/tartu-development/dashboard.html @@ -581,7 +581,7 @@