de_sh: backfill 2023-2025; per-edition column names, German date, decimal-comma area - #223
Open
haithcoatj wants to merge 1 commit into
Open
de_sh: backfill 2023-2025; per-edition column names, German date, decimal-comma area#223haithcoatj wants to merge 1 commit into
haithcoatj wants to merge 1 commit into
Conversation
Backfilling 2023-2025 surfaced three separate per-edition problems. All four
editions now convert, validate and produce a STAC collection.
1. The archive member, not the archive. The 2023 GeoPackage was written with
user_version = 0, so GDAL's GPKG driver identifies it by the .gpkg extension
alone; handed "/vsizip/Feldbloecke_2023_GPKG.zip" it matched no driver at
all ("not recognized as being in a supported file format"). The member is
named differently every year (Feldbloecke_2023, FB_2024, FB_20250101,
FB_20260101), so glob it, as cz and be_vlg already do.
2. fachguelti is DD.MM.YYYY text in every edition, not ISO. Left as text it
reached the STAC step as "Invalid isoformat string: '01.01.2024'". Parsed
with an explicit format, as cz.py does for DATUM_REP; measured over all
rows of all four editions, every value is exactly DD.MM.YYYY (2025 has one
NULL, which becomes NaT).
3. Column spellings drift: 2023 has its own set (flident/flgesamt/hbn), 2024 is
mixed case, 2025 and 2026 are upper case. Only the 2024 spelling was mapped,
so 2025 and 2026 silently dropped determination:datetime and metrics:area
("Column 'fachguelti' not found in dataset, removing from schema") and still
validated, because both columns are optional. The published 2026 edition is
missing them for this reason. The area is also text with a decimal comma in
2023, 2025 and 2026, where post_migrate's ha-to-m2 astype(float) fails.
For 2023 flgesamt (gross) is used for metrics:area; it equals flnetto in all
198,614 rows. 2023 is also the only edition whose fachguelti actually varies
per feature (197,586 rows at 01.01.2023, 1,028 rows older), so there it stays a
column instead of collapsing into collection-level metadata.
Verified with `pytest tests/test_convert.py -k de_sh` and by running
`fiboa publish` for each of 2023-2026 over 100-feature extracts of the real
archives: all VALID, no dropped columns, metrics:area in m2, id == flik.
haithcoatj
force-pushed
the
de_sh-2023-2025-backfill
branch
from
September 1, 2026 17:24
13d4bac to
5991f1f
Compare
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The changes align with established converter patterns in this repo (variants member-globbing, migrate-based renames, and column_migrations) and are self-contained to the de_sh converter.
Pull request overview
This PR updates the de_sh dataset converter to reliably backfill and convert Schleswig-Holstein field-block editions from 2023–2026 by handling per-edition GeoPackage member names, column-name casing drift, German-formatted dates, and decimal-comma area values so that determination:datetime and metrics:area are consistently produced.
Changes:
- Switches
variantsto read the.gpkgmember(s) inside the yearly ZIP archives (rather than passing the ZIP path directly). - Adds per-edition column renames via a
migrate()override to normalize drifting source column names/casing. - Adds column migrations to parse
fachgueltias a date and parseFlaechevalues that use a decimal comma.
File summaries
| File | Description |
|---|---|
fiboa_cli/datasets/de_sh.py |
Makes the de_sh converter robust across 2023–2026 by targeting ZIP members, normalizing column names, and parsing date/area fields for consistent STAC/fiboa output. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
saidy-moregeo
left a comment
saidy-moregeo
left a comment
Collaborator
There was a problem hiding this comment.
Looks good to me and the bugs are real — I reproduced the date one on main
(determination:datetime lands in collection metadata as the string 01.01.2024;
with your patch, 2024-01-01T00:00:00Z). Few points though:
- The PR is against
publish-portolanrather thanmain— not sure if that's by
choice, but the workflows only trigger on PRs tomain, so nothing has run: no
tests, no lint, no matrix. - No CHANGELOG entry, and the diff is well over changelog.yml's 20-line threshold.
- Yes to the fixture. I mutation-tested the three fixes: breaking the date format
fails the suite, but sabotaging all fiveCOLUMN_RENAMEStargets and making
the decimal-comma branch raise still passes — the test picks2026(first key)
and lands on the 2024-shaped fixture, so neither path runs. A genuine
2026-shaped extract replacingFeldbloecke_2026_GPKG.zipwould fix the
mislabeling and cover both at once, no harness changes.
m-mohr
requested changes
Sep 4, 2026
m-mohr
left a comment
m-mohr
left a comment
Contributor
There was a problem hiding this comment.
I'm following @saidy-moregeo. @haithcoatj Could you fix these issues before we merge, please?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backfills de_sh 2023-2025. All four editions now convert, validate and produce a
STAC collection; see the commit message for the three per-edition fixes.
Worth knowing beyond de_sh: the source changed column casing in 2025
(
fachguelti/Flaeche→FACHGUELTI/FLAECHE). vecorel-cli logsColumn 'Flaeche' not found in dataset, removing from schemaand continues, andfiboa validatestill returns VALID becausemetrics:areais optional — so thepublished 2026 edition has been missing its area column with nothing flagging it.
The test fixture didn't catch it either:
Feldbloecke_2026_GPKG.zipis actually a2024-shaped extract, so pytest only ever exercised the old spelling.
Verified (built all four editions,
pytest -k de_shpasses, ruff clean):rows 198,614 / 197,673 / 195,747 / 194,503 with 2026 matching the published
edition exactly; zero dropped-column warnings;
metrics:areain m² with mediandeclared/polygon 0.999984; geometries 0 invalid, CRS EPSG:4647 throughout. The
rebuilt 2026 is byte-identical to the published one across all 194,503 geometries
and every existing column — the only difference is the added area.
Two things for a maintainer to decide:
file here. Happy to add real 2023- and 2025-shaped extracts if wanted.
metrics:areais also declared-but-absent in publishedes_cbandes_md(both
DN_SURFACE). It can't be a collapsed constant, and es_cb was freshlybackfilled on 2026-08-31 and still lacks it, so it looks like the same bug.
Upstream data note, converted faithfully and not corrected: 18% of 2026 rows
declare
FLAECHElarger than their own polygon (median +2.0%, +0.65% on thetotal); zero such rows in 2023-2025. Raw string format is identical between
editions and the ratios are a continuum, so it's upstream rather than parsing.
Unrelated: the data survey lists DE-SH as EPSG:25832; it's EPSG:4647. Happy to PR
that separately.