Simplify fiboa publish for catalog-driven publication - #214
Open
ivorbosloper wants to merge 88 commits into
Open
Simplify fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 88 commits into
fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 88 commits into
Conversation
Co-authored-by: Matthias Mohr <matthias@mohr.ws>
# Conflicts: # CHANGELOG.md # pixi.lock
…easing memory requirements for large data sets
`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ivorbosloper
changed the title
fiboa publish for catalog-driven publication
…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
The tests fail. That's not intentional, I guess? |
…dules Covers the new PerFileBaseConverter paths (multi-source merge, part cleanup, re-sort of non-Hilbert parts, error handling), the vendored hilbert module, and the one-line command modules — bringing coverage back over the 80% gate (72.2% -> 81.5%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ocal mapping fixtures - per_file.py and duckdb.py kept pyarrow readers open across os.remove / os.replace / in-place rewrites; harmless on POSIX, fails on Windows (part cleanup silently no-opped, duckdb 1.1 post-processing silently downgraded). All readers now close deterministically. - commit the 17 crop-mapping CSVs the converter tests previously fetched from the network at test time (a fiboa.org 503 failed the ch test on a Windows runner); tests are now hermetic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
I fixed it. Coverage was too low, and windows handles open files in a different way. Also fixed failed network issues (to fiboa.org) |
…anges Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Old dk editions carry a handful of rows with a null Marknr; the id column is non-nullable, so the write failed after a full conversion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…daries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gpio check flagged published files as spatially unordered. Measured from the published footers: the plain writer yields mediocre locality, and the DuckDB converter's 1-arg ST_Hilbert is meaningless without bounds (jp row groups span 38% of Japan). The per-file merge sort measures excellent (groups 0.1% of extent). One sort implementation now serves every path: _ensure_hilbert_sorted (cheap bbox-only sortedness probe, 50k-row groups) runs at the end of the DuckDB converter and in fiboa publish — re-running publish over an existing parquet doubles as the repair tool for published data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
take() concatenates chunks; int32 offsets overflow past 2 GB of WKB (us/jp editions). Parquet's BYTE_ARRAY is identical for large_binary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Engines only use the bbox column for pushdown when the geo metadata declares it; the post-processing appended the column without the covering entry (visible on the published jp editions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fallbacks vecorel/cli#23 is merged and released, so the converter output arrives pre-sorted; fiboa_cli/conversion/hilbert.py and the try/except fallback imports are gone. The in-place resort now writes without the embedded arrow schema: the large_binary widening used for >2GB takes must not leak into the file, or a rewritten part stops schema-matching untouched pre-sorted siblings during the streaming merge (masked before 0.2.16, when every part needed rewriting). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… write store_schema=False silently drops ALL key-value metadata (geo, collection), not just the embedded arrow schema — the first production resort produced a file geopandas could not read. Write through a ParquetWriter against the original narrow schema instead, casting each sub-2GB batch back from the widened types; regression test asserts the resorted file keeps its metadata and narrow schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rByFields Cantabria's 2010-2014 SIGPAC layers are server-side joins: every field is table-qualified, so where=OBJECTID>x returned 'Failed to execute query'. Probe the layer for the real key name and page on it. Also 'sortBy' is not an ArcGIS parameter (orderByFields is); it only worked on layers whose default order happens to be the key. es_cb strips the table prefixes after download, first occurrence wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vecorel.org/fiboa.org fail intermittently; a blip at write time killed conversions after 25-minute source downloads (four times this week). The needed URIs are known up front, so fetch them first — retrying with backoff, failing fast and cheap — and let load_file's per-process cache make the write itself network-free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
I'm not really involved in the publishing (yet?), so I can't really judge this. But this PR contains a lot of unrelated changes, so I suspect we need to merge something else first to get this easier to review. |
orderByFields costs ~100 s per request on joined layers (the server sorts the whole join every page), which made the Cantabria 2010-2014 layers take days. A range filter on the unique key answers in ~1 s and a window of page_size ids cannot overflow a page, so fetch min/max once (two sorted one-row queries) and page by half-open id windows, skipping empty ones. Pages cached by the old scheme are reused when their ids prove they cover a window exactly. A where filter in rest_params is now ANDed in instead of being clobbered by the paging clause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
…y variant ideg.xunta.gal serves SIXPAC_2014..SIXPAC_2026 (2010-2013 never existed, 2025/2026 were missing). The older campaigns differ: 2014 names the layer RECINTO with SUP_SIGPAC and no DN_OID, 2015 uses SUP_SIX/USO_SIX, 2020 lacks DN_OID but has IDGEOM. Rename per page in file_migration so the land-use filter and id checks see the canonical names; 2014 gets the SIGPAC recinto reference as id, 2020 the geometry id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
Every SIXPAC_<year> MapServer of es_ga has its Recintos layer at id 2, so pages of different years collided in the cache and later years silently re-read the first year's pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
~16k pages of 1000 features are concatenated per edition; the 20 unmapped attribute columns would otherwise stay in memory until the very end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fiboa publishnow only converts, validates, builds PMTiles and writes acollection.jsonwith relative links,file:size/file:checksum(sha256 multihash), a web-map-links v1.3.0pmtileslink withpmtiles:layers, and avisualasset. README/LICENSE generation, the data-survey lookup and theaws s3 syncupload are removed;--generate-meta,--data-url,--s3-upload-path,--yes,--editor,--converted-by,--data-survey-urlare gone,--no-pmtilesand--tippecanoe-optsare new. README and CHANGELOG updated.spdx-license-listis only used by tests and moves to the pixidevfeature.FiboaBaseConverter: thedetermination:datetimecolumn added byuse_variant_as_determinationwas dropped again by the base converter's "remove unlisted columns" step; it is now listed incolumnsautomatically (affected dk, hr).be_vlg: drop plots without a crop code (one plot in the 2023 edition failed validation), derive the determination date from the variant year instead of a constant, add the 2026 edition.Why
Publishing moves to the git-backed Portolan catalog at https://github.com/fieldsoftheworld/harmonized-field-data-catalog (source.coop/ftw/harmonized-field-data), which owns the catalog layer (styles, thumbnails, AGENTS.md/README, uploads). fiboa-cli stays the single home of conversion logic and produces the per-dataset building blocks.
Verification
The three pilot collections built with this branch pass
rashid check --livewith 0 errors against the published catalog.🤖 Generated with Claude Code