PerFileBaseConverter - #207
Conversation
…easing memory requirements for large data sets
|
@m-mohr There's a highly relevant part here that fixes the hilbert-sort . Would be great to see this on main before other people start publishing new data sets in the wrong sort-order... |
There was a problem hiding this comment.
Pull request overview
Adds per-file conversion and streaming GeoParquet merging to reduce peak memory usage for large datasets.
Changes:
- Introduces Hilbert-sorted, batched merging of converted parts.
- Migrates Spain’s converter to the new pipeline and HCAT/admin mixins.
- Prevents temporary part files from being uploaded.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
pyproject.toml |
Updates Vecorel dependency. |
fiboa_cli/publish.py |
Excludes part files from uploads. |
fiboa_cli/datasets/es.py |
Adopts per-file conversion and updated mappings. |
fiboa_cli/datasets/data-files/es_cl_prv.csv |
Completes Spanish province data. |
fiboa_cli/datasets/commons/hcat.py |
Tightens mapping validation. |
fiboa_cli/conversion/per_file.py |
Implements per-file conversion and merging. |
CHANGELOG.md |
Documents the feature and fixes formatting. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| part = os.path.join(dirname, f"{filename}_{index}_part{ext}") | ||
| part_files.append(part) | ||
| if os.path.exists(part): |
| write_kwargs = {"compression": compression} | ||
| if compression_level is not None: | ||
| write_kwargs["compression_level"] = compression_level | ||
| writer = pq.ParquetWriter(output_file, out_schema, **write_kwargs) |
| if col.get("bbox") is not None: | ||
| bboxes.append(col["bbox"]) |
| merged_bbox = None | ||
| if bboxes: | ||
| merged_bbox = ( | ||
| min(b[0] for b in bboxes), | ||
| min(b[1] for b in bboxes), | ||
| max(b[2] for b in bboxes), | ||
| max(b[3] for b in bboxes), | ||
| ) |
| **kwargs, | ||
| ) | ||
|
|
||
| # Multi-source: convert each URI to its own GeoParquet part, then merge. |
Isn't that primarily in vecorel CLI? I'll try to review and merge those first, then create a release... |
Per-file migration and merging the result, decreasing memory requirements for large data sets.
Depends on a new release of Vecorel