Eea bliki

.github/workflows/release.yml

@@ -113,11 +113,18 @@ jobs:
             fi
           done
 
+      # target_commitish is required here: without it, the GitHub API creates
+      # a brand-new tag from the *repository's default branch* tip, not from
+      # github.sha (the commit that triggered this run) and not from the
+      # version-bump/README commits the earlier steps just pushed to this
+      # branch. Pointing it at github.ref_name makes the release tag land on
+      # this branch's actual current tip.
       - name: Create GitHub release
         uses: softprops/action-gh-release@v3
         with:
           tag_name: v${{ steps.version.outputs.version }}${{ github.ref_name == 'staging' && '-staging' || '' }}
           name: v${{ steps.version.outputs.version }} (${{ github.ref_name }})
+          target_commitish: ${{ github.ref_name }}
           files: dist/*
           generate_release_notes: true
           prerelease: ${{ github.ref_name == 'staging' }}

README.md

@@ -36,7 +36,7 @@ pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"
 ```
 
 # staging's latest release (early access) — pin to the tag the "staging" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.7-staging"
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.10-staging"
 ```
 
 ## Usage
@@ -238,10 +238,10 @@ To pin to one specific release instead, use the exact tag the live badges under
 [Releasing a new version](#releasing-a-new-version)):
 
 ```python
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.6"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.10"
 
 # staging's latest release (early access)
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.7-staging"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.10-staging"
 ```
 
 Use the `%pip` magic rather than `!pip` — it installs into the kernel the

docs/read-only-ingest-client-plan.md

@@ -0,0 +1,113 @@
+# Read-only ingest — the client half (DI-11.9) — **DONE**
+
+> Landed: `sub_path` (DI-11.12), the retry guard and `placement` (DI-11.7),
+> `CommitResult.storage_path` and the docs below (DI-11.9). mypy clean, 52 tests
+> pass. What is still open is recorded at the end of this file.
+
+Companion note for **this** repository. The full design lives with the server
+work it depends on:
+
+```
+EEALakeHouse/Components/DremioDocumentService/devplans/2026-08-21/
+  Development Plan- Read-Only Ingest to Permanent S3 (DI-11).md
+```
+
+Read that first — this file only records what is built **here**, and when.
+
+## The short version
+
+`FolderIngest` already has the attribute: `intent="read_only" | "editable"`
+(`src/eea_datalakehouse/dds_ingestion/folder.py:103`, `models.py:13`), sent to
+`POST /ingest/begin` (`client.py:121`). On EEA production's managed catalog the
+server currently ignores it for placement — every ingest stages to a temp prefix,
+is copied into the catalog's Iceberg storage, and the upload is deleted.
+
+DI-11 makes `read_only` mean *store the files permanently* under
+`local_s3/dh-prod-data/read/<catalog mirror>/<table>/` and register them.
+**Where the bytes land is chosen by the server**, in `begin`'s presigned targets,
+so this package needs no new parameter and no new call.
+
+## What changed here
+
+0. ✅ **`sub_path` — the one new parameter** (DI-11.12). `FolderIngest(...,
+   sub_path="2026")`, passed straight through to `begin`; the server validates,
+   normalises and scopes on it. It is what lets a read-only table accumulate:
+
+   ```python
+   FolderIngest(folder="./bw_2026", target_catalog_path="…/bwd/reference",
+                data_format="parquet", intent="read_only",
+                table_name="water_temperature", sub_path="2026").run()
+   # → read/…/water_temperature/2026/*.parquet, alongside 2024/ and 2025/
+   ```
+
+   A custodian whose local folder *already* has the structure needs nothing —
+   `scan_folder` (`folder.py:80-92`) preserves sub-folders and the server keeps
+   the original `rel_path`. `sub_path` is for filing a **flat** folder under a
+   name they choose. Worth a worked example in `README.md`, and note that
+   `replace` with a `sub_path` clears only that year.
+1. ✅ **Docs and docstrings** — `folder.py` class docstring + the `intent`
+   parameter, `dds_ingestion/README.md:38`, `__init__.py:16`: say what each
+   intent now does (permanent raw folder + catalog view vs. Iceberg copy).
+2. ✅ **`CommitResult.storage_path`** (`models.py`) — optional, additive; the
+   physical Dremio path the server reports for a permanent ingest, so a notebook
+   can print where the files actually are. Parse permissively as everywhere else
+   (absent ⇒ `None`), so it works against an older server.
+3. ✅ **Resume/retry note** (`folder.py:305-322`, `retry()`): `_already_done`'s
+   docstring says a re-run is "an S3 overwrite of the identical key, which is
+   harmless". Against a permanent folder in `append` mode that is only true if
+   the server reuses the keys instead of numbering them. Align the wording with
+   whatever DI-11.7 settles, and steer users to `attach()` + `retry()` rather
+   than re-running `run()`.
+4. ✅ **Tests** — `tests/dds_ingestion/`: `storage_path` parsing, and a
+   respx-mocked `begin` whose `key_prefix` is a permanent `read/...` prefix,
+   asserting the client uploads to the issued targets unchanged.
+
+## What does not change
+
+- No new constructor argument, no new method, no new endpoint.
+- `editable` behaviour, the upload/parallelism/progress machinery, credentials.
+- Nothing has to change for the client to keep working while the server runs
+  with `DDS_INGEST_READ_MODE=staged` (its default).
+
+## When
+
+DI-11 **step 4** ("Prove") in the rollout table — after the server feature exists
+behind its flag, and it is the first client pointed at a `permanent` server. The
+JupyterLab extension is deliberately left alone until step 5.
+
+## Noted for later: deleting a read-only table
+
+Because read-only files are permanent, dropping the view or dataset from the
+catalog leaves the objects in S3. The agreed home for that cleanup is a
+**delete-table operation in this package** — `eea_datalakehouse.catalog` has
+`deleteview` and `deletefolder` today, but nothing that removes a table together
+with its backing data. **Not implemented as part of DI-11**, recorded here (and
+in §8.5 of the DI-11 plan) so it is a known gap rather than a surprise.
+
+When it is picked up: this package holds no S3 credentials by design, so the
+delete needs a DDS-side endpoint for the read folder to call — not boto3 in a
+notebook.
+
+## Related: `common/catalog.py` moves server-side
+
+`dds_ingestion/common/catalog.py` (on `development`) walks a catalog path and
+creates the missing dataflow levels, refusing to invent a domain or subdomain.
+That walk is exactly what a read-only ingest needs before its view can be
+created, so **DI-11.2 ports it into DDS**, where the extension and any REST
+client get it too and the taxonomy rule is enforced in one place. Once that
+lands, the client-side copy is either a thin convenience wrapper over
+`POST /api/v1/catalog/{path}/folder` or redundant — decide when DI-11.2 is done,
+not before.
+
+This is a *catalog* concern only: the `read/` prefixes in S3 need no creating at
+all — see §3.6 of the DI-11 plan.
+
+## One thing to settle first
+
+There are two copies of this client: this repo
+(`src/eea_datalakehouse/dds_ingestion/`) and a snapshot in the DDS repo
+(`clients/dds_ingest/dds_ingest/`). They have already drifted — `folder.py`,
+`client.py` and `models.py` differ; `credentials.py` and `progress.py` are
+identical. **This repo is where the client is developed.** Whether the DDS-side
+copy is re-synced, thinned to a test fixture, or dropped is a separate decision,
+not part of DI-11 — but decide it before editing both by hand again.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "EEADataLakehouse"
-version = "0.1.7"
+version = "0.1.10"
 description = "Data preparation and validation utilities for the EEA data lakehouse pipeline."
 readme = "README.md"
 requires-python = ">=3.11"

src/eea_datalakehouse/dds_ingestion/README.md

@@ -0,0 +1,116 @@
+# eea_datalakehouse.dds_ingestion
+
+Notebook-side client for the **Dremio Document Service (DDS) Ingest API** (DI-8.4 / DI-8.5).
+
+A generated Jupyter notebook imports `FolderIngest` to transfer a folder of data
+files to S3 and register it as a Dremio table — coordinated entirely through the
+DDS REST API. The **only** object-storage access is the presigned URLs issued by
+DDS; this package never uses an S3 SDK or S3 credentials.
+
+## Install
+
+This is a subpackage of the **EEADataLakehouse** library, not a distribution of
+its own — install the library from the repository root:
+
+```bash
+pip install -e ".[dev]"   # dev extra adds ruff / mypy / pytest / respx
+```
+
+In a notebook kernel, pinned to a ref that carries this subpackage:
+
+```python
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@development"
+```
+
+## Usage (inside a notebook kernel)
+
+Dremio credentials are read from the injected kernel env vars `_DREMIO_USER` /
+`_DREMIO_PWD`, and the service URL from `DDS_BASE_URL`. None of these are ever
+logged or printed.
+
+```python
+from eea_datalakehouse.dds_ingestion import FolderIngest
+
+outcome = FolderIngest(
+    folder="./my_data",
+    target_catalog_path="biodiversity.uploads",
+    data_format="parquet",      # one of parquet | csv | json
+    intent="read_only",         # or "editable" — see below
+    conflict_mode="fail",
+    parallelism=4,              # concurrent uploads (default 4)
+).run()
+
+print(outcome.commit.table_path, outcome.commit.record_count)
+print(outcome.commit.storage_path)   # where the files are, when they are kept
+```
+
+`intent` decides what the transfer leaves behind:
+
+| | `read_only` | `editable` |
+|---|---|---|
+| your uploaded files | kept — they **are** the table | copied in, then deleted |
+| the table | the folder, registered, with a view at the catalog path | an Iceberg table |
+| good for | published data, data that accumulates | a table you will write to |
+
+(Storing read-only files permanently is a server setting. Where it is not
+enabled, both intents stage and load as before and only the table's shape
+differs — `storage_path` is then `None`.)
+
+`run()` performs `begin → upload(all files) → commit`, and leaves the session
+handle available afterwards.
+
+## Read-only data that grows
+
+`intent="read_only"` on a server configured for permanent storage keeps your
+files as uploaded and registers them as the table — nothing is copied into the
+catalog. `sub_path` files each upload under a named sub-folder, so one table can
+accumulate:
+
+```python
+FolderIngest(
+    folder="./bw_2026",                       # a flat folder of parquet files
+    target_catalog_path="water_management_resources/bathing_water/bwd/reference",
+    data_format="parquet",
+    intent="read_only",
+    table_name="water_temperature",
+    sub_path="2026",                          # → .../water_temperature/2026/
+    conflict_mode="fail",                     # refuses if 2026 is already there
+).run()
+```
+
+Next year, the same call with `sub_path="2027"` adds to the same table;
+`conflict_mode="replace"` with a `sub_path` re-does **that year only** and leaves
+the others standing. If your local folder already has the structure
+(`2026/*.parquet`), you do not need `sub_path` — the layout is preserved as-is.
+The name is normalised server-side to the EEA convention (lowercase,
+underscores), so `"Q1 2026"` becomes `q1_2026`.
+
+## Managing the transfer
+
+A transfer is a server-side session, so it can be inspected and resumed:
+
+```python
+with FolderIngest(folder="./my_data", target_catalog_path="catalog/theme/sub",
+                  data_format="parquet") as job:
+    try:
+        job.run()
+    except Exception:
+        if job.status().is_resumable:
+            job.retry()      # re-runs only the step that failed — no re-upload
+```
+
+`job.session_id` · `status()` · `estimate()` · `retry()` · `cancel()` ·
+`close()` · `FolderIngest.attach(session_id, ...)` to pick a transfer up in a
+later kernel.
+
+**Full class reference:** [`docs/python-client-guide.md`](../../docs/python-client-guide.md).
+
+## Layout
+
+| Path | Purpose |
+|---|---|
+| `dds_ingestion/credentials.py` | env-var creds + redacted `DremioCreds` |
+| `dds_ingestion/models.py` | typed request/response models for the contract |
+| `dds_ingestion/client.py` | thin, unit-testable HTTP client (`IngestClient`) |
+| `dds_ingestion/progress.py` | tqdm progress bar with graceful fallback |
+| `dds_ingestion/folder.py` | `FolderIngest` orchestration (scan/parallel/session) |

src/eea_datalakehouse/dds_ingestion/__init__.py

@@ -1,4 +1,5 @@
-"""Ingestion into DDS (Dremio Document Service), notebook-side (DI-8.4/8.5).
+"""eea_datalakehouse.dds_ingestion — notebook-side client for the Dremio
+Document Service Ingest API.
 
 A generated Jupyter notebook imports :class:`FolderIngest` to transfer a folder
 of data files to S3 (via DDS-issued presigned URLs only) and register it as a
@@ -12,15 +13,21 @@
         folder="./my_data",
         target_catalog_path="biodiversity.uploads",
         data_format="parquet",
-        intent="read_only",
+        intent="read_only",   # keep the files as the table | "editable" → Iceberg
         parallelism=4,
     ).run()
     print(outcome.commit.table_path, outcome.commit.record_count)
+    print(outcome.commit.storage_path)   # where the files are, if kept
+
+``intent`` decides what the transfer leaves behind. On a server configured for
+permanent read-only storage, ``"read_only"`` keeps the uploaded files where the
+table lives and registers them; ``"editable"`` loads them into an Iceberg table
+and deletes the upload. See :class:`FolderIngest` for the full rule.
 """
 
 from __future__ import annotations
 
-from .client import IngestApiError, IngestClient
+from .client import IngestApiError, IngestClient, S3UploadError
 from .credentials import (
     DremioCreds,
     MissingCredentialsError,
@@ -31,17 +38,20 @@
     DEFAULT_PARALLELISM,
     FolderIngest,
     IngestOutcome,
+    IngestStateError,
     ingest_folder,
     scan_folder,
 )
 from .models import (
     BeginResult,
     CommitResult,
     DataFormat,
+    EstimateResult,
     FileSpec,
     Intent,
     Progress,
     S3Plan,
+    StageResult,
     StatusResult,
     UploadPart,
     UploadTarget,
@@ -53,15 +63,19 @@
     "CommitResult",
     "DataFormat",
     "DremioCreds",
+    "EstimateResult",
     "FileSpec",
     "FolderIngest",
     "IngestApiError",
     "IngestClient",
     "IngestOutcome",
+    "IngestStateError",
     "Intent",
     "MissingCredentialsError",
     "Progress",
     "S3Plan",
+    "S3UploadError",
+    "StageResult",
     "StatusResult",
     "UploadPart",
     "UploadTarget",