Eea bliki - #39
Merged
Merged
Conversation
Merge development: catalog set*/deletewiki rename, table/view tag gating, main/staging alias tag
Merge staging: catalog set*/deletewiki rename, table/view tag gating, main/staging alias tag
Fix release tags pointing at the wrong commit
Fix release tags pointing at the wrong commit
The working tree held a verbatim copy of the old standalone `dds_ingest` package laid over the restructured one. That copy is the richer code — it carries the whole session-management surface (attach/status/estimate/retry/ cancel, context-manager support, IngestStateError, EstimateResult/StageResult, is_resumable/is_terminal) that the committed version lacks — so it is kept. What it dropped or reverted is fixed here instead. - Restore IngestClient.__repr__, which withholds the Bearer PAT. Without it a traceback, log handler or debugger renders the token. - Point the module references back at eea_datalakehouse.dds_ingestion, in the docstrings and in the logger name, which had gone back to "dds_ingest" and so was no longer filterable under the package it lives in. - Delete the copy-over debris: two pyproject.toml files inside the package declaring the old `dds-ingest` distribution on the hatchling backend, a duplicate README one level too high, and two identical in-package test trees that still imported `dds_ingest` and could not load. - Port the one thing those trees had that the tracked suite did not: test_session.py, 13 tests covering the session surface above. The suite goes 31 -> 46, all passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G9tqi2Kd1f8gynmSYP4j6e
Observations and ideas collected about eea_datalakehouse while the development team is away. Nothing here has been applied to the code — every entry is a proposal to discuss when the team is back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9VStQfVW6N2nH3Mn6cSid
Three proposals, all still open for discussion: - half the public surface has no docstring (35% on this branch, 49% with catalog/ on development), which is exactly what Shift+Tab and `obj?` render in a notebook; - Catalog's methods are thin delegates that drop the docstrings operations.* already has, so none of that text reaches the object a notebook holds; - nothing in the repo pins the JupyterLab 4 environment the library is written for, and py.typed covers only dds_ingestion/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K9VStQfVW6N2nH3Mn6cSid
The design lives with the server work it depends on, in the DDS repo's devplans; this records what is built HERE and when — the sub_path parameter, storage_path, the resume wording that a permanent folder changes, and the two things noted but deliberately not built: deleting a read-only table with its backing data, and whether the client-side ensure_catalog_path is now redundant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VV1gKHekMUx69LJKSjP8Q3
… data Two client-side halves of DI-11, where the server now stores a read-only ingest's files permanently instead of staging and deleting them. `FolderIngest(..., sub_path="2026")` files an upload under a named sub-folder of the table, so a read-only dataset can accumulate a year at a time. It is passed straight through to begin: validating, normalising and scoping it are the server's job, or the client becomes a second opinion about where data lives. A folder that already has the structure locally needs nothing — scan_folder keeps sub-folders and the server preserves them. `retry()` no longer falls back to "open a fresh session and upload again" for a transfer whose files are stored permanently. That fallback is right when the staged copy was discarded and actively harmful here: the server numbers an incoming name that already exists, precisely so an append cannot overwrite live data, which turns a silent re-run into duplicated rows. It now raises and says what to do instead — conflict_mode="replace" to redo it, or a sub_path for data that belongs beside what is there. StatusResult gains `placement` / `stores_permanently` to tell the two apart. A server without DI-11 omits the field and reads as staged, so nothing changes against one. Verified: mypy clean, 51 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VV1gKHekMUx69LJKSjP8Q3
… (DI-11.9) `CommitResult` parses `storage_path` — where a permanently-stored read-only table's files physically are, against `table_path`, which stays where the table is queried. Absent parses as None, so an older server is unaffected. The docs described `intent` as a choice of table shape, which is what it meant before the server could keep the upload. The class docstring, the README and the package docstring now make the same distinction in the space each has: read-only keeps your files and registers them; editable copies them in and deletes the upload; and permanent storage is a server setting, so where it is off both intents stage as before. The resume note is the one that was actively misleading. "An S3 overwrite of the identical key, which is harmless" is true of re-running the SAME session and false of starting a new one against permanent storage — the server numbers a colliding name to protect live data, so the re-run leaves a second copy instead of overwriting. It now says so and points at attach() + retry(). Verified: mypy clean, 52 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VV1gKHekMUx69LJKSjP8Q3
Ported from the DDS repo (EEALakeHouse@904c1a4), where a folder ingest failed with "DDS ingest API error 500: Internal Server Error" and nothing said more. A transfer makes several calls to two different systems, so a status code with no call name does not identify the broken one. * IngestApiError carries ``where`` — the method and path of the call that failed — and puts it in the message; status_code/message stay available. * S3UploadError (a subclass, so existing handling still catches it) is raised when a pre-signed upload fails: those bytes go straight to object storage, and the message names the file and the URL that actually answered rather than blaming the Document Service. * Error bodies are capped at 500 chars: a proxy's HTML error page ran to kilobytes and buried the status line it came with. An empty body now reads as "(empty 500 response body)" instead of nothing at all. Ported, not copied: this package keeps its own divergences (the PAT-redacting __repr__ and begin(sub_path=...)), which the DDS copy does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qh3Bnb8mdNjtFbmZxfmGEZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.