refactor(ratings): drive both stages through the shared executor - #369
Conversation
Four page walks existed in the package; three went through transport.pagination.paginate. The fourth was ratings' hand-rolled synchronous STAC loop, and its per-feature RDB downloads were a serial for-loop of N sequential GETs -- so ratings alone had no per-attempt retry, no stall budget, no progress line, no resumable interruption, and no bounded concurrency. Route both stages through the executor every other getter uses (stats.py's one-item fan-out was the template; ADR 0008's plan shape -- a plain list -- covers the downloads with no adapter class): - _search drives paginate as a one-item FanOut with STAC strategies: pages wrap raw feature dicts in a single ``feature`` column, the ``next`` href still passes through resolve_next_url's shared cross-host/credential policy, and a mid-search 429 now surfaces as a resumable QuotaExhausted instead of a raw RateLimited. - The downloads fan out over the feature list itself, bounded by API_USGS_CONCURRENT, with headers still evaluated per asset href. - The log-and-skip swallow is gone -- deliberately. A deterministic per-feature failure (missing asset, malformed RDB, 404) now surfaces typed instead of silently dropping that rating from the result dict, and a transient one is retried then raised resumable. Silent data loss under rate limiting was the old behavior's worst case. A new fitness function pins that ratings can no longer import the executing sync transport helpers, alongside the existing wateruse rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Health metrics vs
|
| main | this PR | |
|---|---|---|
| Average cyclomatic complexity | 2.377 | 2.359 |
| Average cognitive complexity | 5.893 | 5.823 |
| Average nesting depth | 1.055 | 1.035 |
| High-risk entries | 19 | 19 |
| Modules | 57 | 57 |
| Resolved dependency edges | 140 | 142 |
| Functions | 363 | 368 |
| Package LOC (non-blank) | 13,090 | 13,140 |
Where it comes from
get_ratings was the most complex function in the file and the only medium-risk entry among everything these three PRs touch:
get_ratings cyclomatic 11 (medium) → 9 (low) cognitive 14 → 11
_search cyclomatic 5 (low) → 4 (low) cognitive 9 → 9
The retry/paging/failure-handling that inflated it now lives behind paginate/FanOut instead of in the adapter. The five new entries (_fetch_rating = 4, _download_all = 2, and three closures at 1) are all low-risk, which is why the file grew ~50 lines while the package's averages fell.
Also up: architecture compliance 87.1% → 87.3% (banded score stays 87), and the +2 dependency edges are transport.pagination / transport.fanout / transport.retry replacing the direct transport.http sync calls — the intended direction under ADR 0006.
Duplication is unchanged at 8.83% (clone groups 5, pairs 50, total clones 25 — identical), so the new fan-out code introduced no clone of the existing wateruse/stats fan-out shapes.
One regression worth naming: waterdata/ratings.py maintainability index 63.3 → 61.3 (still rank A). That is the file growing; the per-function complexity above is the better read of whether it got harder to work on.
Merge gates (xenon, complexipy, lint-imports, mypy --strict): pass. 783 tests pass offline.
Reminder that the behavior changes in the PR description — removed log-and-skip swallow, search 429 becoming a resumable QuotaExhausted, concurrent downloads — are the substantive review surface here; the metrics are the easy part.
…warning
Split the download stage's failure policy by whether retrying could help,
instead of treating every failure as fatal to the batch:
- A transient failure (429 / 5xx / timeout / connection drop) keeps the
executor semantics: retried per API_USGS_RETRIES, then raised as a
resumable interruption. Skipping these would silently drop every
remaining feature under rate limiting -- the old serial loop's worst
case, and the reason this branch retired log-and-skip.
- A deterministic per-feature failure (stale catalog entry 404ing, a
feature with no data asset, a malformed RDB) now skips that feature
under a new SkippedRatingWarning naming it, and returns the rest of
the batch. Aborting would have discarded every other site's rating
over one bad catalog entry; retrying would reproduce the failure.
- OSError writing file_path propagates: a local disk problem is not a
per-feature condition (the old loop's skip was overbroad here).
SkippedRatingWarning is a UserWarning (visible by default, unlike the
old logger.warning, which was silent unless logging was configured) in
dataretrieval.exceptions, re-exported at the package root. Strict
all-or-nothing behavior is one filter away:
warnings.filterwarnings("error", category=SkippedRatingWarning).
A skipped feature hands the executor an inert 204 placeholder pair so
the item is marked complete and a later resume() continues past it.
A site with no published rating still warns nothing: it matches no
feature in the search, so there is nothing to skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
Four parallel review agents (reuse, simplification, efficiency, altitude) over the branch diff; applied the converging findings: - Generalize the warning taxonomy: SkippedItemWarning carries the batch-skip policy once; SkippedRatingWarning subclasses it, so users can escalate every future skip with one filter while still targeting ratings alone. Both exported at the package root. The get_ratings and _download_all docstrings now cross-reference the policy instead of restating it in full (three copies -> one). - Delete the dead borrow-or-open client branch in _fetch_rating: FanOut publishes the shared client before any fetch runs, so active_client() is never None inside a drive; the fallback executed nowhere. The open_async_client import goes with it, and the architecture test's allowlist tightens to default_headers alone. - Drop the redundant placeholder.elapsed = timedelta(0): the aggregate path reads elapsed only through combining._safe_elapsed, which already zero-fills hand-built responses. The timedelta import goes with it. - Return body-less responses to the executor (_inert_response) for successes as well as skips: the executor keeps every completed item's response until the drive ends but aggregates only status, headers, and URL, so retaining full RDB bodies pinned ~N x tens-of-KB for the whole retry-prone drive. Real status and quota headers are kept. - Deduplicate the thrice-spelled assets.data.href traversal into _asset_href. - Drop the derivable empty-frame guard in _search: every page frame is built with a feature column and the combine helpers preserve it. - Tests: hoist the QuotaExhausted import and the _GOOD_ASSET/_BAD_ASSET constants (the 429-resume test silently depended on two spellings of the same URL agreeing); extract _imports_from in architecture_test so the three import-surface rules share one mechanism, standardized on ast.walk (function-local imports can no longer slip past the two older rules). Skipped as out of scope, for a follow-up issue: a shared one-request paginate-through-FanOut driver (second copy of the stats.py scaffold), and a FanOut per-item result mode (native skip outcome; resume() returning the public dict shape instead of requiring a fresh call, which currently re-downloads the whole batch after an interruption). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
Absorbs DOI-USGS#368, DOI-USGS#369, and DOI-USGS#371. Eight files conflicted; three of the resolutions were more than textual: - ``wateruse.py``: main edited the implementation this branch had already renamed to ``nwdc.py``, so git conflicted the shim against it. Kept the shim and ported DOI-USGS#371's ``run_paginated`` migration into ``nwdc.py``, preserving ``adapter="nwdc"``. - ``cql.py``: took main's version wholesale. This branch's two changes there (``redirected(OGC_API_URL)`` and ``adapter="waterdata"``) are subsumed by DOI-USGS#368 routing ``get_cql`` through ``waterdata.utils.get_ogc_data``, which already applies both. - ``ratings.py``: took main's rewritten implementation and re-applied this branch's only contribution to it (``redirected(STAC_URL)``), plus adapter scoping on both drives. Three breaks were semantic, not textual -- git merged them cleanly and they would have failed at import or call time, because main added callers of names this branch renamed or deleted: - ``transport/pagination.run_paginated`` imported ``_CONCURRENCY_DEFAULT`` (deleted here in favour of ``configuration.DEFAULT_CONCURRENCY``) and called ``RetryPolicy.from_env`` (renamed to ``from_configuration``). It now takes an ``adapter`` argument and threads it to both the retry policy and the executor, which is what makes per-adapter ``retries`` and ``concurrency`` tables reach the three getters that use it. - ``ogc/engine``'s ``cql_body`` branch (new in DOI-USGS#368) called ``from_env`` and dropped the adapter; both fixed. - ``get_ogc_data`` gained ``cql_body`` from main and ``adapter`` here; both parameters kept. 969 passed, mypy --strict clean, all hooks including import-linter pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
What
Moves
get_ratingsonto the shared transport machinery: the STAC search's page walk now drivestransport.pagination.paginateas a one-itemFanOut, and the per-feature RDB downloads fan out over the feature list itself (a plain list satisfiesFanOutPlan— ADR 0008 — no adapter class needed).stats.py's one-item fan-out was the template.Why
Four page walks existed in the package; three went through
paginate. The fourth was ratings' hand-rolled synchronous loop, and its per-feature downloads were a serial for-loop of N sequential GETs — so ratings alone had no per-attempt retry, no stall budget, no progress line, no resumable interruption, and no bounded concurrency. Deleting the bespoke loop concentrates pagination in the one home transport already owns.Behavior changes (deliberate — flagging for review)
API_USGS_RETRIESand then raised as a resumable interruption. Rate limiting is systematic — once it starts, every remaining download fails — so silently skipping was the old behavior's worst case.SkippedRatingWarningnaming the feature; the rest of the batch is returned. One bad catalog entry shouldn't cost every other site's rating. The warning is visible by default andwarnings.filterwarnings("error", category=SkippedRatingWarning)restores strict all-or-nothing.OSErrorwritingfile_pathpropagates — a local disk problem is not a per-feature condition (the old skip was overbroad here).QuotaExhausted(parity with every other getter) instead of a rawRateLimited.exc.call.resume()finishes the interrupted stage; sinceget_ratingsreturns a dict rather than(df, md), the assembled per-feature dict comes from a fresh call — documented in the docstring.API_USGS_CONCURRENT, instead of serially.Unchanged: the
next-link cross-host/credential policy still goes throughresolve_next_url(both existing safety tests pass untouched), headers are still evaluated per asset href (test updated to the new seam), and the wire behavior of search parameters is identical.Testing
filterwarnings("error"); a download 429 raises resumable and provably never skips), and a fitness function pinning that ratings can no longer import the executing sync transport helpers (alongside the existing wateruse rule).mypy --strict,ruff,xenon,complexipy,lint-importsall pass.Independent of #367/#368 (branches off main).
🤖 Generated with Claude Code