Skip to content

refactor(ratings): drive both stages through the shared executor - #369

Merged
thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:refactor/ratings-shared-executor
Aug 12, 2026
Merged

refactor(ratings): drive both stages through the shared executor#369
thodson-usgs merged 3 commits into
DOI-USGS:mainfrom
thodson-usgs:refactor/ratings-shared-executor

Conversation

@thodson-usgs

@thodson-usgs thodson-usgs commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

What

Moves get_ratings onto the shared transport machinery: the STAC search's page walk now drives transport.pagination.paginate as a one-item FanOut, and the per-feature RDB downloads fan out over the feature list itself (a plain list satisfies FanOutPlan — ADR 0008 — no adapter class needed). stats.py's one-item fan-out was the template.

Why

Four page walks existed in the package; three went through paginate. The fourth was ratings' hand-rolled synchronous loop, and its per-feature downloads were a serial for-loop of N sequential GETs — so ratings alone had no per-attempt retry, no stall budget, no progress line, no resumable interruption, and no bounded concurrency. Deleting the bespoke loop concentrates pagination in the one home transport already owns.

Behavior changes (deliberate — flagging for review)

  1. The failure policy for per-feature downloads is now split by whether retrying could help. Previously every per-feature failure — including a 429 rate limit — was logged at warning level (invisible unless logging was configured) and the rating silently dropped from the result dict. Now:
    • A transient failure (429 / 5xx / timeout / connection drop) is retried per API_USGS_RETRIES and then raised as a resumable interruption. Rate limiting is systematic — once it starts, every remaining download fails — so silently skipping was the old behavior's worst case.
    • A deterministic per-feature failure (a stale catalog entry 404ing, a feature with no data asset, a malformed RDB) is skipped under a new SkippedRatingWarning naming the feature; the rest of the batch is returned. One bad catalog entry shouldn't cost every other site's rating. The warning is visible by default and warnings.filterwarnings("error", category=SkippedRatingWarning) restores strict all-or-nothing.
    • An OSError writing file_path propagates — a local disk problem is not a per-feature condition (the old skip was overbroad here).
    • A site with no published rating (or a nonexistent site) is unchanged: it matches no feature in the search, so it is simply absent from the dict — no warning, no error.
  2. A mid-search 429 surfaces as a resumable QuotaExhausted (parity with every other getter) instead of a raw RateLimited. exc.call.resume() finishes the interrupted stage; since get_ratings returns a dict rather than (df, md), the assembled per-feature dict comes from a fresh call — documented in the docstring.
  3. Downloads now run concurrently, bounded by API_USGS_CONCURRENT, instead of serially.

Unchanged: the next-link cross-host/credential policy still goes through resolve_next_url (both existing safety tests pass untouched), headers are still evaluated per asset href (test updated to the new seam), and the wire behavior of search parameters is identical.

Testing

  • Full offline suite: 786 passed — includes the resumable-search test plus four new failure-policy tests (deterministic 404 warns and skips, keeping the rest of the batch; missing data asset warns and skips; the warning escalates to an error under filterwarnings("error"); a download 429 raises resumable and provably never skips), and a fitness function pinning that ratings can no longer import the executing sync transport helpers (alongside the existing wateruse rule).
  • mypy --strict, ruff, xenon, complexipy, lint-imports all pass.

Independent of #367/#368 (branches off main).

🤖 Generated with Claude Code

Four page walks existed in the package; three went through
transport.pagination.paginate. The fourth was ratings' hand-rolled
synchronous STAC loop, and its per-feature RDB downloads were a serial
for-loop of N sequential GETs -- so ratings alone had no per-attempt
retry, no stall budget, no progress line, no resumable interruption,
and no bounded concurrency.

Route both stages through the executor every other getter uses
(stats.py's one-item fan-out was the template; ADR 0008's plan shape --
a plain list -- covers the downloads with no adapter class):

- _search drives paginate as a one-item FanOut with STAC strategies:
  pages wrap raw feature dicts in a single ``feature`` column, the
  ``next`` href still passes through resolve_next_url's shared
  cross-host/credential policy, and a mid-search 429 now surfaces as a
  resumable QuotaExhausted instead of a raw RateLimited.
- The downloads fan out over the feature list itself, bounded by
  API_USGS_CONCURRENT, with headers still evaluated per asset href.
- The log-and-skip swallow is gone -- deliberately. A deterministic
  per-feature failure (missing asset, malformed RDB, 404) now surfaces
  typed instead of silently dropping that rating from the result dict,
  and a transient one is retried then raised resumable. Silent data
  loss under rate limiting was the old behavior's worst case.

A new fitness function pins that ratings can no longer import the
executing sync transport helpers, alongside the existing wateruse rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@thodson-usgs

Copy link
Copy Markdown
Collaborator Author

Health metrics vs main

Same method as on #367/#368 (pyscn 1.29.0 / radon / xenon / complexipy / import-linter, isolated worktrees, ProjectRoot verified).

pyscn composite: 82/100 (B) — unchanged; all seven sub-scores identical to main. But this is the one of the three branches that moves the complexity averages in the right direction:

main this PR
Average cyclomatic complexity 2.377 2.359
Average cognitive complexity 5.893 5.823
Average nesting depth 1.055 1.035
High-risk entries 19 19
Modules 57 57
Resolved dependency edges 140 142
Functions 363 368
Package LOC (non-blank) 13,090 13,140

Where it comes from

get_ratings was the most complex function in the file and the only medium-risk entry among everything these three PRs touch:

get_ratings   cyclomatic 11 (medium) → 9 (low)     cognitive 14 → 11
_search       cyclomatic  5 (low)    → 4 (low)     cognitive  9 →  9

The retry/paging/failure-handling that inflated it now lives behind paginate/FanOut instead of in the adapter. The five new entries (_fetch_rating = 4, _download_all = 2, and three closures at 1) are all low-risk, which is why the file grew ~50 lines while the package's averages fell.

Also up: architecture compliance 87.1% → 87.3% (banded score stays 87), and the +2 dependency edges are transport.pagination / transport.fanout / transport.retry replacing the direct transport.http sync calls — the intended direction under ADR 0006.

Duplication is unchanged at 8.83% (clone groups 5, pairs 50, total clones 25 — identical), so the new fan-out code introduced no clone of the existing wateruse/stats fan-out shapes.

One regression worth naming: waterdata/ratings.py maintainability index 63.3 → 61.3 (still rank A). That is the file growing; the per-function complexity above is the better read of whether it got harder to work on.

Merge gates (xenon, complexipy, lint-imports, mypy --strict): pass. 783 tests pass offline.

Reminder that the behavior changes in the PR description — removed log-and-skip swallow, search 429 becoming a resumable QuotaExhausted, concurrent downloads — are the substantive review surface here; the metrics are the easy part.

thodson-usgs and others added 2 commits August 12, 2026 15:31
…warning

Split the download stage's failure policy by whether retrying could help,
instead of treating every failure as fatal to the batch:

- A transient failure (429 / 5xx / timeout / connection drop) keeps the
  executor semantics: retried per API_USGS_RETRIES, then raised as a
  resumable interruption. Skipping these would silently drop every
  remaining feature under rate limiting -- the old serial loop's worst
  case, and the reason this branch retired log-and-skip.
- A deterministic per-feature failure (stale catalog entry 404ing, a
  feature with no data asset, a malformed RDB) now skips that feature
  under a new SkippedRatingWarning naming it, and returns the rest of
  the batch. Aborting would have discarded every other site's rating
  over one bad catalog entry; retrying would reproduce the failure.
- OSError writing file_path propagates: a local disk problem is not a
  per-feature condition (the old loop's skip was overbroad here).

SkippedRatingWarning is a UserWarning (visible by default, unlike the
old logger.warning, which was silent unless logging was configured) in
dataretrieval.exceptions, re-exported at the package root. Strict
all-or-nothing behavior is one filter away:
warnings.filterwarnings("error", category=SkippedRatingWarning).

A skipped feature hands the executor an inert 204 placeholder pair so
the item is marked complete and a later resume() continues past it.

A site with no published rating still warns nothing: it matches no
feature in the search, so there is nothing to skip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
Four parallel review agents (reuse, simplification, efficiency, altitude)
over the branch diff; applied the converging findings:

- Generalize the warning taxonomy: SkippedItemWarning carries the
  batch-skip policy once; SkippedRatingWarning subclasses it, so users
  can escalate every future skip with one filter while still targeting
  ratings alone. Both exported at the package root. The get_ratings and
  _download_all docstrings now cross-reference the policy instead of
  restating it in full (three copies -> one).
- Delete the dead borrow-or-open client branch in _fetch_rating: FanOut
  publishes the shared client before any fetch runs, so active_client()
  is never None inside a drive; the fallback executed nowhere. The
  open_async_client import goes with it, and the architecture test's
  allowlist tightens to default_headers alone.
- Drop the redundant placeholder.elapsed = timedelta(0): the aggregate
  path reads elapsed only through combining._safe_elapsed, which already
  zero-fills hand-built responses. The timedelta import goes with it.
- Return body-less responses to the executor (_inert_response) for
  successes as well as skips: the executor keeps every completed item's
  response until the drive ends but aggregates only status, headers,
  and URL, so retaining full RDB bodies pinned ~N x tens-of-KB for the
  whole retry-prone drive. Real status and quota headers are kept.
- Deduplicate the thrice-spelled assets.data.href traversal into
  _asset_href.
- Drop the derivable empty-frame guard in _search: every page frame is
  built with a feature column and the combine helpers preserve it.
- Tests: hoist the QuotaExhausted import and the _GOOD_ASSET/_BAD_ASSET
  constants (the 429-resume test silently depended on two spellings of
  the same URL agreeing); extract _imports_from in architecture_test so
  the three import-surface rules share one mechanism, standardized on
  ast.walk (function-local imports can no longer slip past the two
  older rules).

Skipped as out of scope, for a follow-up issue: a shared one-request
paginate-through-FanOut driver (second copy of the stats.py scaffold),
and a FanOut per-item result mode (native skip outcome; resume()
returning the public dict shape instead of requiring a fresh call,
which currently re-downloads the whole batch after an interruption).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
@thodson-usgs
thodson-usgs marked this pull request as ready for review August 12, 2026 21:16
@thodson-usgs
thodson-usgs merged commit e9026ab into DOI-USGS:main Aug 12, 2026
11 checks passed
thodson-usgs added a commit to thodson-usgs/dataretrieval-python that referenced this pull request Aug 12, 2026
Absorbs DOI-USGS#368, DOI-USGS#369, and DOI-USGS#371. Eight files conflicted; three of the
resolutions were more than textual:

- ``wateruse.py``: main edited the implementation this branch had
  already renamed to ``nwdc.py``, so git conflicted the shim against
  it. Kept the shim and ported DOI-USGS#371's ``run_paginated`` migration into
  ``nwdc.py``, preserving ``adapter="nwdc"``.
- ``cql.py``: took main's version wholesale. This branch's two changes
  there (``redirected(OGC_API_URL)`` and ``adapter="waterdata"``) are
  subsumed by DOI-USGS#368 routing ``get_cql`` through
  ``waterdata.utils.get_ogc_data``, which already applies both.
- ``ratings.py``: took main's rewritten implementation and re-applied
  this branch's only contribution to it (``redirected(STAC_URL)``),
  plus adapter scoping on both drives.

Three breaks were semantic, not textual -- git merged them cleanly and
they would have failed at import or call time, because main added
callers of names this branch renamed or deleted:

- ``transport/pagination.run_paginated`` imported ``_CONCURRENCY_DEFAULT``
  (deleted here in favour of ``configuration.DEFAULT_CONCURRENCY``) and
  called ``RetryPolicy.from_env`` (renamed to ``from_configuration``).
  It now takes an ``adapter`` argument and threads it to both the retry
  policy and the executor, which is what makes per-adapter ``retries``
  and ``concurrency`` tables reach the three getters that use it.
- ``ogc/engine``'s ``cql_body`` branch (new in DOI-USGS#368) called
  ``from_env`` and dropped the adapter; both fixed.
- ``get_ogc_data`` gained ``cql_body`` from main and ``adapter`` here;
  both parameters kept.

969 passed, mypy --strict clean, all hooks including import-linter pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BTaSm7HmVb94RSJiKW4WAS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant