explore(ogc): aggregate raw feature pages before shaping

[thodson-usgs]

Status

Experimental alternative to #373; do not merge until comparison is complete.

Explores #376. This leaves #373 and its fix/empty-result-frame-type branch unchanged. The final diff is an alternative implementation of the same empty-frame contracts; this branch carries the narrow fix commits as setup and then moves OGC page aggregation to raw features.

What changes

• Parse each flat OGC API Features response once into a validated private list of feature dictionaries plus its next cursor.
• Aggregate feature pages in stable order, apply max_rows to raw features, and convert once per completed chunk.
• Keep fan-out, interruption snapshots, and resume state frame-based; raw features never cross the chunk-completion seam.
• Generalize the service-neutral paginator with one optional adapter-supplied page combiner. Existing Statistics, Ratings, and Water Use callers retain DataFrame concatenation by default.
• Preserve Water Data and NGWMN public (DataFrame or GeoDataFrame, metadata) contracts, WGS84 geometry, empty schemas, CQL2 POST replay, metadata, progress, ordering, and deduplication.
• Leave NLDI, deprecated NWIS geometry formatting, and Statistics' nested response shaping unchanged.

Comparison

Evidence	Page-frame path / #373	Raw-feature exploration
Frame conversions for 10 pages	10	1 per completed chunk
OGC shaping seam	Shape every page, then concatenate	Validate/flatten pages, shape once
OGC helper change	_get_resp_data(response, ...)	_feature_frame(features, ...) plus _combine_feature_pages
Shared transport change	None	Optional combine_pages; DataFrame default retained
Empty spatial result	Active EPSG:4326 GeoDataFrame	Same
Offline suite	#373 previously validated	800 passed, 26 live deselected
Live services	Representative smoke checks	14 passed across Water Data, NGWMN, Statistics, Ratings, and Water Use
10k spatial wall time	1.00x	0.84x
10k spatial peak RSS	1.00x	1.01x
10k nonspatial wall time	1.00x	0.92x
10k nonspatial peak RSS	1.00x	1.02x

The benchmark is reproducible with:

python benchmarks/ogc_raw_feature_pagination.py

Timing thresholds are informational, not CI assertions.

Structural health

Compared with upstream/main (61428b5e) using pyscn 1.29.0 and wily 1.25.0:

• pyscn health remains 82/B.
• No change in high-risk functions, dead code, CBO/LCOM, dependency cycles/depth, architecture compliance, or clone-group count.
• Cloned-fragment share improves from 8.8339% to 8.7413%.
• Package Maintainability Index improves from 48.3473 to 49.2859 (+1.9%).
• ogc/shaping.py loses 33 production lines and improves MI 44.84 → 54.47; validation/interface complexity moves into ogc.engine and transport.pagination.

This is a modest net maintainability gain rather than an unequivocal simplification, so the behavioral and performance evidence should drive the decision.

Validation

pytest -q tests/                         800 passed, 26 deselected
pytest -q tests/ogc_live_test.py -m live 14 passed
ruff format --check .                    passed
ruff check .                             passed
mypy --strict dataretrieval              passed (57 source files)
pytest -q tests/architecture_test.py     17 passed
lint-imports                             7 contracts kept
pre-commit run --all-files               all hooks passed

The pre-commit run includes Xenon, complexipy, Import Linter, mypy, Ruff, and nbstripout.

Review notes

A two-axis review against #376 found no Standards violations. Spec findings around progress counts, terminal empty pages, serial-versus-parallel equality/request counts, live rating retrieval, Water Use fan-out, and benchmark clarity were addressed before this branch was published.

[thodson-usgs]

Follow-up simplification is now in 896e3770.

LOC / complexity result

Compared with pinned upstream/main (61428b5e):

Measure	Original exploration (b2a94ff5)	Current (896e3770)
dataretrieval/ textual diff	+75 net lines	-17 net lines
pyscn raw SLOC	+86	+4 (10,795 vs 10,791)
pyscn health	82/B	82/B
high-risk functions / dead code	20 / 0	20 / 0
cycles / dependency depth	0 / 8	0 / 8
architecture compliance	86.60%	86.60%

The refactor removes the dead OGC compatibility paginator, collapses two overloads plus an Any implementation into one Sized page-payload generic, avoids copying validated feature lists, and stops threading concrete geopandas state through production fetch layers. Public/frame/fan-out behavior is unchanged.

Wily's bottom-15 package MI remains improved versus main (48.3473 -> 49.2859). Within the moved complexity, ogc/engine.py improved from the original exploration's 61.55 to 62.66, and transport/pagination.py from 59.37 to 61.06.

Revalidation

• Offline: 800 passed, 26 deselected
• Live shared-paginator/OGC matrix: 14 passed
• Focused refactor suite: 331 passed
• Strict mypy: 56 tracked production files clean
• Ruff check/format: passed repository-wide
• Architecture tests: 17 passed
• Import Linter: 7/7 contracts kept
• Pre-commit: all hooks passed (Ruff, mypy, Xenon, complexipy, Import Linter, nbstripout, whitespace)
• Two-axis review: no Spec findings; the one Standards note (intentional private imports in the white-box benchmark) is now explicitly documented

Benchmark still performs 10 -> 1 frame conversions per 10-page chunk:

Mode	wall raw/page-frame	peak RSS raw/page-frame
spatial	0.86x	1.02x
nonspatial	0.90x	1.02x

All informational guardrails pass.

The required warning still applies: Experimental alternative to #373; do not merge until comparison is complete.