Experiment: aggregate OGC pages as raw features before shaping

[thodson-usgs]

Problem Statement

A query to a Water Data or NGWMN collection can span several pages within each chunk. The OGC machinery currently converts every page into a pandas DataFrame or GeoDataFrame and then concatenates those page frames. That makes a request-level output decision—whether the completed chunk is spatial, geometry-free, or nonspatial—at every page boundary. Empty pages and pages whose features omit geometry therefore need special frame construction solely to remain safe inputs to pandas concatenation.

PR #373 fixes the resulting all-empty frame-family bug by choosing the frame shape once per query and carrying that decision through pagination and finalization. The fix is narrow and validated, but it leaves open an architectural question: can the OGC path become simpler and safer by retaining GeoJSON feature dictionaries while walking a chunk's pages and converting the combined features into one frame only after the page walk completes?

This work is an experiment, not a presumed replacement for PR #373. It must independently reproduce #373's behavior, preserve every public getter contract, remain compatible with shared pagination and fan-out, and demonstrate that moving frame construction out of the page loop reduces total conceptual complexity without unacceptable time or memory costs.

Solution

Keep raw GeoJSON feature dictionaries as the private page payload for OGC pagination. Parse each response body once, structurally validate and extract its features, follow its next-page cursor, and combine feature lists in stable page order. Apply the row cap to the combined features, then perform one DataFrame or GeoDataFrame conversion for the completed chunk before handing that chunk to fan-out.

Generalize the internal service-neutral paginator with one adapter-supplied page-combination strategy. Existing DataFrame-based adapters continue to use DataFrame concatenation by default. The OGC adapter supplies feature-list combination. Transport continues to own cursor walking, repeated-cursor protection, progress, liveness, response aggregation, and row counting; it does not import or understand OGC.

Apply the experiment by response pipeline, not by output class. NLDI already fetches one complete FeatureCollection and converts it once, so it has no page-frame concatenation seam to remove and needs no change. The separate Water Data Statistics API is paginated, but one nested feature expands into multiple output rows; raw feature counts would no longer equal its established progress row counts and supporting that difference would require a broader page-weight abstraction. Statistics therefore keeps the default DataFrame page strategy. Deprecated NWIS adds geometry only after its tabular response has been assembled and likewise remains unchanged.

From a user's perspective, nothing changes: Water Data and NGWMN getters still return (DataFrame or GeoDataFrame, metadata), spatial and nonspatial collections retain their established frame families, empty results retain useful schema columns, chunked and unchunked queries produce equivalent data, and interruptions remain resumable through completed chunk frames.

Deliver the implementation as an independent draft PR based on upstream/main, explicitly marked as an experimental alternative to PR #373. Leave PR #373 unchanged. The narrower #373 implementation wins unless this experiment passes all correctness, live-service, and performance gates and is clearly simpler after counting the shared paginator change.

User Stories

1. As a Water Data user, I want OGC-backed getters to keep returning DataFrames or GeoDataFrames, so that this internal experiment does not break my analysis code.
2. As an NGWMN user, I want its getters to retain the same return contracts as before, so that shared OGC machinery remains invisible to me.
3. As a user requesting a spatial collection with geopandas installed, I want an empty result to remain a GeoDataFrame with an active WGS84 geometry column, so that empty and non-empty executions have the same frame family.
4. As a user requesting skip_geometry, I want both empty and non-empty results to be plain DataFrames without geometry, so that the request option determines the result shape consistently.
5. As a user requesting a nonspatial collection, I want a plain DataFrame even when geopandas is installed, so that installation details do not change the collection's contract.
6. As a user retrieving a spatial collection whose returned features omit or contain null geometry, I want the result to remain geospatial when the request contract is spatial, so that page contents do not change the frame family.
7. As a user without geopandas installed, I want spatial feature properties and available coordinate lists in a plain DataFrame, so that OGC retrieval continues to degrade gracefully.
8. As a user receiving a completely empty result, I want useful collection columns populated from my requested properties or the collection schema, so that downstream code can inspect and select expected columns safely.
9. As a user selecting explicit properties, I want their order and service-facing names preserved, so that raw feature accumulation does not alter projection semantics.
10. As a user retrieving several pages, I want rows to retain stable page and feature order before established final sorting, so that the result remains reproducible.
11. As a user setting max_rows, I want pagination to stop when enough features have arrived and return exactly the requested maximum, so that preview queries remain bounded.
12. As a user following OGC next links, I want repeated or empty cursors to terminate safely, so that malformed pagination metadata cannot cause an infinite walk.
13. As a user running a query that requires chunking, I want each chunk to walk all of its pages before it is marked complete, so that no partial chunk is presented as complete data.
14. As a user opting into parallel_chunks, I want chunked and serial executions to return equivalent rows, columns, frame type, geometry, and CRS, so that concurrency changes speed rather than meaning.
15. As a user whose chunks overlap, I want duplicate non-null feature IDs removed across completed chunks, so that chunking does not duplicate records.
16. As a user whose records have null IDs, I want those records preserved rather than collapsed as duplicates, so that missing identity does not cause data loss.
17. As a user interrupted by a transient failure during fan-out, I want completed chunk frames preserved and resumable, so that raw page payloads never leak into interruption state.
18. As a user resuming an interrupted query, I want only unfinished chunks re-issued and the final result shaped normally, so that the experiment preserves quota-saving resume behavior.
19. As a user watching progress, I want page and row counts to remain accurate when pages contain feature lists rather than frames, so that progress output remains trustworthy.
20. As a user inspecting metadata, I want the canonical query URL, cumulative elapsed time, and relevant response headers preserved, so that retrieval metadata still describes the full query.
21. As a Water Data user issuing CQL2 POST queries, I want body replay and cursor following to remain correct, so that the raw-feature path supports both GET and POST pagination.
22. As a Water Data user retrieving reference data, I want nonspatial reference tables to remain plain DataFrames, so that the experiment covers more than spatial feature collections.
23. As an NGWMN user retrieving providers or observation collections, I want their geometry-free contracts preserved, so that spatial sites do not force every NGWMN collection into geopandas.
24. As a Statistics user, I want its nested DataFrame page parser to behave unchanged, so that generalizing the shared paginator does not alter an unrelated adapter.
25. As a Ratings user, I want STAC search pagination and rating retrieval to behave unchanged, so that the paginator remains service-neutral.
26. As a Water Use user, I want NWDC pagination and location fan-out to behave unchanged, so that an OGC experiment does not regress non-OGC services.
27. As an adapter author, I want the paginator to accept an explicit page-combination strategy, so that an adapter can aggregate its natural page payload without pretending it is a DataFrame.
28. As an adapter author, I want DataFrame concatenation to remain the default strategy, so that existing adapters need no gratuitous custom code.
29. As a transport maintainer, I want transport to count pages through their length without knowing feature schemas or protocols, so that the service-neutral dependency boundary remains intact.
30. As an OGC maintainer, I want frame construction to happen once per completed chunk, so that geometry mode and empty-frame policy have one shaping seam.
31. As a maintainer diagnosing malformed upstream data, I want invalid features container shapes rejected at the page where they arrive, so that errors retain useful page context.
32. As a maintainer tolerating normal upstream variation, I want missing IDs, properties, and geometry accepted, so that structural validation does not become unnecessary schema rigidity.
33. As a contributor, I want offline behavior tests to mock HTTP at public getter boundaries, so that the default suite remains deterministic and tests observable contracts.
34. As a contributor changing the shared paginator, I want a focused component test for its page-combination contract, so that the new generic seam is pinned once rather than re-tested through private OGC helpers.
35. As a release maintainer, I want opt-in live tests for every adapter using shared pagination, so that mocked success is corroborated against current service responses.
36. As a release maintainer, I want live tests excluded from normal push CI and available through the established live workflow, so that service outages do not block unrelated changes.
37. As a reviewer, I want forced chunking examples to prove that more than one chunk was executed and that serial and chunked results agree, so that a nominal chunking test cannot pass through a one-chunk path.
38. As a reviewer, I want reproducible wall-time and peak-memory comparisons against PR fix(ogc): keep the frame type when a result is empty #373, so that the experiment's trade-offs are evidence-based.
39. As a reviewer, I want the experiment's OGC simplifications assessed together with its transport changes, so that complexity is not merely moved across a module boundary.
40. As a maintainer, I want PR fix(ogc): keep the frame type when a result is empty #373 left unchanged while the alternative is evaluated, so that the validated narrow fix remains available if the experiment fails.
41. As an agent implementing the experiment, I want explicit acceptance gates and an independent branch, so that I can complete the work without inventing product decisions.
42. As a project maintainer, I want a tie to favor the narrower fix, so that broader infrastructure changes must earn their additional blast radius.
43. As an NLDI user, I want its established single-response GeoDataFrame and as_json behavior left unchanged, so that an unrelated pagination experiment does not alter a service that already converts once.
44. As a Statistics user watching progress, I want row counts to continue describing flattened statistical rows rather than outer GeoJSON features, so that applying a generic raw-feature count does not make progress misleading.
45. As a deprecated NWIS user, I want its post-assembly geometry formatting left untouched, so that the experiment does not modernize or destabilize legacy code without a matching pagination problem.
46. As a maintainer, I want raw accumulation adopted only where several flat feature pages are otherwise shaped and concatenated, so that use of GeoDataFrame alone does not become an architectural abstraction criterion.

Implementation Decisions

• Implement this as an independent experimental branch from upstream/main and open a draft PR against main. Do not cherry-pick, amend, supersede, or otherwise modify PR fix(ogc): keep the frame type when a result is empty #373. Cross-link the two PRs and state that the experiment is an alternative implementation.
• Preserve public getter contracts. Water Data and NGWMN getters continue to return a frame and metadata. Do not add a raw-output option and do not return JSON or feature lists from public getters.
• Preserve the adapter-facing OGC facade as a deep module. Its orchestrated getter continues to return a completed frame and metadata; raw features are a private intermediate representation hidden inside its page walk.
• Scope the raw-feature strategy to the shared OGC API Features engine, where a chunk contains several flat feature pages that are currently shaped and concatenated. Producing a GeoDataFrame is not by itself an inclusion criterion.
• Leave NLDI unchanged. It has no cursor page walk or chunk-frame concatenation: it already retains one FeatureCollection until a single final GeoDataFrame conversion, and its public as_json option is an established NLDI-specific contract rather than a model for OGC getters. Treat this as confirming prior art for the conversion boundary, not a code-sharing opportunity.
• Leave the separate Water Data Statistics adapter on the default DataFrame page strategy. Its nested feature-to-row expansion means len(features) is not the number of output rows used for progress, and adding page-weight/size callbacks solely to include it would make the transport seam broader than the experiment justifies. Continue to live-smoke Statistics because it uses the generalized paginator.
• Leave deprecated NWIS geometry formatting unchanged. It converts a completed tabular response to a GeoDataFrame after assembly and has no raw GeoJSON page aggregation seam.
• Represent each OGC page payload as a list of GeoJSON feature mappings. Do not aggregate complete FeatureCollection envelopes because links and counts are page metadata, not records. Do not normalize features into row dictionaries during pagination because that would reintroduce shaping policy at the page seam.
• Parse each OGC response body once. Extract the next cursor and feature list from that parsed body.
• Structurally validate page payloads before accumulation. Missing, null, or empty features denotes an empty page. A non-list features value or a non-mapping feature fails deterministically with page-parse context. Missing or null id, properties, and geometry remain supported.
• Generalize the internal paginator with an adapter-supplied page-combination operation that receives collected pages and the optional row cap. The paginator may require page payloads to support len for progress and cap accounting.
• Keep DataFrame concatenation as the default page-combination operation so Statistics, Ratings, Water Use, and other current users preserve their behavior without custom strategies.
• Have OGC supply feature-list combination. Combine pages in arrival order and slice the combined features to the row cap before frame construction.
• Continue to stop requesting additional pages as soon as the accumulated feature count satisfies the row cap. The change in page representation must not weaken the early-download bound.
• Convert the combined features into one DataFrame or GeoDataFrame exactly once per successfully completed chunk. A multi-chunk query therefore performs one conversion per chunk, not necessarily one conversion for the entire query.
• Select geometry mode once from collection semantics, skip_geometry, and geopandas availability. Spatial collections with geopandas installed produce a GeoDataFrame even when every feature has null or missing geometry. skip_geometry and nonspatial collections produce plain DataFrames. Without geopandas, available coordinates remain represented through the established plain-DataFrame fallback.
• Keep WGS84 as the active CRS for geospatial OGC results.
• Keep one final empty-result schema-completion branch. If the caller supplied properties, use them; otherwise query the collection schema after determining that the complete result is empty. Construct the empty frame according to the same request-level geometry mode as a non-empty result.
• Do not hardcode collection schemas or fetch schemas for every query.
• Keep fan-out frame-based. A chunk enters the completion map only after raw features have been combined and converted successfully. Partial frames, interruption snapshots, resume, chunk ordering, response combination, and cross-chunk deduplication remain unchanged.
• Preserve stable page order, stable chunk order, non-null ID deduplication, and preservation of rows with null IDs.
• Preserve metadata semantics: canonical query identity, cumulative page elapsed time, response headers used for quota state, and conservative response aggregation across chunks.
• Preserve retry and failure taxonomy. Structural parse or conversion failures are deterministic; transient HTTP and transport failures retain bounded retry and resumable interruption behavior. A chunk that fails during its page walk is incomplete and is re-walked from its first page on resume.
• Keep transport service-neutral. The new paginator strategy must not import OGC, service adapters, frame-shaping policy, or service schemas. OGC continues to depend inward on transport, never the reverse.
• Preserve adapter facades and collection-family boundaries described by the accepted architecture decisions. Do not move collection logic into compatibility facades.
• Avoid a general accumulator framework, bespoke page-wrapper hierarchy, or object-column DataFrame used only to smuggle dictionaries through a frame-oriented interface. The intended extension is one narrow page-combination strategy.
• Add no new runtime dependency.
• Mark the PR title and description as experimental. Include a comparison table covering frame-conversion count, OGC-specific branches/helpers, shared transport changes, offline tests, live tests, wall time, and peak memory. Include the warning: “Experimental alternative to fix(ogc): keep the frame type when a result is empty #373; do not merge until comparison is complete.”
• The experiment succeeds only if it reproduces fix(ogc): keep the frame type when a result is empty #373's behavior, passes every shared-paginator consumer, meets the live and benchmark gates, and is clearly simpler after the paginator change is counted. If evidence is tied, prefer PR fix(ogc): keep the frame type when a result is empty #373.

Testing Decisions

• Prefer the highest existing seam: exercise observable OGC behavior through public Water Data and NGWMN getters with mocked HTTP. These tests should assert returned frame family, columns, geometry/CRS, rows, metadata, and requests—not private helper calls or raw accumulator representation.
• Reproduce the behavioral regressions covered by PR fix(ogc): keep the frame type when a result is empty #373 on the independent branch: spatial all-empty results are active WGS84 GeoDataFrames; skip_geometry all-empty results are plain DataFrames without geometry; nonspatial NGWMN providers and observation collections remain plain DataFrames; spatial pages with missing geometry remain GeoDataFrames; and page/concatenation order is stable.
• Add multi-page behavior tests with empty pages in different positions and with geometry absent from selected pages. Assert that page order cannot change the final frame family or schema.
• Test explicit properties and lazy schema completion through getter behavior. Verify that schema HTTP is performed only for a completely empty result lacking explicit properties.
• Test row caps at the getter boundary: truncation within the first page, stopping across pages, exact feature count, and no unnecessary next-page request after the cap is met.
• Test GET and CQL2 POST pagination through existing Water Data getter/CQL seams, including replay of the original method, headers, and body for next pages.
• Test malformed page structures at the OGC adapter boundary: non-list features and non-mapping feature entries fail with deterministic page context, while missing/null features and missing optional feature members remain tolerated.
• Add one focused component seam for the generic paginator because its page-combination strategy is a new transport contract. Verify default DataFrame aggregation, custom list aggregation, progress row counts from len(page), cap forwarding/application, repeated-cursor termination, response aggregation, and failure wrapping. Do not duplicate this contract in every adapter test.
• Run all existing Statistics, Ratings, Water Use, progress, transport, chunking, interruption/resume, architecture, and public-contract suites. Their external behavior must remain unchanged under the default DataFrame combiner.
• Run the existing NLDI and deprecated NWIS suites as part of the full offline validation, but add no raw-feature pagination behavior tests for them because neither has the page-frame aggregation seam under experiment. Do not add NLDI to the live release gate when no NLDI production code changes.
• Retain existing fan-out tests as the authority for completion tracking, resume, interruption snapshots, stable chunk combination, non-null ID deduplication, and null-ID preservation. Do not expose raw features to those tests because raw features must not cross the chunk-completion seam.
• Run architecture fitness functions and Import Linter. Confirm that transport has no OGC or adapter dependency, collection-family modules remain independent, facades remain logic-free, and the runtime graph remains acyclic.
• Add opt-in tests marked with the repository's existing live marker and excluded from default test runs. Use the established scheduled/manual Live API workflow rather than adding service-dependent checks to push CI.
• Live-test Water Data OGC behavior with representative low-volume queries covering a spatial collection, skip_geometry, a nonspatial reference table, an all-empty result requiring schema, forced pagination, max_rows, a CQL2 POST, and parallel_chunks(2).
• Live-test NGWMN with representative low-volume queries covering spatial monitoring locations, nonspatial providers, a geometry-free observation collection, an all-empty result, forced pagination, and parallel_chunks(2).
• Because the shared paginator changes, live-smoke Water Data Statistics through one POR or date-range query, Ratings through one STAC search and small rating retrieval, and NWDC Water Use through one paginated retrieval and one list-shaped fan-out.
• For chunking/fan-out live examples, verify from request counts that multiple chunks actually executed. Compare serial and chunked outputs after stable sorting, including concrete frame type, columns, feature IDs, geometry, CRS, and row equality. Use narrow dates and fixed identifiers where reliable; assert invariants rather than brittle global record counts.
• Do not print or persist API keys in live-test output. Live tests should run without a key where service limits permit and naturally use the existing environment-based credential policy when a key is configured.
• Add a reproducible benchmark comparing this experiment with PR fix(ogc): keep the frame type when a result is empty #373 on approximately 10,000 spatial and 10,000 nonspatial features over multiple pages. Report frame-construction count, total wall time, and peak memory.
• Treat a wall-time regression greater than approximately 10% or a peak-memory regression greater than approximately 25% as a failed gate unless the PR documents a compelling reason and receives explicit maintainer approval. Do not encode timing thresholds as flaky CI assertions.
• Validate with the full offline test suite, formatting, lint, strict type checking, import/architecture checks, and whitespace checks before live calls. Record exact commands and results in the draft PR.
• A good test for this work observes a getter, paginator contract, or service response boundary. Tests should not assert local variable names, helper count, a particular loop implementation, or the internal type alias used for feature mappings.

Out of Scope

• Changing, closing, rebasing, or force-updating PR fix(ogc): keep the frame type when a result is empty #373.
• Returning raw JSON, FeatureCollections, or feature lists from public Water Data or NGWMN getters.
• Adding a public raw=True mode or a second public result model.
• Breaking the adapter-facing OGC getter into separate public fetch and shape calls.
• Carrying raw features through fan-out, interruption objects, partial results, or resume state.
• Removing schema-complete empty results, hardcoding collection schemas, or fetching schema for non-empty queries.
• Changing established DataFrame/GeoDataFrame contracts, column naming, dtype coercion, final sorting, CRS, metadata, retry, deduplication, or error taxonomy.
• Rewriting Statistics, Ratings, or Water Use response parsers to use raw payload accumulation. They retain the default DataFrame page strategy. Statistics' nested one-feature-to-many-rows shape is explicitly not generalized in this experiment.
• Changing NLDI's single-response FeatureCollection conversion or its public as_json option. NLDI already converts once and does not use cursor pagination or fan-out.
• Changing deprecated NWIS geometry formatting or using this experiment to modernize legacy NWIS retrieval.
• Applying raw accumulation to modules merely because they return GeoDataFrames; inclusion requires the same paginated flat-feature aggregation problem.
• Reworking chunk planning, byte-budget rules, CQL2 splitting, concurrency policy, or the public parallel_chunks control.
• Introducing a broad generic accumulator framework beyond the single page-combination strategy justified by current callers.
• Exhaustively live-testing every collection and every getter. The required live matrix covers every service adapter affected by shared pagination and every distinct OGC shaping mode with representative low-volume calls.
• Moving live service tests into ordinary push CI.
• Claiming a performance improvement without benchmark evidence.
• Modifying the R project, unrelated notebooks, or unrelated local/untracked files.

Further Notes

• Git history shows that OGC has converted each page into a frame since its first implementation in commit 1295e91d on 2025-08-08. Raw FeatureCollection aggregation was not the behavior until a recent refactor.
• The plain-DataFrame all-empty finalizer also originated in 1295e91d. The observable frame-family mismatch began when geometry-bearing non-empty results became GeoDataFrames in commit a33d201b on 2025-09-25 while all-empty finalization remained a plain DataFrame. Recent refactors preserved rather than introduced that behavior.
• PR fix(ogc): keep the frame type when a result is empty #373 currently provides the narrow request-level frame-shape fix and has already passed the full offline suite, formatting/lint, strict type checking, pre-commit, independent review, and representative live smoke checks. This experiment has a deliberately higher burden of proof.
• Domain vocabulary matters: a query may have several chunks, and each chunk may have several pages. Raw features are combined across a chunk's pages; completed chunk frames are then combined by fan-out for the query.
• Accepted architecture decisions require transport to remain service-neutral, dependency direction to point from adapters toward transport, fan-out execution to remain distinct from protocol-specific chunk planning, and service contracts to remain behind stable facades. The implementation must preserve those decisions.
• A production-code audit found three GeoDataFrame-producing paths: the shared OGC engine, NLDI, and deprecated NWIS, plus the separate Statistics adapter that reuses OGC shaping helpers. Only the shared OGC engine both pages flat GeoJSON features and converts each page before concatenation. NLDI already follows the proposed single-conversion shape; NWIS adds geometry after tabular assembly; Statistics pages nested one-to-many data and therefore remains on the DataFrame strategy.
• The existing scheduled/manual Live API workflow is the precedent for durable service conformance tests. Default CI remains offline and deterministic.
• The OGC facade stabilization itself has not appeared in a tagged release, but the approved design does not need to break that facade. The experiment changes a private page representation and internal paginator contract only.