feat(config)!: resolve settings through a layered chain

[thodson-usgs]

Closes #352.

Adds dataretrieval.configuration: one ordered chain that resolves every setting, and a
configure() block that scopes settings to a single call, a single thread, or a single
service — without mutating process-global os.environ.

The shape

A configuration profile is a named set of settings for one adapter, written in
code or stored in the config file. configure() takes those objects positionally, at
most one per adapter, and nothing else:

import dataretrieval
from dataretrieval import ngwmn, waterdata, wqp
from dataretrieval.ngwmn import NgwmnConfiguration
from dataretrieval.waterdata import WaterdataConfiguration
from dataretrieval.wqp import WqpConfiguration

with dataretrieval.configure(
    WaterdataConfiguration.load("overnight"),  # from the file, by name
    NgwmnConfiguration.load("gentle"),         # from the file, by name
    WqpConfiguration(retries=2),               # built here
):
    flow, _ = waterdata.get_daily(monitoring_location_id=sites, time="P30D")
    levels, _ = ngwmn.get_water_level(monitoring_location_id=wells)
    samples, _ = wqp.get_results(siteid=sites)

That block is the case the feature exists for: several services, one configuration each,
some loaded from the file by name and some built in code. The adapter a configuration
targets is a property of its class, so the caller never restates it — which is what
keeps the adapter roster out of every call site. Configuration targets none of them,
which is what makes it package-wide. Two configurations for one adapter raise: they are
the one pairing with no defined order.

Because delivery is a ContextVar, a key set inside the block cannot leak across threads
or asyncio tasks — which is what makes it safe for a server or notebook handling several
users' credentials, the thing assigning to os.environ could never do.

The file

~/.dataretrieval/config.toml, or the path in DATARETRIEVAL_CONFIG. Top-level keys are
package-wide; [<adapter>] is that adapter's default profile, always in effect;
[<adapter>.<name>] is a named profile, inert until a caller selects it, so adding
one never changes an existing script:

api_key = "..."                 # package-wide
retries = 6

[waterdata]
concurrency = 16                # waterdata's default profile: always active

[waterdata.overnight]           # a named profile: only when selected
concurrency = "unbounded"
parallel_chunks = 8

[ngwmn.gentle]
concurrency = 2

Everything a profile does not name still comes from below it, per setting — so the
api_key and retries above are written once and inherited by both profiles. A file
containing a key that is readable by other users warns and prints the chmod to run.

Each adapter accepts only the settings it reads, and they are the fields of its
configuration class, so a setting that means nothing to a single-shot adapter is an
error rather than a line that quietly does nothing:

>>> WqpConfiguration(concurrency=2)
TypeError: WqpConfiguration.__init__() got an unexpected keyword argument 'concurrency'

ConfigurationError: …/config.toml: 'parallel_chunks' at [streamstats] is not a setting
that table accepts. It accepts: base_url, retries, stall_timeout.

Validation is lazy: a file's structure is checked when it is parsed, a table's keys when
that adapter first resolves a setting. That keeps a malformed [nldi] table from failing
a Water Data call, and it is what lets each schema live in a module the parser cannot
import.

Precedence

Seven rungs, highest first, applied per setting:

1. A configuration instance passed to configure()
2. A profile selected in code, WaterdataConfiguration.load("bulk")
3. The setting's API_USGS_* environment variable
4. The adapter's default profile in the file — [<adapter>]
5. The package-wide keys at the top of the file
6. The adapter's own built-in preference — NWDC asks for 4 concurrent requests
7. The package built-in default

Rung 2 above rung 3 is the one place this inverts ADR 0009's environment-above-file rule.
A profile named in code is a more deliberate act than a variable inherited from whatever
started the process, and losing to that variable is what a caller would file a bug about.
The inversion covers what the profile names and nothing else. Rungs 1 and 2 both name a
single adapter and two configurations for one adapter raise, so they cannot tie; between
nested blocks the innermost decides.

Pointing an adapter at another host

An adapter's configuration may carry its base_url, which redirects that adapter's
requests and no other's — a staging instance, a mirror, a recording proxy:

with dataretrieval.configure(
    WaterdataConfiguration(base_url="https://staging.example/waterdata")
):
    df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000")

It replaces that adapter's own base and the package appends its usual paths, so for Water
Data one value moves the OGC collections, the Samples database, the statistics service and
the STAC catalog together. NGWMN is served from the same host and is deliberately left
where it was.

Code only. A file that silently redirected a data-retrieval library to another host
would be a supply-chain-shaped hazard, and a variable that was quietly ignored would leave
a caller believing they had redirected something. Both refuse out loud:

ConfigurationError: …/config.toml: 'base_url' at [waterdata] may only be set in code,
in a configure() block, never from a file.

ConfigurationError: $API_USGS_BASE_URL is set, but 'base_url' may only be set in code,
in a configure() block, never from the environment. Unset it and pass the value on the
adapter's configuration, e.g. WaterdataConfiguration(base_url=...).

The API key does not follow. It is scoped to the single host that honours it, so a
redirected call goes out without it — the host you redirected to is not the host you gave
a credential to.

Introspection

show_configuration() names the exact source of each value, including which table of the
file and — when a caller selected one — which profile. It never prints the key, and it
never raises, because a broken configuration is exactly when you reach for it. The sample
below is generated by running the function; a test rebuilds the scenario and compares the
output verbatim against both the docstring and the user guide:

>>> with dataretrieval.configure(WaterdataConfiguration.load("bulk")):
...     dataretrieval.show_configuration()
config file  /home/u/.dataretrieval/config.toml (found)
api_key          <set>  /home/u/.dataretrieval/config.toml
concurrency      16     /home/u/.dataretrieval/config.toml
retries          8      $API_USGS_RETRIES
progress         auto   built-in default
parallel_chunks  1      built-in default
stall_timeout    60s    built-in default

A built-in default is package-wide. An adapter may prefer its own for
its own calls; a value from any source above overrides both.

adapter overrides
  waterdata  parallel_chunks  8  configure() block [waterdata.bulk]
  ngwmn      concurrency      4  /home/u/.dataretrieval/config.toml [ngwmn]

profiles in the file: [waterdata.bulk]
  A profile applies only where a row above names it; select one in
  code with <Adapter>Configuration.load("<name>").

not reported: nldi (not imported, so the settings each accepts are unknown here)

The profile section lists what the file defines whether or not this run selected any of
it, which is the answer to "I added a profile and nothing changed". The last line is the
honest cost of lazy validation: NLDI is imported on demand for the geopandas extra, so a
process that has not touched it cannot say what it accepts — and an omitted service would
read as "nothing is configured for it", which is a different claim.

A worked example

The same statewide pull under two profiles, timed. On this branch (pip install -e .),
paste into a file and run:

import os, pathlib, tempfile, time

# Throwaway config file, so this never touches ~/.dataretrieval/config.toml.
cfg = pathlib.Path(tempfile.mkdtemp()) / "config.toml"
cfg.write_text("""
[waterdata.plain]
parallel_chunks = 1     # split only as far as the URL byte limit forces

[waterdata.bulk]
parallel_chunks = 16    # fan the same query out into 16 sub-requests
""")
os.environ["DATARETRIEVAL_CONFIG"] = str(cfg)

import dataretrieval
from dataretrieval import waterdata
from dataretrieval.waterdata import WaterdataConfiguration

# Every stream gage in Delaware (~250) — few enough that the default plan is a
# single sequential page walk, so the profile's fan-out is what changes.
sites, _ = waterdata.get_monitoring_locations(
    state_name="Delaware", site_type_code="ST"
)
ids = sites["monitoring_location_id"].tolist()

def timed(profile, window):
    start = time.monotonic()
    with dataretrieval.configure(WaterdataConfiguration.load(profile)):
        df, _ = waterdata.get_daily(
            monitoring_location_id=ids, parameter_code="00060", time=window
        )
    print(f"{profile:6s} {window}  {len(df):>7,} rows  {time.monotonic() - start:5.1f}s")

# Two different decades on purpose: the API caches by data window, so re-running
# one window would serve the second call from cache and hide the difference.
timed("plain", "1985-01-01/1994-12-31")
timed("bulk", "1995-01-01/2004-12-31")

Measured against the live API:

plain  1985-01-01/1994-12-31   51,391 rows    9.2s
bulk   1995-01-01/2004-12-31   68,710 rows    2.7s

Same query shape, cold windows on both sides — bulk returned 34% more rows in less than
a third of the time. The only difference is which profile was selected.

Breaking changes

Nothing here has shipped, so nothing is deprecated.

configure() no longer takes keywords. configure(api_key=...) and the per-adapter
mappings configure(ngwmn={"concurrency": 4}) are gone, along with the public
WaterdataSettings / NgwmnSettings / … TypedDicts that annotated them. Write
Configuration(api_key=...) and NgwmnConfiguration(concurrency=4). Passing anything
that is not a configuration raises and names the replacement, so an old script says what
to write rather than failing obscurely. This is the most-typed line the feature exists to
enable and making it wordier is a real cost, accepted deliberately for one shape
everywhere.

The global profile table is retired. [profiles.<name>], DATARETRIEVAL_PROFILE and
configure(profile=...) are gone; a profile now belongs to one adapter. A file still
using the old table says what to write instead:

ConfigurationError: …/config.toml: [profiles] is no longer read. A profile now belongs
to one adapter: write [<adapter>.<name>] and select it with
<Adapter>Configuration.load("<name>").

dataretrieval.config → dataretrieval.configuration, with no alias — the concept is
spelled in full everywhere else. The user-facing ~/.dataretrieval/config.toml is
unaffected and keeps its name.

RetryPolicy.from_env() → RetryPolicy.from_configuration(). It now reads a
configure() block and the config file too, so a name saying "environment" is exactly the
drift this PR exists to prevent.

dataretrieval.wateruse → dataretrieval.nwdc. Every other adapter is named for the
service it retrieves from — ngwmn, nldi, wqp, streamstats, nwis — and this one
was named for a subset of what its service offers. The National Water Availability
Assessment Data Companion serves ten modeled datasets; the water-use models are five of
them, the rest hydrologic ensembles, WRF atmospheric forcing, and CONUS 2025 assessment
outputs:

$ curl -s https://api.water.usgs.gov/nwaa-data/ | jq -r .message
Welcome to National Water Availability Assessment Data Companion v2.0.0 API

dataretrieval.wateruse remains as a deprecated alias until 2027-08-11 or later,
following the dated-removal convention nwis uses. It re-exports rather than copies, so
wateruse.get_wateruse is nwdc.get_wateruse. It is an alias for reading, not a second
name for the module: assignment and private names do not forward, and the docstring says
so. import dataretrieval stays silent — the package imports nwdc directly, so only
code naming wateruse sees the warning. Public function and constant names are unchanged.

Decision records

ADR 0009 records the layered chain. ADR 0010 scopes settings per adapter, since
surveying all seven APIs showed ADR 0009's premise — that every service accepts the same
settings — is false. ADR 0011 makes a profile a named set of settings for one adapter
and moves each schema into the module that reads it; it supersedes two clauses of ADR 0010
and three of ADR 0009, each marked in place rather than rewritten.

ADR 0010's credential finding was settled by measurement rather than argument, and
reversed the assumption we started from. NGWMN is served from the Water Data host, so it
already receives that key. Probing both live: each returns 200 with no key and a
rate-limit header with one, and alternating authenticated calls decrement a single
counter (997, 996, 996, 994, 993, 992). One key, one quota pool, two adapters. Water
Data's OpenAPI declares ApiKeyHeader; NGWMN's declares no security scheme across 34
paths, yet the gateway meters it anyway (via: … api-umbrella on every response). The key
belongs to the gateway fronting the host, not to either adapter — so per-adapter keys
would model a distinction that does not exist, and credentials keeps its host scoping
unchanged.

ADR 0011 re-probed all three hosts and narrowed that further: NWDC and NLDI meter by
address and report the same limit with or without a key, and the three hosts keep
independent counters. Sending the key there would gain nothing and would turn a stale key
into 403s on calls that work anonymously today.

The deprecated nwis deliberately gets no table: its calls pin max_retries=0, so one
could only be reported as live and then ignored.

Also included

Bug fix: API_USGS_STALL_TIMEOUT was read straight from os.environ, so it could not
be set by a configure() block or the config file and never appeared in
show_configuration() — a gap in ADR 0009's own claim that every setting resolves through
one chain. stall_timeout now resolves like the rest, and configuration is the only
module left that reads the environment for a setting.

ssl_check deliberately stays a per-call argument. It disables certificate verification,
so a config key would make a security downgrade process-wide and invisible at the call
site — and its legitimate use, a TLS-intercepting proxy, is served better by
SSL_CERT_FILE, which httpx honours on both sync and async clients and which therefore
covers every getter including the OGC ones that have no ssl_check at all. The
configuration guide documents it.

CONTEXT.md gains the configuration vocabulary and separates built-in default
(package-wide) from adapter default (what an adapter supplies in code), because they
are not the same number and show_configuration() should not imply otherwise. The
**queryables passthrough still refuses credential-shaped keyword names before request
construction, now naming the current spelling in the error a caller sees.

Verification

949 tests · mypy --strict clean across 59 files · 7/7 import contracts · 14 pre-commit
hooks · docs build adds no new warnings. The Windows path resolution is covered
specifically: _home_id mirrors expanduser per platform, since ntpath reads
USERPROFILE then HOMEDRIVE+HOMEPATH and ignores HOME entirely — where Git Bash and
MSYS both set it.

[thodson-usgs]

@davetapley, this PR is AI generated but I'm happy to incorporate any high level feedback. Focus on the public interface for now.

[davetapley]

@thodson-usgs thanks for the fast reply! The configure block is definitely be my preference.

Config file has the same problem as env var: yes, technically possible to write it during runtime, but feels like it should be set up before the program runs, which is clunky if invoking from other parts of a different codebase.

[thodson-usgs]

I considered using a class: with Configuration:, which is a pattern I've seen in other libraries. Claude didn't like it for this case, but I plan to reconsider before implementing.