Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions dataretrieval/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@
RateLimited,
RequestTooLarge,
ServiceUnavailable,
SkippedItemWarning,
SkippedRatingWarning,
TransientError,
Unchunkable,
URLTooLong,
Expand Down Expand Up @@ -95,6 +97,8 @@
"RateLimited",
"RequestTooLarge",
"ServiceUnavailable",
"SkippedItemWarning",
"SkippedRatingWarning",
"TransientError",
"URLTooLong",
"Unchunkable",
Expand Down
44 changes: 43 additions & 1 deletion dataretrieval/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
aren't a plain status: :class:`RequestTooLarge` (with :class:`URLTooLong` /
:class:`Unchunkable`), :class:`NetworkError` (a failed connection, per above),
:class:`NoSitesError`, and :class:`ConfigurationError` for an unusable setting.
:func:`error_for_status` maps a status to its type.
:func:`error_for_status` maps a status to its type. The *warning* side of the
taxonomy lives here too: :class:`SkippedItemWarning` (specialized by
:class:`SkippedRatingWarning`), a per-item skip inside a batched retrieval.

This module has no third-party runtime dependencies -- ``httpx`` is imported only
for type checking. Any module can therefore import it without pulling in pandas
Expand Down Expand Up @@ -42,6 +44,8 @@
"NetworkError",
"NoSitesError",
"ConfigurationError",
"SkippedItemWarning",
"SkippedRatingWarning",
"error_for_status",
"parse_retry_after",
]
Expand Down Expand Up @@ -289,6 +293,44 @@ def __str__(self) -> str:
)


# --- Skipped work ---------------------------------------------------------


class SkippedItemWarning(UserWarning):
"""One item of a batched retrieval was skipped; the rest were returned.

The policy for batch getters whose items are independent documents: an
item that fails *deterministically* -- so retrying would reproduce the
failure -- is dropped from the result under a warning naming it, because
aborting would discard every other item's data over one bad entry.
Transient failures (429 / 5xx / timeouts / connection drops) are never
skipped -- they are retried and, if retries run out, raised as a
resumable interruption. Rate limiting in particular is systematic, so
skipping there would silently drop most of a batch; that silent loss is
the failure mode this policy exists to prevent.

A warning rather than a log line so it is visible by default. To make
any skip fatal (strict all-or-nothing behavior)::

warnings.filterwarnings("error", category=SkippedItemWarning)

Getters emit a subclass naming their surface (e.g.
:class:`SkippedRatingWarning`), so a filter can also target one getter.
"""


class SkippedRatingWarning(SkippedItemWarning):
"""A rating feature was skipped by
:func:`dataretrieval.waterdata.get_ratings`.

Emitted when a single STAC feature fails deterministically -- a stale
catalog entry (404 on its data asset), a feature carrying no data asset,
a malformed RDB file. The failed feature's id is absent from the returned
dict. See :class:`SkippedItemWarning` for the policy and how to escalate
a skip to an error.
"""


def error_for_status(
status: int, message: str, *, retry_after: float | None = None
) -> DataRetrievalError:
Expand Down
Loading