Explain why prometheus-exporter marks an endpoint unavailable

[maoueh]

An alert firing on substreams_healthcheck_status gave no way to tell an unreachable endpoint from an unauthenticated, overloaded or merely late one, and a flapping endpoint could produce no logs at all. This adds a failure taxonomy, separates connection setup from the request, and logs every failed poll.

• Every failure is classified into a reason (connect, connect_timeout, invalid_request, request_timeout, stream_error, stale_block, no_data), exposed on a new substreams_healthcheck_failure_count{reason,grpc_code} counter and carried in the logs. Metrics are declared through dmetrics.

• Connection establishment gets its own --connect-timeout (default 10s), separate from --timeout, which now covers the Blocks request alone. gRPC dials lazily, so DNS/TLS/LB resolution was previously charged to the request budget and a slow connection was reported as an endpoint failure — the source of the waiting for new LB policy update: context deadline exceeded errors. The exporter now waits for the channel to be READY first, and reports substreams_healthcheck_connect_duration_ms and substreams_healthcheck_stream_duration_ms separately; substreams_healthcheck_duration_ms keeps its old meaning of the two combined. Both deadlines carry a cause, so the error names which budget was exceeded instead of saying context deadline exceeded.

• Every failed poll is logged, not just the transition into unavailable, with reason, gRPC code, both durations and the consecutive failure count; recovery logs the downtime and how many polls failed. A block age above half of --max-freshness is logged too, so an alert on substreams_healthcheck_block_age_ms is no longer silent.

• New substreams_healthcheck_consecutive_failures gauge to alert on instead of status when single-poll hiccups should be ignored, and block_age_ms now resets to NaN when a poll returns no block instead of reporting the age of the last block ever seen.

• The exporter speaks sf.substreams.rpc.v4.Stream/Blocks only. The v3-to-v2 fallback is removed — it closed the connection and then kept reading from it — and --force-protocol-version accepts only 4 (or 0), kept for the protocol versions to come; an invalid value is now rejected at startup instead of being silently ignored.

• Fixes a panic on inconsistent label cardinality when endpoints are given different sets of query-parameter labels.

Verified end-to-end against the live fleet with a real API key: the v4 success path works on mainnet.eth and mainnet.sol, and connect_timeout, stream_error/Unauthenticated, stale_block, the half-freshness notice, the NaN block-age reset and the version-flag rejection were each exercised.

[dfuse-bot]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:7520bc4

📦 Image Reference ghcr.io/streamingfast/substreams:7520bc4

digest	sha256:fb153ff514dd7f28ebab74dbbd20c303bb401263292090986c443ed4539d9d63
vulnerabilities
platform	linux/amd64
size	124 MB
packages	380

📦 Base Image ubuntu:24.04

also known as	c1ca75be10a22ea09ff0b7bbe8b82ee03553a4f9b795030ee2ec921e42418fc8 noble noble-20260810
digest	sha256:1e0a86e57d247923571b75e0aaf48a1449cf8c543d51fb3e07a4a7d7bfa79316
vulnerabilities