HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers

[ss77892]

What changes were proposed in this pull request?

HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers

XceiverClientGrpc.initStreamRead arms a gRPC deadline (withDeadlineAfter, ozone.client.read.timeout, default 30 seconds) on the long-lived streaming ReadBlock call introduced by HDDS-13974. A gRPC deadline bounds the entire call, not a single request, so every block stream is cancelled with DEADLINE_EXCEEDED 30 seconds after it opens, even when it is perfectly healthy. The failure then becomes permanent on the client side:

1. The cancellation lands in StreamingReader.onError and completes the reader future exceptionally. StreamBlockInputStream never re-initializes a reader whose future has failed, so checkError replays the stale exception on every subsequent read of that block for as long as the input stream stays open.
2. The failure is not classified as retryable: isConnectivityIssue only accepts UNAVAILABLE, and checkError wraps the original exception in a new IOException whose cause chain (IOException -> ExecutionException -> StatusRuntimeException) the retry classification cannot unwrap. So the failover/refresh path in handleExceptions never engages.

Long-lived readers hit this hard. On an HBase-on-Ozone cluster, RegionServers keep store file input streams open indefinitely; after each stream's first 30 seconds, every pread through it fails instantly. A YCSB read workload showed a steady ~42 percent error rate (reads served from HBase block cache or memstore still succeeded, masking the problem for the first 30 seconds of each stream's life). Short-lived readers (CLI, file copies) close before the deadline fires.

A related capacity problem: initStreamRead held a permit from the shared request semaphore for the stream's whole lifetime, so a client with many open files could exhaust the permits, starve short RPCs, and block new streams indefinitely (observed as 30-second RegionServer-wide stalls).
What fix does:
Remove the trigger, make recovery work, and separate stream capacity from request capacity:

1. Drop the call-lifetime deadline from the streaming call. Per-request timeliness is already enforced by ozone.client.stream.read.timeout in streamRead flow-control waits and StreamingReader.poll, which fails a stuck request with a retryable TimeoutIOException.
2. Never poison the stream. handleExceptions now marks the reader failed and, on the non-retryable path, tears down the failed stream and resets the request high-water mark before rethrowing, so the next read builds a fresh stream instead of replaying the stale failure. closeReader also resets requestedLength, since requested-but-unreceived data dies with the stream (this also fixes read-after-unbuffer resuming from a stale prefetch offset and stalling).
3. Make the failure classifiable. checkError rethrows the original IOException from the ExecutionException cause so retry classification sees the real failure, and isConnectivityIssue treats DEADLINE_EXCEEDED as a connectivity issue alongside UNAVAILABLE, enabling datanode failover.
4. Cancel failed streams instead of half-closing them. closeReader now calls cancel on a failed call; onCompleted only half-closes it, which would leave the RPC alive against a stuck datanode.
5. Account long-lived streams separately from short RPCs. Streams now take permits from a dedicated semaphore sized by a new config, ozone.client.stream.read.max-concurrent-streams. Any non-positive value (the default) inherits hdds.ratis.raft.client.async.outstanding-requests.max, preserving the capacity streams had when they shared the request semaphore. Acquisition waits up to ozone.client.stream.read.timeout and then fails with an actionable message instead of blocking indefinitely; a failed initStreamRead releases its permit.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16241

How was this patch tested?

UT has been added.
Basic freon/cli workloads to confirm that basic functionality hasn't been broken
HBase on Ozone cluster with YCSB workloads. The rate of failures dropped from ~60% to less than 1%. This 1% would be addressed as a separate jira because it has different root cause.

[ss77892]

One note. The change grew beyond removing the deadline: cross-review of the initial fix by several AI models (Claude Fable 5, GPT-5.6 Terra) revealed additional latent defects in the streaming read path: permanent stream poisoning, broken retry classification, half-closed failed calls, a stale prefetch offset after unbuffer, and request-permit exhaustion by long-lived streams. Each was confirmed with a reproducing unit test before its fix was included.
CC: @szetszwo, @sodonnel, @chihsuan

[yandrey321]

@ss77892 why stream is getting closed after 30 seconds with DEADLINE_EXCEEDED?

[ss77892]

@ss77892 why stream is getting closed after 30 seconds with DEADLINE_EXCEEDED?

That's in the jira description: XceiverClientGrpc.initStreamRead sets a gRPC deadline of ozone.client.read.timeout (30 seconds). A gRPC deadline bounds the whole call, and since HDDS-13974 one stream stays open per block for the lifetime of the input stream, so every stream is cancelled with DEADLINE_EXCEEDED 30 seconds after it opens, even when healthy.

[chihsuan]

Thanks for putting this together! @ss77892 I was able to reproduce the failure end to end by shortening ozone.client.read.timeout, and the new tests
are indeed red without the production changes.

Since this PR fixes multiple reproducible issues, would it be worth considering separate Jira/PRs?I think that could make each behavior change easier to understand and review. I’ve also left two inline questions for your consideration. Thanks!

[chihsuan]

This also changes the classic BlockInputStream. DEADLINE_EXCEEDED now triggers an OM block-location refresh instead of a simple retry. Is this intentional?

[chihsuan]

I may be missing an existing safeguard, but could long-lived streams keep server-side files open for an extended period? The current limits don’t seem to apply across the whole datanode. Is there another server-side limit or cleanup mechanism?