[Bug] (fe) Client disconnect causes orphaned query hung in active_queries and leaks workload group queue slots

[GJ100]

Search before asking

• I had searched in the issues and found no similar issues.

Version

3.0.x / 4.1.x / master

What's Wrong?

When a client application (e.g., a microservice or web backend with a 3–5 second query timeout) disconnects due to client-side timeout while the query is still waiting on the BE (e.g. waiting for rowset/delete-bitmap lock or waiting in queue during heavy ingestion/compaction), the following cascade failure occurs:

1. FE Network Layer Removes Connection Silently:
   FE's AcceptListener detects the TCP FIN/RST from the client and triggers connection.setCloseListener(...) -> connectScheduler.getConnectPoolMgr().unregisterConnection(context). The connection is removed from connectionMap.

2. No Cancel Signal Is Dispatched:
   ConnectPoolMgr.unregisterConnection() removes the context from connectionMap, but does NOT call context.cancelQuery(). No cancel signal is sent to the Coordinator, and no cancel_plan_fragment RPC is dispatched to BE nodes.

3. Query Escapes TimeoutChecker Entirely (Ghost Query):
   FE's background TimeoutChecker (checkTimer) iterates strictly over connectionMap.values(). Because the connection was already removed in step 1, checkTimeout() is never called again for this context. The query bypasses query_timeout (e.g. 300s) and hangs indefinitely in QeProcessorImpl / information_schema.active_queries (observed running for >3400 seconds / 57 minutes as RUNNING).

4. FE Worker Thread Stalls in coordBase.getNext():
   Because the query was actively executing and ReadListener.suspendAcceptQuery() had already suspended reading on the socket, the worker thread remains blocked waiting for BE results. Since no data is written to the closed socket, no IOException / EPIPE is raised to break the loop.

5. Workload Group Queue Slot Leak & Cluster Stall:
   Because Coordinator.close() is never executed, the query's QueueToken is never returned to the QueryQueue. When all slots (max_concurrency) in the workload group are occupied by these orphaned queries, all subsequent queries in that workload group are stuck in WAIT_IN_QUEUE forever, until they fail with query queue timeout.

What You Expected?

When a client disconnects or closes the connection:

1. unregisterConnection() must immediately cancel any active query on that connection.
2. The Coordinator must abort BE fragment execution via cancel RPCs, release the workload group's QueueToken, and unblock the worker thread.
3. The query must be immediately removed from QeProcessorImpl and information_schema.active_queries.
4. Workload group slots must be promptly freed for queued queries.

How to Reproduce?

1. Create a workload group with strict concurrency:

   CREATE WORKLOAD GROUP wg_test PROPERTIES ('max_concurrency'='1', 'max_queue_size'='10', 'queue_timeout'='60000');

2. Assign wg_test to user test.
3. Submit a query that takes several seconds (or simulate a lock wait / slow scan on BE).
4. Abruptly kill the client process (send TCP FIN/RST) within 2 seconds before any result row is returned.
5. Inspect information_schema.active_queries:
   • The query remains RUNNING long after query_timeout.
6. Submit a second query from a new connection:
   • The new query stays in WAIT_IN_QUEUE and eventually times out.

Anything Else?

Observed in production where point-lookups (SELECT ... FROM tbl WHERE user_id = ...) encountering lock wait were abandoned by client microservices after 3s. The orphaned queries lived for >57 minutes in active_queries while subsequent queries piled up in WAIT_IN_QUEUE.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_e692bb511ebb

Initial assessment

This is a valid FE connection-teardown/cancellation gap, but the reported “indefinite query after query_timeout” mechanism is only partially established. A disconnected MySQL session can leave its active query running until an execution-side timeout or another terminal event. That is enough to retain a workload-group slot unnecessarily and is worth fixing. However, static inspection does not support the stronger conclusion that removal from connectionMap by itself bypasses every query timeout.

I checked the current heads named by the report: master at 44e3ae2b, branch-4.1 at 8d2018bd, and branch-3.0 at 4090f6c4. The relevant teardown structure is present in all three.

Confirmed from code

• The close listener calls only unregisterConnection(context) (AcceptListener).
• ConnectPoolMgr.timeoutChecker() iterates only connectionMap.values(), while unregisterConnection() closes a transaction and removes the context without cancelling its active executor (timeout scan and unregister). Therefore, after the close callback wins the race, that context no longer receives ConnectContext.checkTimeout() calls. This part of the report is correct.
• For a normal coordinator query, cancellation already has the required downstream behavior: Coordinator.cancel() cancels the queue token and invokes asynchronous BE-fragment cancellation (source, RPC path). When the worker unwinds, Coordinator.close() releases the queue token (source), and normal query finalization removes the query from QeProcessorImpl (source). The missing link is invoking cancellation when the transport closes.
• Until that happens, no write is required while coordBase.getNext() waits, so losing the client socket need not produce an EPIPE that wakes the worker. The queue token remains held until the query exits and the finally block closes the coordinator (source, cleanup).

Important qualification

The ordinary BE-result path also enforces an absolute execution deadline independently of TimeoutChecker: Coordinator derives timeoutDeadline from queryOptions.execution_timeout and gives it to each ResultReceiver (source); ResultReceiver.getNext() uses that deadline in its timed future wait and returns a timeout status (source). Waiting for a workload-group token is likewise bounded by the smaller applicable queue/execution timeout (source).

Consequently, “the context was removed from connectionMap, therefore this getNext() can run forever past its effective execution timeout” is not yet proven. The observed >3,400-second lifetime may reveal another blocked path or timeout discrepancy, but it needs runtime evidence. Also, although the production SQL is described as a point lookup, the short-circuit PointQueryExecutor branch does not register a coordinator in QeProcessorImpl (source); that path cannot directly explain the reported active_queries row plus workload-group token. An EXPLAIN/profile is needed to identify the actual execution path.

Information needed to confirm the long-lived symptom

Please add:

1. Exact FE and BE version strings/commit IDs for one occurrence, including whether the cluster was mixed-version.
2. The query ID and timestamped active_queries snapshots, plus the effective session/user query_timeout, any SET_VAR override, and the profile/query-options execution timeout for that same query.
3. FE logs for that query ID from before disconnect through the point where it exceeded the expected timeout, especially get result timeout, kill query timeout, Cancel execution, and deregistration messages.
4. An FE thread dump while the query is already past its effective timeout. The exact blocking frame is necessary; stating coordBase.getNext() alone does not distinguish the receiver, short-circuit, or another coordinator path.
5. A deterministic reproducer for the BE wait (including table model and the operation holding the rowset/delete-bitmap lock), the client/driver and close method, and EXPLAIN or the query profile. The current “slow query” step proves prompt disconnect cancellation only if the blocking condition is repeatable.

Recommended next steps

1. Treat connection teardown as a cancellation source: cancel the active execution with a specific client-disconnected CANCELLED reason before removing/releasing session state. Keep this idempotent and race-safe. Avoid directly removing the query from QeProcessorImpl in the close callback; cancellation should wake the execution thread, whose existing finally path owns coordinator close, queue-token release, profile finalization, and query deregistration.
2. Do not blindly block the XNIO close-listener thread. ConnectContext.cancelQuery() reaches StmtExecutor.cancel(), which may wait for a CancelableCommand; the implementation should preserve prompt asynchronous transport teardown while still propagating coordinator/BE cancellation.
3. Coordinate the patch with open PR #68101, which is actively changing ConnectPoolMgr.unregisterConnection(). Its current MySQL session-release hook is intentionally a no-op, so that refactor does not by itself close this gap.
4. Add a focused unit test that unregistering a context with an active executor dispatches cancellation exactly once, plus an integration/regression test with max_concurrency=1 that closes the client during a deterministic BE wait and asserts: the first query is cancelled, the queue token is released, the active_queries row disappears, and a second query starts without waiting for queue timeout. Cover both FIN and RST/dropped-client behavior and the race with normal query completion.

Triage metadata: the issue currently has no labels and no assignee; the author is willing to submit a PR. Applying the repository’s FE/query-execution/workload-group bug ownership labels would help route it.

[GJ100]

Thanks for the thorough and helpful review!

Regarding the points raised:

1. Why it stayed alive for >3400s:
   In our production environment, the effective query_timeout for this database/user session was configured to 3600s (1 hour) (for occasional large ETL queries), while client microservices have a strict short network timeout of 3-5s.
   When a point-lookup or simple query encountered a lock wait / heavy compaction on BE, the client timed out at 3s and disconnected, but because of this gap, the orphaned query held the workload group slot for nearly an hour (approaching 3600s), starving all queued short queries in the same group!

2. Non-blocking cancellation in XNIO close listener:
   Agreed! In our patch for ConnectPoolMgr.unregisterConnection(), we will ensure cancellation is invoked with needWaitCancelComplete = false (or asynchronously) so that the XNIO transport thread is never blocked.

3. Coordination with PR [refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions #68101:
   We will base our patch on current master and coordinate with PR [refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions #68101 to ensure the MySQL disconnect path properly triggers non-blocking cancellation before session teardown.

We will prepare the PR with the unit tests and non-blocking cancellation accordingly!