Performance degrades under high load because of ConnectionNotAvailable exceptions

[kramar11]

Hi,

I'm using httpcore in a project and while doing some performance tests with locust, I noticed, that the performance degraded under very high load. After digging, I found, that the AsyncConnectionPool may (and under load will!) assign connections to more than one pool_requests, which then results in ConnectionNotAvailable Exceptions (lots of them) which lowers the performance as the pool_request needs again to wait for an available connection.

I then made a simple modification in _assign_requests_to_connections so that already assigned connections are not considered as available, which greatly increased performance and totally got rid of ConnectionNotAvailable Exceptions.

    def _assign_requests_to_connections(self) -> list[AsyncConnectionInterface]:
        ...
        # Assign queued requests to connections.
        queued_requests = [request for request in self._requests if request.is_queued()]
        assigned_connections = [ request.connection for request in self._requests if request.connection is not None ]
        for pool_request in queued_requests:
            origin = pool_request.request.url.origin
            available_connections = [
                connection
                for connection in self._connections
                if connection not in assigned_connections and connection.can_handle_request(origin) and connection.is_available()
            ]
        ...

Note: Only the 2 lines with the assigned_connections var were added/changed.

My test results:

-	without modifications	with above modifications
# Requests	51739	69643
Average (ms)	42.64	31.62
RPS	431	580

Don't know if this is the right fix to avoid the ConnectionNotAvailable Exceptions but it increases high load performance by more than 30 percent. I think it could be reasonable to keep two lists of connections, those which are available and those which are assigned - and move assigned connections accordingly.

What do you think?

[TayfurYldz]

The observation is real, but excluding every connection that already has an assigned request would also throw away HTTP/2 multiplexing.

Current pool assignment intentionally allows more than one queued request to be assigned to a connection for which is_available() is true. That is necessary for HTTP/2, where one connection can accept many concurrent streams. For HTTP/1.1, the connection transitions out of the idle/available state when the first request actually acquires it, and a second assignment can lose that race and raise ConnectionNotAvailable; the pool catches that exception, clears the assignment, and retries.

So the proposed assigned_connections filter improves the HTTP/1 contention case by effectively reserving a connection at assignment time, but globally applying it serializes assignment to an HTTP/2 connection to one request at a time. That is the wrong invariant for the generic pool.

A robust optimization needs capacity-aware reservation rather than a boolean “already assigned” set. HTTP/1 has capacity one; HTTP/2 has dynamic capacity based on negotiated concurrent streams. Today is_available() answers whether a connection can potentially accept work, but it does not reserve capacity atomically with pool assignment, which is why the pool deliberately treats ConnectionNotAvailable as a recoverable race.

I'd benchmark a fix that adds reservation/capacity semantics at the connection interface (or an equivalent atomic acquire) and include both cases in tests: high-contention HTTP/1 should stop producing avoidable retry exceptions, while concurrent HTTP/2 requests must still multiplex over a single connection.