Skip to content

pubsub: concurrent ACK/modACK queue additions exceed maxMessages in 6.0.1 #9303

Description

@vienneraphael

Please make sure you have searched for information in the following guides.

Library Name

@google-cloud/pubsub 6.0.1

A screenshot that you have tested with "Try this API".

Not applicable: this reproduction isolates a client-side batching problem before requests reach the Pub/Sub API.

It uses the installed SDK’s real subscriber, messages, and acknowledgement queues. Only the Google RPC boundary is replaced with a recorder. No credentials, cloud resources, or API calls are required.

Link to the code that reproduces this issue. A link to a public Github Repository or gist with a minimal reproduction.

https://gist.github.com/vienneraphael/0148221c957167d7adf033c391df2ba4

A step-by-step description of how to reproduce the issue, based on the linked reproduction.

  1. Download reproduce.cjs from the linked gist into an empty directory.
  2. Install the unpatched published package:
npm init -y
npm install --ignore-scripts --no-audit --no-fund @google-cloud/pubsub@6.0.1
  1. Run:
node reproduce.cjs
  1. Observe the JSON output. Sequential additions respect the configured limit, while concurrent additions exceed it for both AckQueue and ModAckQueue.

Environment: Node.js 22.22.2, npm 10.9.7, macOS 26.6.2 (ARM64). We downloaded the published gist into a fresh directory and verified the results with a clean npm installation.

The script deliberately exercises internal queue classes to isolate this behavior. It asserts that every requested ACK ID is recorded exactly once and reports batching-limit violations through respectsLimit: false.

A clear and concise description of what the bug is, and what you expected to happen.

Concurrent calls to MessageQueue.add() can produce acknowledgement and modify-ack-deadline RPC batches containing more ACK IDs than the configured maxMessages.

Expected: Each outgoing RPC contains at most maxMessages ACK IDs, including when additions overlap.

Actual:

Queue Additions maxMessages Messages added Recorded RPC batch sizes
AckQueue Sequential 1 3 [0, 1, 0, 1, 0, 1]
AckQueue Concurrent 1 3 [0, 0, 0, 3]
AckQueue Concurrent 2 6 [1, 1, 1, 3]
ModAckQueue Sequential 1 3 [1, 1, 1]
ModAckQueue Concurrent 1 3 [3]
ModAckQueue Concurrent 2 6 [1, 1, 1, 3]

All four concurrent cases exceed the configured limit. Both sequential controls pass.

The ACK queue also invokes the recorded RPC with empty batches in some cases; the primary issue here is exceeding the maximum batch count.

A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

MessageQueue.add() explicitly checks the configured message-count and byte limits before adding a message, flushing the existing batch when necessary. This indicates the limits are intended to constrain outgoing RPC batches.

The suspected race is between that capacity check and insertion:

  1. A call checks the current batch size.
  2. It reaches await this.flush(reason) before appending its message.
  3. Other concurrent calls can reach the same point.
  4. The calls resume and append to the shared queue without rechecking its capacity.

This appears to explain why sequential additions respect the limit while concurrent additions do not. In particular, setting maxMessages: 1 does not reliably produce one ACK ID per RPC.

Related issue and prior fix

This is related to googleapis/nodejs-pubsub#2026, addressed by googleapis/nodejs-pubsub#2027. That issue concerned failed ACK requests being re-queued by MessageQueue.handleRetry() without respecting batch limits.

The reproduction here exercises concurrent initial additions through MessageQueue.add(), with every recorded RPC succeeding and no retries. It still exceeds maxMessages on a clean, unpatched 6.0.1 installation. The prior retry-path fix therefore does not cover this reproduction.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: pubsubIssues related to the Pub/Sub API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions