perf: configurable pool buffer retention limit via SetPoolBufferLimit

[xe-nvdk]

Summary

Closes #62.

PutEncoder/PutDecoder drop internal buffers above 32KiB, so workloads with consistently larger messages re-grow the buffer on every pooled use. This adds SetPoolBufferLimit(n) to raise the retention limit. Of the three options in the issue, the configurable limit was chosen over (1) raising the default — which would silently change memory retention for all users — and (3) adaptive sizing — which adds racy bookkeeping for marginal benefit.

Design notes:

• Default (32 KiB) unchanged. Values below the default clamp to it; n <= 0 restores it.
• Limit stored in an atomic.Int64; the Put path checks the constant default first (poolBufOversized), so small buffers — the common case — never pay for the atomic load. Verified inlinable.
• Applied to both encoder wbuf and decoder buf, per the issue.

Benchmarks (Apple M3 Max, benchstat, count=8)

100KB payloads, limit raised to 256 KiB vs default:

benchmark	default limit	256k limit	delta
MarshalAppendLarge100KB	9.31µs, 104KiB, 3 allocs	4.61µs, 24B, 1 alloc	-50% ns/op
DecodeLargeStream100KB	16.14µs, 208KiB, 3 allocs	10.23µs, 104KiB, 1 alloc	-37% ns/op

No regression at default settings vs v6 (the atomic is not touched for small buffers):

benchmark	v6	branch	delta
StructMarshal	368.5n	363.6n	-1.33%
StructUnmarshal	338.8n	338.9n	~

Testing

• pool_test.go (first internal package msgpack test file — needed to inspect unexported wbuf/buf): retention under raised limit, dropping under default, clamping behavior.
• go test ./... and go test -race ./... pass.
• Internal review pass (correctness+security, quality+perf) completed; only finding was the missing CHANGELOG entry, fixed in the follow-up commit.

[gemini-code-assist]

Code Review

This pull request introduces a configurable pooled buffer retention limit via SetPoolBufferLimit(n) to allow workloads with large messages to retain grown buffers in the encoder/decoder pools instead of re-allocating. It includes benchmarks, implementation using atomic operations, and unit tests. Feedback points out that the unit tests in pool_test.go rely on the non-deterministic behavior of sync.Pool, which can lead to flaky test failures during garbage collection or goroutine rescheduling. It is recommended to test the underlying poolBufOversized function directly to ensure deterministic test results.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

These tests rely on sync.Pool returning the exact same encoder/decoder instance that was just put back. However, sync.Pool behavior is non-deterministic: the pool can be cleared at any time by the garbage collector (GC), or it may return a new instance if the goroutine is rescheduled onto a different P or thread. When this happens, GetEncoder() or GetDecoder() will return a newly allocated instance with a nil buffer, causing the test to fail with a false positive (e.g., wbuf cap=0, want >= 102400 retained under raised limit).

To make the tests 100% deterministic and robust, we should test the underlying poolBufOversized function directly with various capacities and configured limits, rather than relying on sync.Pool side effects.

func TestPoolBufferLimit(t *testing.T) {
	// Test default limit (32 KiB)
	if poolBufOversized(16 * 1024) {
		t.Error("16 KiB buffer should not be oversized under default limit")
	}
	if poolBufOversized(32 * 1024) {
		t.Error("32 KiB buffer should not be oversized under default limit")
	}
	if !poolBufOversized(33 * 1024) {
		t.Error("33 KiB buffer should be oversized under default limit")
	}

	// Test raised limit (256 KiB)
	SetPoolBufferLimit(256 * 1024)
	defer SetPoolBufferLimit(0)

	if poolBufOversized(100 * 1024) {
		t.Error("100 KiB buffer should not be oversized under 256 KiB limit")
	}
	if poolBufOversized(256 * 1024) {
		t.Error("256 KiB buffer should not be oversized under 256 KiB limit")
	}
	if !poolBufOversized(257 * 1024) {
		t.Error("257 KiB buffer should be oversized under 256 KiB limit")
	}
}

[xe-nvdk]

Good catch — fixed in 385e866.

I confirmed the flake is real rather than theoretical. GOGC=1 with -count=200 did not reproduce it, but forcing the exact interleaving you described — two runtime.GC() calls between PutEncoder and GetEncoder — fails with precisely the symptom you predicted:

--- FAIL: TestOriginalPatternWithGC (0.00s)
    wbuf cap=0, want >= 102400 retained under raised limit

So in CI it's a coin flip on whether a GC lands in that window.

Took your suggestion: retention is now covered by a table test over poolBufOversized — the predicate that actually makes the drop/retain decision — across default, raised, and clamped limits, including boundary values (at-limit vs one byte over). The Get/Put tests are kept but reduced to the one direction that holds regardless of pool behavior: an oversized buffer is never handed back. A fresh encoder with a nil buffer also satisfies that, so it can't false-fail.

Verified: -count=200 under GOGC=1, plus full go test ./... and go test -race ./....