Two Config ceilings are preallocations, not ceilings: WorkPort's ring and the buffer pool cost 4.2 MiB in an idle Loop

[proggeramlug]

Problem

slots.rs opens with the principle this project already committed to:

Index-addressed storage that allocates in pages, so a capacity ceiling costs nothing until it is used. […] Building every slot at construction makes that number a preallocation instead of a ceiling, which forces hosts to choose between refusing work and paying for a loop they mostly do not use.

turnloop#75 applied that to the handle table. Two other structures sized from Config never got the treatment, and together they are 4.2 MiB of resident memory in an idle Loop.

Measurement

One Loop per process (RSS is a high-water mark, so building several in one process makes every later row inherit the earlier ones — the first version of this probe did exactly that and read as noise). macOS/aarch64, turnloop 0.1.0-alpha.5, RSS delta across Loop::new:

config	RSS delta	vs baseline
Perry's net profile (max_handles 65 536, max_operations 32 768, pooled_buffers 64 × 16 KiB, post_capacity 256)	4 800 KiB	—
max_operations 32 768 → 1 024	1 648 KiB	−3 152 KiB
max_operations 32 768 → 256	1 504 KiB	−3 296 KiB
pooled_buffers 64 → 0	3 792 KiB	−1 008 KiB
max_handles 65 536 → 1 024	4 880 KiB	−0 (noise)
post_capacity 256 → 16	4 816 KiB	−0 (noise)
all four minimised	496 KiB	−4 304 KiB

max_handles costs nothing, exactly as #75 intended. The other two are the whole floor.

Cause 1 — WorkPort's ring is sized by max_operations (≈3.15 MiB)

driver.rs builds WorkPort::new(config.max_operations, …), and WorkPort::new builds Queue::new(capacity.max(2).next_power_of_two()). Queue::new collect()s every slot and writes state: AtomicUsize::new(0) into each one, so the pages are resident, not merely reserved.

The deeper point is that this ring should not be sized by max_operations at all. max_operations is the ceiling on outstanding I/O operations — a server holding 8 000 idle connections with a read armed on each legitimately needs it large. WorkPort carries results from the blocking pool, whose in-flight count is bounded by the pool's thread count, typically ncpu. A 32 768-slot result ring for a pool that can have ~8–16 jobs outstanding is pure waste, and it forces exactly the choice the slots.rs header says hosts should not have to make: Perry either accepts a connection ceiling or pays 3 MiB in every process that starts a loop.

Suggested fix: give the work port its own capacity — a separate Config field, or derive it from the blocking pool's thread count — rather than reusing max_operations. The ring can stay eagerly allocated (a turn must not allocate); it just needs to be sized by the thing it actually bounds.

Cause 2 — the buffer pool is preallocated (≈1.0 MiB)

pooled_buffers × pooled_buffer_size is materialised at construction: 64 × 16 KiB = 1 024 KiB, and the measurement shows 1 008 KiB. For a loop that never does I/O, that is a megabyte of resident memory for buffers nothing will lease.

Suggested fix: grow the pool on demand up to pooled_buffers instead of preallocating it. That keeps the DESIGN §10 rule 1 property — steady state allocates nothing, once the working set is warm — while making the ceiling a ceiling. It is the same trade #75 made for handles.

Why this matters concretely

Perry is measuring turnloop against its previous tokio-based wait driver. At 8 000 idle keep-alive connections turnloop wins decisively — 13 191 bytes per connection vs 30 894, −57.3 % — but it starts 3.61 MiB above tokio at zero connections, so the two arms only break even at ~214 connections. Cause 1 alone is 3.15 MiB of that 3.61 MiB; the two together turn the one remaining memory regression into a win at every connection count.

Neither fix changes any contract: max_operations keeps meaning what it means, the buffer pool keeps its ceiling, and steady-state allocation behaviour is unchanged.

[proggeramlug]

Correction: cause 1 as I described it is wrong, and the fix I suggested would panic

I need to retract the suggested fix for WorkPort's ring before someone implements it.

I wrote that the ring "should not be sized by max_operations at all" and suggested deriving it from the blocking pool's thread count. That is wrong, and blocking.rs says so at the push site:

// One reserved core operation credit per job remains held until delivery.
// Thus this separate queue (>= max_operations) cannot overflow.
assert!(
    self.queue.push(result).is_ok(),
    "blocking completion credit invariant"
);

A full ring is an assert!, not a refusal. The invariant that makes it safe is that every submitted blocking job holds a core operation credit until its result is delivered, so outstanding undelivered results can genuinely reach max_operations and the ring must be that large.

I had reasoned from PoolConfig::queue_capacity bounding in-flight jobs. It does not bound the thing that matters: reserved counts slots reserved before work exists (typed filesystem requests), and a job whose result is sitting in the ring undelivered has already freed its queue slot while still holding its operation credit. So a host can fill the queue, let workers drain it into the ring, and refill — repeatedly — until the credits run out. Sizing the ring by threads + long_threads_max would turn that into a panic in a shipped binary.

A correct fix has to separate the accounting, not just the number. Give undelivered blocking results their own ceiling — a PoolConfig field — enforced at blocking submission the way queue_capacity already is (ResourceLimit), decremented on delivery, with the ring sized to that ceiling. The assert! invariant then holds against the new counter rather than against max_operations, and a host that wants 32 768 outstanding I/O operations no longer pays 3.15 MiB for a blocking-result ring it will never fill. I have not implemented this; it changes an admission path and deserves its own review.

Cause 2 is correct, and is now fixed

pooled_buffers really was a preallocation: BufferPool::new built and zeroed every buffer up front. Buffers are now minted on demand up to the same ceiling and pooled for reuse afterwards, so the steady state is unchanged — a loop at its high-water mark still allocates nothing per turn — while an idle loop costs nothing.

Re-measured with the same probe, one Loop per process:

config	before	after	saved
Config::default() (256 x 16 KiB)	5 072 KiB	960 KiB	−4.02 MiB
a 64 x 16 KiB host profile	4 800 KiB	3 808 KiB	−0.97 MiB
pooled_buffers = 0 (control)	3 792 KiB	3 776 KiB	~0

The control is the check that this is the buffer pool and nothing else.

Branch fix/88-ceilings-not-preallocations. portable_tests' zero-allocation assertion warms the pool to its high-water mark first, since the first mint is warm-up rather than steady state; that does not soften it, and stubbing out reuse still fails three tests.

So the issue stands, but it is now one cause, not two: the ring is the remaining 3.15 MiB and it needs the accounting change above, not a smaller number.