MPSC Support Possible?

[sphw]

I might be missing something very obvious here, but it seems to me that BBQueue is very well setup to act as an MPSC queue. When taking a grant there is already a "lock" of sorts with write_in_progress. It seems like you could create a function that allows for the creation of multiple producers from the same underlying buffer. Something similar to #78 would work. Of course this would make writes effectively have a lock, but there are cases where that might be an acceptable trade-off.

Are there any safety issues with the approach I outlined above, or other trade-offs I might have missed?

[Sympatron]

I don't think there are safety issues if done right, but there might be performance implications.
I think you can effectively achieve the same by putting the producer in an Arc<Mutex>. The performance of this wouldn't be much worse than doing it in bbqueue directly since no write grant can be given out until a write is in progress anyway.
That way no one that does not need MPSC would have to pay the performance cost.

[jamesmunns]

Yeah, this is generally correct. I think we could be a bit more efficient than Arc<Mutex<Producer>>, but MPSC is a bit more tricky than SPSC.

• An SPSC configuration only needs to notify the Producer when the Consumer releases a grant, which triggers the "not empty" notification (the producer only waits when the buffer is full).
• An MPSC configuration needs to notify one/all waiting producers for ANY of the following:
  • The consumer releases a grant (in case the buffer was full)
  • A producer commits a grant (in case one producer is waiting for the other)

Right now, Notify currently has two async-relevant hooks:

• "not full" when a consumer releases a grant
• "not empty" when a producer commits a grant

I think these both could be reasonably be done with a single maitake-sync::WaitQueue (edit: one WaitQueue for the Producer side, one WaitCell for the Consumer side), which allows for multiple waiters. However, we would EITHER need to revise Notify to add a specific hook for this, or we could slightly misuse both "not full" and "not empty" hooks to pend the Producer.

There's also some discussion with a WaitQueue whether to do a "wake one" vs "wake all". "wake one" is more efficient: we don't have multiple producers race for the grant, where only one will win and the rest will have to re-queue. However with "wake one" we have to be careful about cancellation: if one producer is awoken, but cancels before it actually takes a grant, we need to then ensure that the NEXT waiting producer is also awoken so we don't observe a hang.

Thinking about what I said above, we also need to be careful about misusing the existing hooks if we want to support MPSC, as we need to cover the case where for example a producer takes a grant, but drops it without performing a commit at all. IIRC, this does an implicit commit(0), and we need to ensure this still pings the proper waker hooks.

Despite all of this, I still think there's merit to natively supporting MPSC: it means we don't have separate waker queues for producer and consumers. Practically maybe it's not that different, it does bother me on a conceptual level that it could be done better :)

[tommasoclini]

@sphw wrapping the producer in a maitake-sync::Mutex should be a reasonable solution, this wouldn't even touch bbqueue's code.