Pub/Sub: Idempotent subscribers without hand-rolling it

[violetta98]

Hello everyone!

I got tired of rewriting duplicate handling in every subscriber, so I packaged it. Pub/Sub is at-least-once and the docs say to handle duplicates without saying how.

from idemkit import QueueConfig, RedisBackend
from idemkit.contrib.pubsub import pubsub_consumer

consumer = pubsub_consumer(
    backend=RedisBackend.from_url("redis://localhost:6379"),
    config=QueueConfig(max_attempts=5),
)

@consumer.handle
def process(msg) -> None:
    fulfil(msg.data)

Dedup is on the Pub/Sub message_id. A duplicate arriving while the first is still running waits and gets the same result.

One thing worth knowing either way: if your callback takes longer than the ack deadline, Pub/Sub sends the message again while the first one is still running. That's where most "why did this happen twice" reports come from.

https://github.com/idemkit/idemkit, Apache-2.0

If it's useful, I'd appreciate a star. It's new, so visibility helps a lot right now.