bullmq rate limiter usage

[RichardWright]

Dear bullmq team

I have three k8s pods which I want to rate limit performing an action. For performance(?) reasons, I would like the work to happen on the same pod it entered the queue but be restricted by the rate limiting of the shared queue.

An example - work added to the queue on pod one, will done by a worker on pod one, a job added to the queue on pod two will be handled by a worker on pod two. Is this possible with bullmq when using a rate limiter?

Thanks

[manast]

If you use the same queue for different pods then you cannot restrict that the jobs added "from" a given pod so that they are also processed in the same pod. This sounds a bit like an antipattern, as you are limiting yourself if you need to scale up in the future.

[manast]

Maybe the reason you want to do this is because you are using the local filesystem of the pod or something like that. I would advise to not do this if you can avoid it, as it will be more robust to use a distributed filesystem instead, or some storage that is accessible to all of the pods.

[RichardWright]

No file system involved, my first usagr for a rate limiter was limiting http requests to an upstream system. The response was immediately used so having the request and response traverse the network would be less fast/reliable than keeping it within the same system. I use this (https://www.npmjs.com/package/bottleneck) to do so.

My second rate limiter use was sending data via redis pubsub (limiter implemented here to protect consumers), no response needed for this one so a better fit than http. Due to the size of the data being sent (around 1 meg), again network traversal makes this seem like something which would be slower (the data leaving the box, picked up by another one, processed and sent along again).