Is there a way for multiple consumers?

[bennyKY]

Hi, Bull uses Redis pubsub so it wont allow multiple consumers.
But Bullmq uses Redis streams so it should be feasible.
Is there a way to do it with Bullmq? If not maybe someone can advise how to achieve it without Rabbit or Kafka?

[manast]

Bull and BullMQ both support multiple consumers, just create as many workers as you see fit and they will consume your jobs in parallell.

[bennyKY]

@manast
I think i wasnt clear, sorry for that. Im talking about broadcasting. Multiple services that consume from a queue. As far as i worked with Bull (1 year in production) if 2 services listening to messages from a queue only 1 services will receive the message.

[manast]

Ok, so you mean fanout (https://en.wikipedia.org/wiki/Fan-out_(software)), i.e. same job to many consumers. That is not supported and there are no plans for it, in that case using XTREAMS is a really good solution if you do not want to experiment with more complex solutions such as Kafka.

[bennyKY]

@manast thanks a lot for your answers, sorry didnt find anything related to XTREAMS, can you please give me a reference?

[manast]

@bennyKY I meant streams as you wrote on the issue, sorry for the confusion: https://redis.io/commands#stream

[audenaert]

@manast Came across this while searching for the same goal as the OP.

It looks like I could achieve a similar result by listening to queue events. Seems like a bit of an abuse of the intention of BullMQ, but wondering if something along the following lines would get a similar result.

const queue = new Queue('Cars');
await queue.add('paint', { color: 'red' });

const worker = new Worker('paint', paintHandler);
...
const queueEvents = new QueueEvents('Paint');
queueEvents.on('completed', queueListener1);
queueEvents.on('completed', queueListener2);
queueEvents.on('completed', queueListener3);

The downside here (I assume) is that I can't set these up to run in multiple workers because each listener would be called in each worker. That could lead to surprising behavior unless my listeners are idempotent. For example, if the app that's listening scales horizontally, each listener would be called on each running instance.

Am I understanding how Queue listeners work correctly?

[notflip]

@audenaert Did you ever find an answer to your question?

[manast]

The answer to the question is that there will be no surprising behaviour here, the job will be processed by one of the workers and all the global listeners (queueEvents instances), listening to the 'completed' event will receive the event.

[adamlamaa]

Had to eliminate BullMQ because it does not provide fanout, looking at RabbitMQ or LavinMQ instead.

[manast]

Can you elaborate what do you mean with "fanout" ?

[adamlamaa]

Did not see your detailed reply before my follow up question. Thanks for sharing your perspective. I will have a look at just using Redis on its own.

[manast]

Btw, you should probably use the BullMQ version instead of bull: https://docs.bullmq.io/guide/nestjs

[adamlamaa]

The architecture I am looking to implement is the ability to broadcast an event, which can then be picked up by multiple subscribers. Each subscriber can then evaluate on their own if they need to perform any actions as a result of the event.

Enabling a microservice setup, where logic can be split up and the code be structured in a more event driven way.

It was my understanding that in the queue world the pattern that fits this need is fanout.

And yes, was using the nestjs version, thanks!

[adamlamaa]

Update: I was not able to acheive the dream setup I had invisioned in my head without any MQ.

Ended up using BullMQ with a FlowProducer.

[Mostbet1giris]

Yes, BullMQ supports multiple consumers by leveraging Redis Streams. You can achieve this by creating multiple workers listening to the same queue. BullMQ ensures that each job is processed by only one worker, effectively distributing the workload among consumers.

Steps:

1. Create a queue with BullMQ.
2. Start multiple Worker instances pointing to the same queue.
3. Each worker will pick jobs from the queue and process them independently.

Example:

const { Worker } = require('bullmq');

const worker1 = new Worker('queueName', async job => {
  console.log(`Worker 1 processing job: ${job.id}`);
});

const worker2 = new Worker('queueName', async job => {
  console.log(`Worker 2 processing job: ${job.id}`);
});

This setup provides concurrent processing without RabbitMQ or Kafka. Just ensure your workers are stateless or handle shared resources correctly.

[ElonCas1no1giris]

BullMQ with Multiple Consumers: Yes, BullMQ supports multiple consumers using Redis Streams. Create multiple Worker instances for the same queue, and jobs will be distributed among them.

[ElonCas1no1giris]

Without Rabbit/Kafka: Use Redis Streams directly with libraries like redis in Rust or Node.js. Use XADD for adding jobs, XREADGROUP for processing with consumer groups, and XACK for acknowledgments. This achieves distributed processing without additional tools.

[Sanjayvarma27]

hey