How to deal with redis while developing applications using bullmq

[elhananjair]

Hello friends, I am developing a sveltekit-based web application and using Bullmq for some background processes. I have used patterns to repeat the process every day at a certain time.
While working on development, I stopped and ran the application multiple times after I changed something. But, I am having a problem, the job starts before the time I set and will run at the exact time too, is it because I am restarting the application multiple times?

1. How do you deal with this behavior while developing?
2. What will happen on production when the server/workstation that the application running on suddenly shuts down?

Thanks

[manast]

In general restarting the app will not induce a repeatable job to be processed. We will need more context to understand why you are getting this behaviour.

[manast]

If you are restarting the app and flushing the contents in Redis, then obviously yes, as it will start from scratch every time.

[elhananjair]

I am trying to send an SMS text and this is the scenario,

1. There is a function that sends SMS to some phone numbers fetching from the database
2. Rerun the above function but only to the numbers that didn't receive an SMS before, it is like a retrial. This function will repeatedly execute multiple times using setInterval() and get executed daily using BullMQ.

Here is the overview of the code I implemented:

Background process which sends SMS

  ## ../lib/server/sendsms.js 
  
  export function sendSMS(jobName) {
      if(jobName === 'send-to-all'){
           sending SMS ...
      }else if(jobName === 'retry'){
           Retrying sending to numbers that didn't receive 
      }
  
  }

Creating Queue

   ## hooks.server.js

   import { sendSMS } from '$lib/server/sendsms.js'
   import { Queue, Worker } from 'bullmq';
   
 const send_sms_queue = new Queue('send_sms_queue', {
    connection: {
        host: 'localhost',
        port: 6379
    }
});

const stop_sms_retrial = new Queue('stop_sms_retrial', {
    connection: {
        host: 'localhost',
        port: 6379
    }
});

async function addJobs() {
    await send_sms_queue.add('send_to_all', {}, {
        repeat: {
            pattern: '0 40 16 * * *',
        }
    });
    await send_sms_queue.add('sendSMS_to_uncharged', { interval: 5, delayInBetween: 20000 }, {
        repeat: {
            pattern: '0 41 16 * * *',
        }
    });
    await stop_sms_retrial.add('stop_interval', {}, {
        repeat: {
            pattern: '0 45 00 * * *',
        }
    })
}

await addJobs();

Worker to process the jobs

const worker = new Worker('send_sms_queue', async job => {
    if(job.name === 'send_to_all'){
        sendSMS('send_to_all') //Start the function with interval
        console.log(`Executed at ${new Date().toLocaleTimeString()}`)
    }else if(job.name === 'retry'){
        sendSMS_to_undelivered = setInterval(() => sendSMS('retry', job.data.interval), job.data.delayInBetween) //Start the function with interval
        console.log(`Executed at ${new Date().toLocaleTimeString()}`)
    }else {
        console.log("No job found!");
    }
}, {
    connection: {
        host: 'localhost',
        port: 6379
    }
});


const stopInterval_worker = new Worker('stop_sms_retrial', async job => {
    clearInterval(sendSMS_to_undelivered)
    console.log("This is a way to stop the setInterval function to retry sending sms!!!")
}, {
    connection: {
        host: 'localhost',
        port: 6379
    }
})

[elhananjair]

@manast I just wanted to make sure that the application I am developing won't messed up when executing some background code implemented with bullmq with repeatable property and redis in general.

[elhananjair]

If you are restarting the app and flushing the contents in Redis, then obviously yes, as it will start from scratch every time.

Actually, that's what I am looking for since I am testing my application, I need to remove previous schedules since I put the new pattern for the next text of my code, but currently on a rerun of the program, the background process will get executed before the scheduled time, I am suspecting that the previous queue won't be removed as rerun the program. But I am still new to how Bullmq and Redis communicate.

[elhananjair]

@manast any suggestions?

[roggervalf]

hi @elhananjair, looks like you need to clean your redis instance in that case, you can use flushdb command https://redis.io/commands/flushdb/ so you can start with an empty redis db before doing your testing process

[elhananjair]

hi @elhananjair, looks like you need to clean your redis instance in that case, you can use flushdb command https://redis.io/commands/flushdb/ so you can start with an empty redis db before doing your testing process

@roggervalf ok that is good info, I will do that. But just one last question on production does redis self clean it self or is there any mechanism to prevent redis to consume much RAM or go nuts and break my app.

[manast]

You have to setup some monitors to avoid running out of RAM. Usually you set up things like "removeOnCompleted/Failed" to avoid consuming too much RAM.

[elhananjair]

Let's say the system runs 24/7, and if one day the server where the application runs is down and restarted after a while, doesn't this cause chaos in scheduled jobs? let's say the time has passed for the scheduled job, how that will be handled?

The reason I am raising this question is, that before flushing db of Redis for testing purposes, I see jobs running out of nowhere regardless of the specific time scheduled.

[manast]

BullMQ will not catchup for jobs that have missed their scheduled window. There is only 1 next scheduled job that is in the delayed status waiting for execution. So if there are no workers running (or Redis is down) nothing will happen until that job is executed and then jobs continue to be scheduled as before the downtime.

[elhananjair]

BullMQ will not catchup for jobs that have missed their scheduled window. There is only 1 next scheduled job that is in the delayed status waiting for execution. So if there are no workers running (or Redis is down) nothing will happen until that job is executed and then jobs continue to be scheduled as before the downtime.

Yeah but while I was locally testing my application with bullmq, if the job didn't exected based on the schedule window, on the next restart of the application, all unexecuted jobs will run even before their scheduled time. Therefore I wanted to this behaviour in my application. Like clearing Redis cache at each start of my application.

This is the way I understand how bullmq schedule works with Redis, when my application starts and set the scheduled time, that value will be saved on Redis as key and value. In case, my application goes down before the scheduled time for the jobs and starts up after the scheduled time, I see the jobs running instantly and the next repeated job will be run on the scheduled time.

I do not understand the relationship between Bullmq and Redis. How to handle the repeatable jobs.

Thank you

[manast]

If you want to clean redis just issue a flushall command (https://redis.io/commands/flushall/). In production you will for sure not want to clean redis everytime a server is restarted...

[elhananjair]

Thank you @manast I finally deployed my Sveltekit application and it is working and the job is being processed at the scheduled time, I have some doubts about the way the job is being executed but I will start a new discussion for that.