[BUG] Cron not working

[ryck]

Describe the bug
The cron job doesn't not work (anymore)
The collector / webapp combo still works if I trigger it manually, but for some reason the cron job doesn't work anymore (it was working before)
I haven't changed my setup (apart from update the images) and I can't see anything wrong on the logs....
Happy to provide you with logs or anything that you need...

Expected behavior
The webapp should update daily with new metrics

Current behavior
The webapp does not get updated unless you run scrutiny-collector-metrics run in each collector

[AnalogJ]

Hey @ryck
Can you confirm that you're using my image analogj/scrutiny (with the default latest tag)?

[ryck]

Hey @ryck
Can you confirm that you're using my image analogj/scrutiny (with the default latest tag)?

Yes, I am using analogj/scrutiny:latest everywhere (2 collectors + one "all-in-one")

[Zorlin]

I actually have the same issue with a manual install (from my playbook of course :))

I believe it's not properly getting stuff out of /etc/profile and that's why it's failing.

Seeing the cron issue on Ubuntu 18.04 + Armbian 20.04 Focal.

[joemiller]

I'm using latest (sha256:bc4caebb56561adf231d9986d290efd983206c42505fe31a5d538b6331c0c4af) as well and running into this. I'm a new user so I don't know what it is supposed to look like when things are working. I've tried running the same command that s6 is using to start the jobberrunner proc but there is no output at all and so it is difficult for me to debug. I've never used jobber either. I'll keep trying to see if I can figure it out.

[AnalogJ]

Sorry about the delay, I'm looking into this now. I have confirmed that I can reproduce this issue locally.

[Klem7ens]

I could suggest little workaround. Crontab (I mean edit crontab -e and add job) failed for me too, but I found that adding a job by creating file in /etc/cron.daily works (Debian & Ubuntu).
I've created a file /etc/cron.daily/scrutiny with content (replace X.X.X.X with name or IP of your WEB container):

#!/bin/sh

COLLECTOR="/opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64"
CONF="/opt/scrutiny/bin/collector.yaml"
WEB="X.X.X.X"

"$COLLECTOR" run --api-endpoint "http://$WEB:8080" --config "$CONF"

[AnalogJ]

So I added some logging to the jobber scheduler, and to be honest, I have no idea why this isn't working correctly. Initially, I thought this could be related to a mis-configured cron string since my collector still runs occasionally, but I realized the number of days between each run seems to be random. @Zorlin I'm curious about your /etc/profile comment. What do you think is misconfigured such that the collector only works intermittently?

I'm considering using real cron rather than jobber inside the container, but it seems to require some hacky workarounds to get working. :(
This might take a while to track down. I'm going to see if I can come up with a workaround in the meantime.

[joemiller]

@AnalogJ You could have a simple script with a sleep loop that is supervised by s6, something like:

#!/bin/bash
while true; do
  /scrutiny/bin/scrutiny-collector-metrics run
  sleep 86400
done

similar to the current jobber config of 'run at midnight', although this wouldn't be at midnight it would just be daily from the time the container started.

Otherwise if cron semantis are desired I suspect there are wealth of lightweight 'cron for docker' tools out there now. This one looks interesting https://github.com/webdevops/go-crond

[AnalogJ]

Hi everyone.
I recently updated both the collector and all-in-one container to use Cron, rather than jobber.
As before, the collector is scheduled to run daily, at midnight. The images now read their cron config from /etc/cron.d/scrutiny, with a default file that looks like the following:

# correctly route collector logs (STDOUT & STDERR) to Cron foreground (collectable by Docker STDOUT)
# cron schedule to run daily at midnight:  '0 0 * * *'
# System environmental variables are stripped by cron, source our dump of the docker environmental variables before each command (/env.sh)
0 0 * * * root . /env.sh; /scrutiny/bin/scrutiny-collector-metrics run >/proc/1/fd/1 2>/proc/1/fd/2

If you would like to change the collector frequency, you can mount your own crontab compatible file, but please include the . /env.sh; prefix when running the collector (it's required to pass docker env vars to the process).

Sorry for the long delay fixing this issue, I had a bunch of personal/professional stuff to take care of.

[Zorlin]

Relieved to see this project alive. Bravo! I'll look at updating things some time soon.

…

On Sun, 25 Apr 2021, 04:46 Jason Kulatunga, ***@***.***> wrote: Hi everyone. I recently updated both the collector and all-in-one container to use Cron, rather than jobber. As before, the collector is scheduled to run daily, at midnight. The images now read their cron config from /etc/cron.d/scrutiny, with a default file that looks like the following: # correctly route collector logs (STDOUT & STDERR) to Cron foreground (collectable by Docker STDOUT) # cron schedule to run daily at midnight: '0 0 * * *' # System environmental variables are stripped by cron, source our dump of the docker environmental variables before each command (/env.sh) 0 0 * * * root . /env.sh; /scrutiny/bin/scrutiny-collector-metrics run >/proc/1/fd/1 2>/proc/1/fd/2 If you would like to change the collector frequency, you can mount your own crontab compatible file, but please include the . /env.sh; prefix when running the collector (it's required to pass docker env vars to the process). Sorry for the long delay fixing this issue, I had a bunch of personal/professional stuff to take care of. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKONLFB53TBX35DWUBCOSLTKMU3RANCNFSM4T7L4SIQ> .

[Panzer1119]

The cron job still/again doesn't seem to work.

I narrowed the problem down to the /env.sh file in the collector container (not using the omnibus image).

The cron interval is exported without quotes export COLLECTOR_CRON_SCHEDULE=*/1 * * * *.

Calling the (cron job) commands . /env.sh; /opt/scrutiny/bin/scrutiny-collector-metrics run >/proc/1/fd/1 2>/proc/1/fd/2 manually outputs this error: bash: export: `*': not a valid identifier.

It's easy to fix by changing the export to export COLLECTOR_CRON_SCHEDULE="*/1 * * * *", but this needs to be changed in the code or something like that, because the cron interval */1 * * * * needs to be in the cron file without the quotes.

[AnalogJ]

Hey @Panzer1119 any chance you have a couple of minutes to chat about this over Discord? I cant seem to replicate this issue when I set the COLLECTOR_CRON_SCHEDULE environmental variable in the docker compose file:

    environment:
      - "COLLECTOR_CRON_SCHEDULE=*/1 * * * *"

[AnalogJ]

ahh I think I found the issue.

COLLECTOR_CRON_SCHEDULE="* * * * *"
vs
"COLLECTOR_CRON_SCHEDULE=* * * * *"

The first command generates the following scrutiny crontab, which I'm guessing is what you're seeing

"* * * * *" root . /env.sh; /opt/scrutiny/bin/scrutiny-collector-metrics run >/proc/1/fd/1 2>/proc/1/fd/2

[Panzer1119]

I think the problem is that the crontab and /env.sh file want values in a different style.

Cron wants the schedule not quoted */1 * * * *, but the /env.sh wants the schedule in quotes "*/1 * * * *".

[AnalogJ]

Yeah, thanks for the pointer @Panzer1119 I made a couple of changes, but the big two are:

eeb086c
a004f85

Basically the env variables are always quoted, and then I strip the quotes when templating the crontab file.
Long term I want to move away from the hacky sed templating to confd/etc (which will allow changes to the cron schedule container start, rather than just once), but this will work for now.

I'm waiting for a docker build to complete, but I've tested this locally and its working.

[AnalogJ]

closing since I have verification that this has been fixed in v0.4.4.
If anyone is still running into an issue with Cron, please reopen this issue.

[MattKobayashi]

Hi @AnalogJ, I'm not able to get cron to work in the master-collector Docker image. The default cron task doesn't seem to trigger, nor does it after overriding with COLLECTOR_CRON_SCHEDULE. I've confirmed that I'm running the latest Docker image (v0.4.4). It seems that the cron task is never even triggered, although I'm not able to figure out why. Here's the output of docker compose logs scrutiny-collector with COLLECTOR_CRON_SCHEDULE=*/1 * * * * after a few minutes of running:

rancher@server:~$ docker compose logs scrutiny-collector
scrutiny-collector  | starting cron
rancher@server:~$

Running the commands manually within the container gathers the SMART data and updates the web interface as expected, so it doesn't appear to be anything wrong with the configuration of either container. I'd appreciate any assistance you could offer.

[AnalogJ]

@MattKobayashi ugh. Sorry about that. I spent a ton of time figuring out the root cause and fixing it in the omnibus container, but it seems I forgot to make the fix in the collector image as well.
I'll do some testing and release v0.4.5 with a fix

[AnalogJ]

alright, confirmed working

  collector:
    image: 'ghcr.io/analogj/scrutiny:master-collector'
    cap_add:
      - SYS_RAWIO
    volumes:
      - '/run/udev:/run/udev:ro'
    environment:
      COLLECTOR_API_ENDPOINT: 'http://web:8080'
      COLLECTOR_CRON_SCHEDULE: '*/1 * * * *'
    depends_on:
      web:
        condition: service_healthy
    devices:
      - "/dev/sda"
      - "/dev/sdb"

I updated the https://github.com/AnalogJ/scrutiny/blob/master/docker/example.hubspoke.docker-compose.yml example compose file, and the collector correctly runs every minute.

Creating a release now @MattKobayashi

[MattKobayashi]

You're a legend @AnalogJ, thank you!

[AnalogJ]

I'm guessing that fixed the issue @MattKobayashi :D

[MattKobayashi]

It certainly did @AnalogJ, thank you again 😄

[c0ldtech]

Does changing the chron schedule currently work? After reading through this issue, I put

environment:
      - TZ=Europe/Berlin
      - COLLECTOR_CRON_SCHEDULE = '0 * * * *'

into my docker-compose (I'm using the omnibus docker image), but inside of the container in

/etc/cron.d/scrutiny

it always says

0 0 * * * root . /env.sh; /opt/scrutiny/bin/scrutiny-collector-metrics run >/proc/1/fd/1 2>/proc/1/fd/2

no matter what I actually put in the env. Looking at the web ui, it even seems as though a refresh never actually happens, running the collectur manually works as expected though.

Edit: just now saw issue #602 - so I guess there's a bug