monitor children and re-fork when they exit

[dgobaud]

Adding to wkonkel's pull request #565 - adding code to monitor children and re-fork them when they exit.

[albus522]

From experience trying to do this, I can say there are a bunch of edge cases this does not cover. Just use the DJ binary.

[dgobaud]

what is the dj binary/does it cover the issue in #565 ?

[albus522]

bin/delayed_job or script/delayed_job for older versions of rails

Sent from my iPhone

On Sep 19, 2014, at 5:42 PM, David Gobaud [email protected] wrote:

what is the dj binary/does it cover the issue in #565 ?

—
Reply to this email directly or view it on GitHub.

[dgobaud]

but those don't support NUM_PROCESSES for running multiple workers on a single dyno correct?

[albus522]

You are correct -n is not supported in such a way that heroku can support it.

The problem is that while your code will possibly work for your use case, it will not work in general. If we included this in the release, we would receive a bunch of issue reports of this not working in all situations.

In order to be included in the main release, the code needs to work for most cases, not just one.

[dgobaud]

Got it - I understand it doesn't make sense to include if it causes problems. What are the problems/bugs? I'd like to fix them if possible.

[albus522]

Lets clean this up and pull it in as a use at your own risk feature.

• Maintain worker name. Store the pids and detect which worker died. wait will return the pid and exit status of the dead process.
• Add logging for worker died
• Cleanup forked workers on exit
• Move the fork_delayed definition out into the namespace
• while doesn't need do

[coveralls]

Coverage decreased (-1.31%) when pulling a6c79c7 on dgobaud:master into 42b1489 on collectiveidea:master.

[dgobaud]

Sounds good. I've made all the changes except "Cleanup forked workers on exit" - I'm not sure how best to do that.

[joshco]

+! here for a solution to this for heroku. I tried a workaround using a shell script, but for some reason logging gets lost.
The logic is to start a bunch of daemons, then give the parent process something to do as in 'rake jobs:work' so you end up with some running as daemons and one running in the foreground to keep the dyno alive. However, the logging gets lost as STDOUT doesnt pass thru.

Here's my shell script (joshco_jobs)

#!/usr/bin/env bash

# dj spawner
bundle exec bin/delayed_job -n $DJ_DAEMONS start
bundle exec rake jobs:work

My procfile entry

jobber: bundle exec bin/joshco_jobs

[johnnyshields]

This PR can be closed in favor of #1138