Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize JobWrapper queueing ActiveJobs #35

Merged
merged 6 commits into from
Jan 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,27 @@ and this project aims to adhere to [Semantic Versioning](http://semver.org/spec/
### Removed <!-- for now removed features. -->
### Fixed <!-- for any bug fixes. -->

## [0.5.3] - 2024-01-12
### Changed

- Bring ActiveJob queue adapter's enqueuing behaviour inline with delayed_job &
upstream adapters, by stopping it serializing & deserializing the job instance
during the enqueue process. Optimises the enqueueing of jobs slightly too.

## [0.5.2] - 2023-10-19
### Fixed

- Fixed regression for Rails 7.1.1 not executing jobs, due to lazy loading change in Rails.
The `JobWrapper` needs to be loaded before `ActiveJob::Base` is loaded, but this wasn't
happening in the worker processes. Fix by extracting `JobWrapper` into it's own file and
loading when `Delayed` is loaded earlier in the process.

## [0.5.1] - 2023-10-11
### Changed

- Explicitly test Rails 7.1 and Ruby 3.1, Ruby 3.2 to verify compatability. Previous
gems were compatible, but this release is now verified to be compatible.
Comment on lines +21 to +33
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smudge I noticed the changelog wasn't up to date, so backfilled these entries based on the diffs/PR descriptions.


## [0.5.0] - 2023-01-20
### Changed
- Reduced handler size by excluding redundant 'job:' key (only 'job_data:' is
Expand Down
2 changes: 1 addition & 1 deletion delayed.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
spec.require_paths = ['lib']
spec.summary = 'a multi-threaded, SQL-driven ActiveJob backend used at Betterment to process millions of background jobs per day'

spec.version = '0.5.2'
spec.version = '0.5.3'
spec.metadata = {
'changelog_uri' => 'https://github.com/betterment/delayed/blob/main/CHANGELOG.md',
'bug_tracker_uri' => 'https://github.com/betterment/delayed/issues',
Expand Down
2 changes: 1 addition & 1 deletion lib/delayed/active_job_adapter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def _enqueue(job, opts = {})
opts.merge!({ queue: job.queue_name, priority: job.priority }.compact)
.merge!(job.provider_attributes || {})

Delayed::Job.enqueue(JobWrapper.new(job.serialize), opts).tap do |dj|
Delayed::Job.enqueue(JobWrapper.new(job), opts).tap do |dj|
job.provider_job_id = dj.id
end
end
Expand Down
12 changes: 10 additions & 2 deletions lib/delayed/job_wrapper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,16 @@ class JobWrapper # rubocop:disable Betterment/ActiveJobPerformable

delegate_missing_to :job

def initialize(job_data)
@job_data = job_data
def initialize(job_or_data)
# During enqueue the job instance is passed in directly, saves us deserializing
# it to find out how to queue the job.
# During load from the db, we get a data hash passed in so deserialize lazily.
if job_or_data.is_a?(ActiveJob::Base)
@job = job_or_data
@job_data = job_or_data.serialize
else
@job_data = job_or_data
end
end

def display_name
Expand Down
10 changes: 10 additions & 0 deletions spec/delayed/active_job_adapter_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,16 @@ def perform; end
ActiveJob::Base.queue_adapter = adapter_was
end

it "does not invoke #deserialize during enqueue" do # rubocop:disable RSpec/NoExpectationExample
JobClass.include(Module.new do
def deserialize(*)
raise "uh oh, deserialize called during enqueue!"
end
end)

JobClass.perform_later
end

it 'serializes a JobWrapper in the handler with expected fields' do
Timecop.freeze('2023-01-20T18:52:29Z') do
JobClass.perform_later
Expand Down
Loading