From fce248d8480e406b967927a11a77823b343f73a3 Mon Sep 17 00:00:00 2001 From: Johnny Shields Date: Sat, 3 Apr 2021 20:50:17 +0900 Subject: [PATCH] Cleanup and clarify README --- README.md | 260 +++++++++++++++++++++++++++++------------------------- 1 file changed, 138 insertions(+), 122 deletions(-) diff --git a/README.md b/README.md index 29ad10782..ac6df8b5b 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ you're reading the documentation for the master branch. [View documentation for the latest release (4.1.9).](https://github.com/collectiveidea/delayed_job/tree/v4.1.9)** -Delayed::Job -============ +# Delayed Job + [![Gem Version](https://badge.fury.io/rb/delayed_job.svg)][gem] ![CI](https://github.com/collectiveidea/delayed_job/workflows/CI/badge.svg) [![Code Climate](https://codeclimate.com/github/collectiveidea/delayed_job.svg)][codeclimate] @@ -14,87 +14,74 @@ Delayed::Job [codeclimate]: https://codeclimate.com/github/collectiveidea/delayed_job [coveralls]: https://coveralls.io/r/collectiveidea/delayed_job -Delayed::Job (or DJ) encapsulates the common pattern of asynchronously executing -longer tasks in the background. +Delayed Job encapsulates the common pattern of asynchronously executing +longer tasks in the background. Examples of such tasks include: + +* Sending emails +* Image resizing +* HTTP downloads +* Updating smart collections +* Updating Solr, our search server, after product changes +* Batch imports +* Spam checks -It is a direct extraction from Shopify where the job table is responsible for a -multitude of core tasks. Amongst those tasks are: +Delayed Job was extracted from Shopify. -* sending massive newsletters -* image resizing -* http downloads -* updating smart collections -* updating solr, our search server, after product changes -* batch imports -* spam checks +## Installation -[Follow us on Twitter][twitter] to get updates and notices about new releases. +### Version Support -[twitter]: https://twitter.com/delayedjob +Delayed Job 4.x only supports Rails 3.0+. -Installation -============ -delayed_job 3.0.0 only supports Rails 3.0+. +### Configuring your Database -delayed_job supports multiple backends for storing the job queue. [See the wiki +Delayed Job supports multiple backends for storing the job queue. [See the wiki for other backends](https://github.com/collectiveidea/delayed_job/wiki/Backends). -If you plan to use delayed_job with Active Record, add `delayed_job_active_record` to your `Gemfile`. +To use with Active Record, add `delayed_job_active_record` to your `Gemfile`. ```ruby gem 'delayed_job_active_record' ``` -If you plan to use delayed_job with Mongoid, add `delayed_job_mongoid` to your `Gemfile`. +To use with Mongoid, add `delayed_job_mongoid` to your `Gemfile`. ```ruby gem 'delayed_job_mongoid' ``` -Run `bundle install` to install the backend and delayed_job gems. +Run `bundle install` to install the backend and `delayed_job` gems. The Active Record backend requires a jobs table. You can create that table by running the following command: - rails generate delayed_job:active_record - rake db:migrate - -For Rails 4.2+, see [below](#active-job) +``` +rails generate delayed_job:active_record +rake db:migrate +``` -Development -=========== -In development mode, if you are using Rails 3.1+, your application code will automatically reload every 100 jobs or when the queue finishes. -You no longer need to restart Delayed Job every time you update your code in development. +### Active Job -Active Job -========== -In Rails 4.2+, set the queue_adapter in config/application.rb +To use Delayed Job with Active Job (Rails 4.2+), set the `queue_adapter` in `config/application.rb`: ```ruby config.active_job.queue_adapter = :delayed_job ``` -See the [rails guide](http://guides.rubyonrails.org/active_job_basics.html#setting-the-backend) for more details. +See the [Rails Guide](http://guides.rubyonrails.org/active_job_basics.html#setting-the-backend) for more details. -Rails 4.x -========= -If you are using the protected_attributes gem, it must appear before delayed_job in your gemfile. If your jobs are failing with: +### Protected Attributes - ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR: null value in column "handler" violates not-null constraint +When using the `protected_attributes` gem, it must appear before `delayed_job` in your `Gemfile`. Otherwise you will see this error: -then this is the fix you're looking for. - -Upgrading from 2.x to 3.0.0 on Active Record -============================================ -Delayed Job 3.0.0 introduces a new column to the delayed_jobs table. +``` +ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR: null value in column "handler" violates not-null constraint +``` -If you're upgrading from Delayed Job 2.x, run the upgrade generator to create a migration to add the column. +## Using Delayed Job in your Application - rails generate delayed_job:upgrade - rake db:migrate +### Queuing Jobs -Queuing Jobs -============ Call `.delay.method(params)` on any object and it will be processed in the background. ```ruby @@ -120,7 +107,7 @@ device = Device.new device.deliver ``` -## Parameters +### Parameters `#handle_asynchronously` and `#delay` take these parameters: @@ -165,10 +152,11 @@ class LongTasks end ``` -If you ever want to call a `handle_asynchronously`'d method without Delayed Job, for instance while debugging something at the console, just add `_without_delay` to the method name. For instance, if your original method was `foo`, then call `foo_without_delay`. +To call a `handle_asynchronously`'d method without Delayed Job, add `_without_delay` to the method name. +For instance, if your original method was `foo`, then call `foo_without_delay`. + +### Rails Mailers -Rails Mailers -============= Delayed Job uses special syntax for Rails Mailers. Do not call the `.deliver` method when using `.delay`. @@ -190,107 +178,134 @@ You may also wish to consider using [Active Job with Action Mailer](https://edgeguides.rubyonrails.org/active_job_basics.html#action-mailer) which provides convenient `.deliver_later` syntax that forwards to Delayed Job under-the-hood. -Named Queues -============ -DJ 3 introduces Resque-style named queues while still retaining DJ-style -priority. The goal is to provide a system for grouping tasks to be worked by -separate pools of workers, which may be scaled and controlled individually. +### Queues -Jobs can be assigned to a queue by setting the `queue` option: +Delayed Job supports assigning jobs to named queues. Each queue may be worked by a +separate pool of workers, which may then be scaled and controlled individually. + +Jobs can be assigned to a queue by setting the `:queue` option: ```ruby -object.delay(:queue => 'tracking').method +object.delay(queue: 'tracking').method -Delayed::Job.enqueue job, :queue => 'tracking' +Delayed::Job.enqueue job, queue: 'tracking' -handle_asynchronously :tweet_later, :queue => 'tweets' +handle_asynchronously :tweet_later, queue: 'tweets' ``` -You can configure default priorities for named queues: +You may configure a default priority for each queue (lower number = higher priority): ```ruby Delayed::Worker.queue_attributes = { - high_priority: { priority: -10 }, - low_priority: { priority: 10 } + high_priority_queue: { priority: -10 }, + low_priority_queue: { priority: 10 } } ``` -Configured queue priorities can be overriden by passing priority to the delay method +To override the queue's default priority, pass `:priority` to the delay method: ```ruby -object.delay(:queue => 'high_priority', priority: 0).method +object.delay(queue: 'high_priority_queue', priority: 0).method ``` You can start processes to only work certain queues with the `queue` and `queues` -options defined below. Processes started without specifying a queue will run jobs -from **any** queue. To effectively have a process that runs jobs where a queue is not -specified, set a default queue name with `Delayed::Worker.default_queue_name` and -have the processes run that queue. +options (refer to "Running Jobs" section below.) Processes started without specifying +a queue will run jobs from **any** queue. To effectively have a process that runs +jobs where a queue is not specified, set a default queue name with +`Delayed::Worker.default_queue_name` and have the processes run that queue. + +## Running Jobs + +### Running as a Daemon Process -Running Jobs -============ `script/delayed_job` can be used to manage a background process which will start working off jobs. -To do so, add `gem "daemons"` to your `Gemfile` and make sure you've run `rails -generate delayed_job`. +To do so, add `gem "daemons"` to your `Gemfile` and run `rails generate delayed_job`. You can then do the following: - RAILS_ENV=production script/delayed_job start - RAILS_ENV=production script/delayed_job stop +``` +# Run a single worker process as a daemon +RAILS_ENV=production script/delayed_job start - # Runs two workers in separate processes. - RAILS_ENV=production script/delayed_job -n 2 start - RAILS_ENV=production script/delayed_job stop +# Run 4 workers in separate processes +RAILS_ENV=production script/delayed_job -n4 start +``` - # Set the --queue or --queues option to work from a particular queue. - RAILS_ENV=production script/delayed_job --queue=tracking start - RAILS_ENV=production script/delayed_job --queues=mailers,tasks start +These commands may be run on any server so long as it has access to your database. +Each worker will check the database at least every 5 seconds. - # Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues. - # The following command will start 1 worker for the tracking queue, - # 2 workers for the mailers and tasks queues, and 2 workers for any jobs: - RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start +### Stopping and Restarting - # Runs all available jobs and then exits - RAILS_ENV=production script/delayed_job start --exit-on-complete - # or to run in the foreground - RAILS_ENV=production script/delayed_job run --exit-on-complete +``` +# Stop all workers and exit +RAILS_ENV=production script/delayed_job stop -**Rails 4:** *replace script/delayed_job with bin/delayed_job* +# Stop all workers and start a single new worker process +RAILS_ENV=production script/delayed_job restart -Workers can be running on any computer, as long as they have access to the -database and their clock is in sync. Keep in mind that each worker will check -the database at least every 5 seconds. +# Stop all workers and start 4 new worker processes +RAILS_ENV=production script/delayed_job -n4 restart +``` -You can also invoke `rake jobs:work` which will start working off jobs. You can -cancel the rake task with `CTRL-C`. +You may also send `SIGTERM` to stop Delayed Job. -If you want to just run all available jobs and exit you can use `rake jobs:workoff` +You must pass the same arguments to `restart` that you used when calling `start`. -Work off queues by setting the `QUEUE` or `QUEUES` environment variable. +### Worker Queues and Pools - QUEUE=tracking rake jobs:work - QUEUES=mailers,tasks rake jobs:work +``` +# Set the --queues option to work from a particular queue +RAILS_ENV=production script/delayed_job --queues=tracking start +RAILS_ENV=production script/delayed_job --queues=mailers,tasks start + +# Use the --pool option to specify a worker pool. You can use this option multiple +# times to start different numbers of workers for different queues. +# The following command will start 1 worker for the tracking queue, +# 2 workers for the mailers and tasks queues, and 2 workers for any jobs: +RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start +``` + +### Exit On Complete mode + +``` +# Run as a daemon and exit after working all available jobs +RAILS_ENV=production script/delayed_job start --exit-on-complete + +# or to run in the foreground +RAILS_ENV=production script/delayed_job run --exit-on-complete +``` + +### Running via Rake + +**IMPORTANT:** For production daemons, use `script/delayed_job` (see above) instead of Rake. -Restarting delayed_job -====================== +As a convenience, you may start a worker process using `rake jobs:work`. +You can exit the rake task with `CTRL-C`. -The following syntax will restart delayed jobs: +To run all available jobs and exit, use `rake jobs:workoff`. - RAILS_ENV=production script/delayed_job restart +Set the `QUEUES` environment variable to work specific queues. -To restart multiple delayed_job workers: +``` +# Start a worker listening to the "tracking" queue +QUEUES=tracking rake jobs:work + +# Work all jobs from the "mailers" and "tasks" queues, then exit +QUEUES=mailers,tasks rake jobs:workoff +``` - RAILS_ENV=production script/delayed_job -n2 restart +### Development -**Rails 4:** *replace script/delayed_job with bin/delayed_job* +In `development` environment, Delayed Job will automatically reload +your application code after each 100 jobs or when the queue finishes. +You should not need to restart Delayed Job each time you update your code. +## Advanced Topics +### Custom Jobs -Custom Jobs -=========== Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner. ```ruby @@ -375,12 +390,13 @@ NewsletterJob = Struct.new(:text, :emails) do end ``` -Hooks -===== -You can define hooks on your job that will be called at different stages in the process: +### Hooks +You can define hooks on your job that will be called at different stages in the process: -**NOTE:** If you are using ActiveJob these hooks are **not** available to your jobs. You will need to use ActiveJob's callbacks. You can find details here https://guides.rubyonrails.org/active_job_basics.html#callbacks +**NOTE:** If you are using ActiveJob these hooks are **not** available to your jobs. +You will need to use ActiveJob's callbacks. +See the [Rails Guides](https://guides.rubyonrails.org/active_job_basics.html#callbacks) for details. ```ruby class ParanoidNewsletterJob < NewsletterJob @@ -414,9 +430,9 @@ class ParanoidNewsletterJob < NewsletterJob end ``` -Gory Details -============ -The library revolves around a delayed_jobs table which looks as follows: +### Gory Details + +The library revolves around a `delayed_jobs` table which looks as follows: ```ruby create_table :delayed_jobs, :force => true do |table| @@ -435,10 +451,10 @@ end On error, the job is scheduled again in 5 seconds + N ** 4, where N is the number of attempts or using the job's defined `reschedule_at` method. -The default `Worker.max_attempts` is 25. After this, the job is either deleted (default), or left in the database with "failed_at" set. +The default `Delayed::Worker.max_attempts` is 25. After this, the job is either deleted (default), or left in the database with "failed_at" set. With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours. -The default `Worker.max_run_time` is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to +The default `Delayed::Worker.max_run_time` is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take. By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set @@ -479,12 +495,12 @@ Delayed::Worker.raise_signal_exceptions = :term Delayed::Worker.logger = Logger.new(File.join(Rails.root, 'log', 'delayed_job.log')) ``` -Cleaning up -=========== +### Cleaning up + You can invoke `rake jobs:clear` to delete all jobs in the queue. -Having problems? -================ +### Having problems? + Good places to get help are: * [Google Groups](http://groups.google.com/group/delayed_job) where you can join our mailing list. * [StackOverflow](http://stackoverflow.com/questions/tagged/delayed-job)