Celery-SQS

Sample Django application that shows how to use Celery with Amazon SQS as the Broker.

Quick guide

• Python 3.7+ is required
• Clone this repo
• Create a virtualenv
• Install requirements

python3 -m venv celery-sqs
source celery-sqs/bin/activate
pip install -r requirements.txt

• Add celerysqs/secret.py with the following template

  Note

  The SQS keys must have both IAM Policies listed below

KEY = "some secret key"  # for Django

SQS = {
    "access_key": "your aws_access_key",
    "secret_key": "your aws_secret_key",
}

Django Commands

• run the server

  $> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws python manage.py runserver

• run the celery worker

  $> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws celery worker -A celerysqs --concurrency=1 -l info

• queue some tasks

  $> ./scripts/curls.sh

IAM Policies

Note

Both policy types are required

(1) ListQueues (*)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

(2) CRUD (prefix)

Note

• Replace {region} with your AWS Region
• Replace {prefix} desired queue prefix

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:{region}:*:{prefix}*"
            ]
        }
    ]
}