[Django]-How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

This is how I extended the answer by @smentek to allow for multiple worker instances and a single beat instance – same thing applies where you have to protect your leader. (I still don’t have an automated solution for that yet).

Please note that envvar updates to EB via the EB cli or the web interface are not relflected by celery beat or workers until app server restart has taken place. This caught me off guard once.

A single celery_configuration.sh file outputs two scripts for supervisord, note that celery-beat has autostart=false, otherwise you end up with many beats after an instance restart:

# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}

# create celery beat config script
celerybeatconf="[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A lexvoco --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# create celery worker config script
celeryworkerconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A lexvoco --loglevel=INFO

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999

environment=$celeryenv"

# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf

# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update

Then in container_commands we only restart beat on leader:

container_commands:
  # create the celery configuration file
  01_create_celery_beat_configuration_file:
    command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
  # restart celery beat if leader
  02_start_celery_beat:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat"
    leader_only: true
  # restart celery worker
  03_start_celery_worker:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker"

If someone is following smentek’s answer and getting the error:

05_celery_tasks_run: /usr/bin/env bash does not exist.

know that, if you are using Windows, your problem might be that the “celery_configuration.txt” file has WINDOWS EOL when it should have UNIX EOL. If using Notepad++, open the file and click on “Edit > EOL Conversion > Unix (LF)”. Save, redeploy, and error is no longer there.

Also, a couple of warnings for really-amateur people like me:

• Be sure to include “django_celery_beat” and “django_celery_results” in your “INSTALLED_APPS” in settings.py file.

• To check celery errors, connect to your instance with “eb ssh” and then “tail -n 40 /var/log/celery-worker.log” and “tail -n 40 /var/log/celery-beat.log” (where “40” refers to the number of lines you want to read from the file, starting from the end).

Hope this helps someone, it would’ve saved me some hours!

As stated in the answer accepted solution needs a lot of outside work other than coding. Now there is a nice library that handles this.
https://github.com/ybrs/single-beat

You install the library and create a redis server with elasticache.
And your procfile can be like this with environment variable targeting cache server.

    web: gunicorn --bind :8000 --workers 3 --threads 2 appname.wsgi:application
celery_beat: SINGLE_BEAT_REDIS_SERVER=$SINGLE_BEAT_REDIS single-beat celery -A proj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
celery_worker: celery -A proj worker -l INFO -P solo

After spending three days searching for a solution, I finally found a straightforward method. To set it up, create a new directory in your project’s main folder named .platform/hooks/postdeploy. Inside this directory, add a file without any extension called ‘postdeploy‘ and paste the provided script into it. Make sure to replace the placeholder with your actual project name.

With this setup, you can easily check the status of Celery by connecting to your instance via SSH and running the command ‘systemctl status celery.service‘. This approach simplifies the process of managing and monitoring Celery using the systemd service manager.

postdeploy

#!/usr/bin/env bash

echo "[Unit]
Name=Celery
Description=Celery service for My App
After=network.target
StartLimitInterval=0

[Service]
Type=simple
Restart=always
RestartSec=30
User=root
WorkingDirectory=/var/app/current
ExecStart=$PYTHONPATH/celery -A your_project_folder.celery worker --loglevel=INFO
ExecReload=$PYTHONPATH/celery -A your_project_folder.celery worker --loglevel=INFO
  EnvironmentFile=/opt/elasticbeanstalk/deployment/env

[Install]
WantedBy=multi-user.target
" | sudo tee /etc/systemd/system/celery.service

# Start celery service
sudo systemctl start celery.service

# Enable celery service to load on system start
sudo systemctl enable celery.service

Also do not forgrt to set up a config file inside .ebextensions. Content should be like:

any_name.config

{
  "commands": {
   
    "01_install_ctl": {
      "command": "sudo yum install -y /usr/bin/systemctl"
    }
  }
}