[Django]-Docker&Celery – ERROR: Pidfile (celerybeat.pid) already exists

I believe there is a pidfile in your project directory ./ then when you run the container, it’s mounted in.
(therefore RUN find . -type f -name "celerybeat.pid" -exec rm -f {} \; had no effect).

You can use celery --pidfile=/opt/celeryd.pid to specify a non mounted path so that it is not mirror on the host.

Although not professional in the slightest, I found adding:

celerybeat.pid

to my .dockerignore file was what fixed said issues.

I had this error with Airflow when I run it with docker-compose.

If you don’t care about the current status of your Airflow, you can just delete airflow containers.

docker rm containerId

And after that, start the Airflow again:

docker-compose up

Had the same issue as part of an Airflow setup (apache-airflow==2.3.4, celery==5.2.7), on Docker Compose:

ERROR: Pidfile (/airflow/airflow-worker.pid) already exists.
Seems we're already running? (pid: 1)

I tried to pass --pidfile (actually --pid under the Airflow umbrella) like so:

airflow celery worker --pid=

However this didn’t work, and a .pid file was still being created. Maybe this is due to the additional Airflow layer.

Eventually, I figured that the original issue had to do with the Docker Compose restart policy (in my case, restart: always). Once the worker had failed once, subsequent restarts would find the already existing .pid file. This is because containers keep state on restart (see this or this).

A more permanent solution was to use a tmpfs, and point the .pid file there:

# docker-compose.yml

worker:
    image: {...}
    tmpfs:
      - /airflow-worker
    entrypoint: airflow celery worker --pid=/airflow-worker/airflow-worker.pid
    ...
...

Other way, create a django command celery_kill.py

import shlex
import subprocess

from django.core.management.base import BaseCommand


class Command(BaseCommand):
    def handle(self, *args, **options):
        kill_worker_cmd = 'pkill -9 celery'
        subprocess.call(shlex.split(kill_worker_cmd))

docker-compose.yml :

  celery:
    build: ./src
    restart: always
    command: celery -A project worker -l info
    volumes:
      - ./src:/var/lib/celery/data/
    depends_on:
      - db
      - redis
      - app

  celery-beat:
    build: ./src
    restart: always
    command: celery -A project beat -l info --pidfile=/tmp/celeryd.pid
    volumes:
      - ./src:/var/lib/beat/data/
    depends_on:
      - db
      - redis
      - app

and Makefile:

run:
    docker-compose up -d --force-recreate
    docker-compose exec app python manage.py celery_kill
    docker-compose restart
    docker-compose exec app python manage.py migrate

The reason of this error is docker container stopped without normal Celery stopping process.
The solution is simple. stop Celery before start.

Solution 1. write celery start command(ex> docker-entrypoint.sh, …) as follow

celery multi stopwait w1 -A myproject
&& rm -f /var/run/celery/w1.pid  # remove stale pidfile
&& celery multi start w1 -A myproject-l info --pidfile=/var/run/celery/w1.pid 

Solution 2. (not recommended)

always run "docker-compose down" before "docker-compose up".