[Django]-Celery what happen to running tasks when using app.control.purge()?

Yes, worker will continue execution of currently running task unless worker is also restarted.

Also, The Celery Way is to always expect tasks to be run in concurrent environment with following considerations:

• there are many tasks running concurrently
• there are many celery workers executing tasks
• same task may run again
• multiple instances of the same task may run at the same moment
• any task may be terminated any time

even if you are sure that in your environment there is only one worker started / stopped manually and these do not apply – tasks should be created in such way to allow everything of this to happen.

Some useful techniques:

• use database transactions
• use locking
• split long-running tasks into faster ones
• if task has intermediate values to be saved or they are important (i.e. non-reproducible like some api calls) and their processing in next step takes time – consider splitting into several chained tasks

If you need to run only one instance of a task at a time – use some sort of locking – create / update lock-record in the database or in the cache so others (same tasks) can check and know this task is running and just return or wait for previous one to complete.

I.e. recursion_function can also be Periodic Task. Being periodic task will make sure it is run every interval, even if previous one fails for any reason (and thus fails to queue itself again as in regular non-periodic task). With locking you can make sure only one is running at a time.

check_loop():

First, it is recommended to save results in one transaction in the database to make sure all or nothing is saved / modified in the database.

You can also save some marker that indicates how many / status of saved objects, so future tasks can just check this marker, not each object.

Or somehow perform check for each element before creating it that it already exists in the database.