[Django]-Python multiprocessing job to Celery task but AttributeError

by

4👍

This issue has a good answer in this other question

Basically, it is a known issue of Celery and a dirty hack is provided: it worked for me, I just added the following code in the same file where my tasks are defined:

from celery.signals import worker_process_init
from multiprocessing import current_process

@worker_process_init.connect
def fix_multiprocessing(**kwargs):
    try:
        current_process()._config
    except AttributeError:
        current_process()._config = {'semprefix': '/mp'}
👤Akhorus

2👍

I don’t know why multiprocessing doesn’t work, but I recommend you to use celery group task.

from celery import task, group

def feeds_fetch(feeds):
    g = group(fetch_one.s(feed) for feed in feeds)
    g.apply_async()


@task()
def fetch_one(feed):
    return feed.fetch()
👤youngrok

Leave a comment