[Django]-How can i split up a large django-celery tasks.py module into smaller chunks?

by

4👍

✅

This answer is about management commands rather than tasks in the sense of Celery background tasks.

I think the most straight-forward solution is to first write the tasks as Django management commands, and then call those from tasks.py. This way you can break up your tasks into one task per file, and have the additional benefit of being able to call your tasks arbitrarily from the command line.

So, in tasks.py:

from django.core import management

@task()
def my_task():
    management.call_command('my_task')
👤Tony

2👍

I’m not familiar with Django-Celery but quite familiar with Celery, so this is more of a general note which may answer your question…

Celery tasks are identified by name when executed. By default, tasks are given the name of their Python location, ie. '%s.%s' % (my_task.__module__, my_task.__name__), or ‘myapp.tasks.function_name’. That said, you can override the task name by supplying the @task decorator with the ‘name’ kwarg. So if you wanted to override a task name:

# We're in myapp.tasks but the task will not be named myapp.tasks.my_task as expected
@task(name='some.overridden.name')
def my_task():
    print 'Something'
👤Loren Abrams

2👍

The following works as well:

  • create a package tasks
  • spread your tasks in python modules inside the tasks package
  • import the modules in the tasks/__init__.py

Example:

django_app
| tasks
   | __init__.py
   | filesystem_tasks.py
| admin.py
| url.py

  • tasks/filesystem_tasks.py can contain this:

    from __future__ import absolute_import, unicode_literals
from celery import shared_task

@shared_task
def add(x, y):
    return x + y

  • tasks/__init__.py can contain this:

    from .filesystem_tasks import *

  • Running celery indicates the tasks as they would have been declared in one tasks.py at the same level as eg. url.py.

👤Raffi

Leave a comment