[Django]-How to configure and run celery worker on remote system

Here is the gist of the idea:

On Machine A:

1. Install Celery & RabbitMQ.
2. Configure rabbitmq so that Machine B can connect to it.
3. Create my_tasks.py with some tasks and put some tasks in queue.

On Machine B:

1. Install Celery.
2. Copy my_tasks.py file from machine A to this machine.
3. Run a worker to consume the tasks

I had the same requirement and experimented with celery. It is a lot easier to do that. I wrote a detailed blog post on that few days back. Check out how to send tasks to remote machine?

You can make use of app.send_task() with something like the following in your django project:

from celery import Celery
import my_client_config_module

app = Celery()
app.config_from_object(my_client_config_module)

app.send_task('dotted.path.to.function.on.remote.server.relative.to.worker',
              args=(1, 2))

First, think about how celery really work?

Celery producer adds a task to queue with name and other important headers to identify the location of your task.

Celery does not add a complete executable function to MQ.

So, When you look at worker(consumer) side.

Celery gets task details from MQ and tries to run this.
To run this task there should be available module/files/environment/codebase to execute this task.

Now lets come to your question …

You try to set worker on a separate machine so logically to execute a function pointed by the task you need complete code environment of tasks and you should connect(Otherwise how you gonna get tasks from MQ ?) with your MQ where tasks live.

basically I will take ChillarAnand’s answer. I would like to add comment on his answer, but I can’t cause I don’t have 50 reputation.

so…

the answer to your question…

First you would like to read “how to send tasks to remote machine?”, as
ChillarAnand mentioned.

That is really good article, with one small flaw, such as “does not have ‘@app.task’ on the function def add(), in the content remote.py”, it caused problem and confused me as a newbie to celery.

And the answer to “[Errno 113] No route to host.” part,

I guess… I guess you have a firewall running in your rabbitmq server,
you might want to have a check. Most of time, it is iptables, but it could something else. Switch it off, or change the rules. Then you can give it another try.