[Django]-Celery creating a new connection for each task

by

8👍

I ran into the same problem on Heroku with CloudAMQP. I do not know why, but I had no luck when assigning low integers to the BROKER_POOL_LIMIT setting.

Ultimately, I found that by setting BROKER_POOL_LIMIT=None or BROKER_POOL_LIMIT=0 my issue was mitigated. According to the Celery docs, this disables the connection pool. So far, this has not been a noticeable issue for me, however I’m not sure if it might be for you.

Link to relevant info: http://celery.readthedocs.org/en/latest/configuration.html#broker-pool-limit

👤jnishiyama

7👍

I wish I was using Redis, because there is a specific option to limit the number of connections: CELERY_REDIS_MAX_CONNECTIONS.

The MongoDB has a similar backend setting.

Given these backend settings, I have no idea what BROKER_POOL_LIMIT actually does. Hopefully CELERY_REDIS_MAX_CONNECTIONS solves your problem.

I’m one of those folks using CloudAMQP, and the AMQP backend does not have its own connection limit parameter.

👤Bryan

0👍

Try those settings :

CELERY_IGNORE_RESULT = True
CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True
👤Stan

-1👍

I had a similar issue involving number of connections and Celery. It wasn’t on Heroku, and it was Mongo and not Redis though.

I initiated the connection outside of the task function definition at the task module level. At least for Mongo this allowed the tasks to share the connection.

Hope that helps.

https://github.com/instituteofdesign/wander/blob/master/wander/tasks.py

mongoengine.connect('stored_messages')

@celery.task(default_retry_delay = 61)
def pull(settings, google_settings, user, folder, messageid):
    '''
    Pulls a message from zimbra and stores it in Mongo
    '''

    try:
        imap = imap_connect(settings, user)
        imap.select(folder, True)
    .......
👤emperorcezar

Leave a comment