[Django]-Kombu.exceptions.EncodeError: User is not JSON serializable

by

22👍

✅

The error is because of Celery expecting a JSON data from your task function while you returned a User instance.

How to solve this ?
You are not using that return data anywhere, so you don’t have to return it. That is you can remove return user from the task function.
Or,
return a Json data from the task function will solve this issue as well

Solution 1

@task(serializer='json')
def task_number_one():
    user = User.objects.create(username="testuser", email="test@test.com", password="pass")

Solution 2

@task(serializer='json')

def task_number_one():
    user = User.objects.create(username="testuser", email="test@test.com", password="pass")
    # return some json data instead of `USER` instance
    return {"status": True}  # Change is here
👤JPG

38👍

This is because you are using the JSON serializer for task serialization (as indicated by the setting CELERY_TASK_SERIALIZER = 'json'), but you are trying to return a model instance (which cannot be serialized into JSON).

You have two options:

1) Don’t pass the instance, pass the primary key of the instance and then look up the object inside your task.

2) Use the pickle task serializer instead. This will allow you to pass objects as arguments to your tasks and return them, but comes with it’s own security concerns.

👤MrName

0👍

My celery task:

response = {}
...
except HTTPError as e:
        response.update(
            {
                'status': False,
                'code': e.status_code,
                'error': e.body,
            },
        )
    ...
return response

I had EncodeError(TypeError('Object of type bytes is not JSON serializable') and kombu.exceptions.EncodeError although the response is a dict that shouldn’t be a problem when JSON encoding.

It turned out that e.body is of type bytes. I changed to e.body.decode('utf-8') and the problem disappeared.

👤Serhii Kushchenko

0👍

Another answer not related to this question but also useful

if you pass obj to task, may got the same error info, then you can:

@shared_task(serializer="pickle")
def email_send_task(msg: EmailMultiAlternatives):
    try:
        msg.send()
    except (smtplib.SMTPException, TimeoutError) as e:
        return f"Email failed with {e}"
👤C.K.

0👍

For the following configuration:

  1. Django==3.0.11
  2. redis==2.10.6
  3. celery==4.0.0
  4. kombu==4.0.0

I updated the following configuration on the settings.py file and it worked.

CELERY_SETTINGS = {
    'CELERY_TIMEZONE': TIME_ZONE,
    'CELERY_ENABLE_UTC': True,
    'CELERY_RESULT_BACKEND': REDIS_URL,
    'CELERY_SEND_TASK_SENT_EVENT': True,
    'CELERY_TASK_SERIALIZER': 'pickle',
    'CELERY_RESULT_SERIALIZER': 'pickle',
    'CELERY_ACCEPT_CONTENT': ['pickle', 'json'],
}

Don’t forget to update the actual value of TIME_ZONE and REDIS_URL

The reason could be the Celery 4 version uses JSON as the serializer by default, while the celery3 version uses Pickle by default.

The old Django may not be expecting the JSON format from the tasks.
So, If you are using the very old Django version, this could help you.

👤Tapan B.K.

Leave a comment