[Fixed]-Can not use celery delay for saved form: object is not JSON serializable

by

31👍

✅

Task’s argument should be serializable (i.e. string, int, etc.). To fix error you can pass topic_id as argument and fetch topic object inside task method:

notify_new_topic.delay( uid, topic.id)

@shared_task
def notify_new_topic(flwd_id, topic_id):
    topic = Topic.objects.get(pk=topic_id)
    title = topic.title
    link = topic.slug

    flwd= cached_user(flwd_id) #User.objects.get(id = flwd_id)
    print 'flwd is', flwd.username
    flwr_ids = FollowUser.objects.filter(followed=flwd).values('follower_id')
    flwrs = User.objects.filter(id__in= flwr_ids).values('id', 'username','email') 

    for f in flwrs:
        print 'flwr username:',  f['username']
        if notify_flwdp_applies(int(f['id'])):
            print 'notify flwdp applies'
            make_alerts_new_topic(flwd_id, f['id'], topic)
            print 'back from make_alerts_new_topic'
👤neverwalkaloner

6👍

Since, the solution is already provided, I will try to explain why we can not pass non-serializable objects to celery tasks.

Why do we need to pass serializable objects to celery tasks?

With celery, we use a message broker (like Redis or RabbitMQ). Suppose we use Redis. When a celery task is called, the parameters are passed to Redis so that the broker can read them. And for this to happen, the datatype of those parameters ought to be supported by Redis.

Workaround

Suppose you want to pass a python dictionary as a parameter to a celery task, add these values to the celery configuration:

task_serializer = "json"  
result_serializer = "json"
accept_content = ["json"]

or you might want to do

celery.conf.update(
    task_serializer="json",
    result_serializer="json",
    accept_content=["json"]
)

For other cases, replace json in above with pickle, xml, etc.

Typical text-based serialization formats are csv, json, xml, yaml, toml, etc. Binary-based formats are protobuf and avro. Python also has several packages like pickle, numpy and pandas that supports serializing custom objects into byte format. You can also make your custom serializer.

What do these configurations do?

  1. instruct celery to serialize the python objects first and then pass them to the message broker.
  2. deserialize the objects from the message broker and then provide them to the celery worker.

References

👤Deepam Gupta

2👍

Change to picke enoding

app.conf.event_serializer = 'pickle' # this event_serializer is optional. 
app.conf.task_serializer = 'pickle'
app.conf.result_serializer = 'pickle'
app.conf.accept_content = ['application/json', 'application/x-python-serialize']
👤Sarath Ak

0👍

You can pass the instance as json and then recreate the instance from json no need to do database query again like this:

serialized_topic = topic.__dict__
serialized_topic.pop('_state')    
notify_new_topic.delay(uid, serialized_topic)

Now inside the task recreate the instance from the serialized topic

@shared_task
def notify_new_topic(flwd_id, serialized_topic):
    topic = Topic(**serialized_topic)
    # reset of the code
👤Ahtisham

Leave a comment