[Django]-How does sqs message deletion flow works in celery-sqs

2👍

When does celery delete message from sqs?
Message will be deleted after completing the task.

What happens if there is some error in task or it raises error?
Message is still with broker, and deleted after max_retries reaches.

Will the message (task) will be there if the tasks is taking too long like and 20 mins?
This depends on visibility timeout. Message goes to “Not Visible” state, till your visibility timeout, after that it is available to worker.
(if visibility timeout is less than retry time, worker will consume same message many times).

Best Practice is (visibility timeout) > (max_retries * retry_time)

1👍

The selected answer is (unfortunately) incorrect for SQS, as this open issue indicates.

There was an attempt at fixing the issue, as evidenced by this merged PR.

However, there are bugs with the above implementation.

Long story short, messages will be deleted from an SQS queue 100% of the time, regardless of any exception that occurs within the task.

edit: this may have been resolved, per this PR

I’ll update this answer after I’ve confirmed via personal testing that this functions correctly

Leave a comment