[Django]-Export a huge django queryset in CSV format with rest-framework

by

4👍

✅

You can make the response much faster by streaming the content, writing each row to the user as it’s generated.

Django has documentation to help with this exact problem:
https://docs.djangoproject.com/en/2.2/howto/outputting-csv/#streaming-large-csv-files

import csv

from django.http import StreamingHttpResponse


class Echo:
    def write(self, value):
        return value


def streaming_csv_view(request):
    queryset = User.objects.values_list("first_name", "last_name", "email")
    echo_buffer = Echo()
    csv_writer = csv.writer(echo_buffer)

    # By using a generator expression to write each row in the queryset
    # python calculates each row as needed, rather than all at once.
    # Note that the generator uses parentheses, instead of square
    # brackets – ( ) instead of [ ].
    rows = (csv_writer.writerow(row) for row in queryset)

    response = StreamingHttpResponse(rows, content_type="text/csv")
    response["Content-Disposition"] = 'attachment; filename="users.csv"'
    return response

For more information on the differences between generator expressions and list comprehensions, this python howto is a good resource:
https://docs.python.org/3/howto/functional.html#generator-expressions-and-list-comprehensions

Here’s the important part:

With a list comprehension, you get back a Python list; …a list containing the resulting lines, not an iterator. Generator expressions return an iterator that computes the values as necessary, not needing to materialize all the values at once.

To optimize your query, use a values_list query instead of calling all(). With a values_list query, you can still fetch fields through relationships. For example:

User.objects.values_list(
    "first_name",
    "last_name",
    "email",
    "profile__phone",  # get the profile.phone value
)
👤damon

Leave a comment