[Django]-Django find_each (like RoR)

by

6👍

✅

One possible solution could be to use the built-in Paginator class (could save a lot of hassle).

https://docs.djangoproject.com/en/dev/topics/pagination/

Try something like:

from django.core.paginator import Paginator
from yourapp.models import YourModel

result_query = YourModel.objects.filter(<your find conditions>)

paginator = Paginator(result_query, 1000) # the desired batch size

for page in range(1, paginator.num_pages + 1):
   for row in paginator.page(page).object_list:
       # here you can add your required code

Or, you could use the limiting options as per needs to iterate over the results.

👤Abhishek Agarwala

0👍

You can query parts of the whole table with a loop and by slicing the queryset.

If you’re working with Debug = True it’s important that you flush your queries after each loop since this can cause memory issues (Django stores all the queries that were run until the script finishes or dies).

If you need to restrict the results of the queryset you can replace “.all()” with the appropriate “.filter(conditions)”

from django import db
from myapp import MyModel

# Getting the total of records in the table
total_count = MyModel.objects.all().count()
chunk_size = 1000 # You can change this to any amount you can keep in memory
total_checked = 0

while total_checked < total_count:
    # Querying all the objects and slicing only the part you need to work
    # with at the moment (only that part will be loaded into memory)
    query_set = MyModel.objects.all()[total_checked:total_checked + chunk_size]

    for item in query_set:
        # Do what you need to do with your results
        pass

    total_checked += chunk_size

    # Clearing django's query cache to avoid a memory leak
    db.reset_queries()
👤Alex Carlos

Leave a comment