[Answer]-Django heavy database query and data processing

1👍

One very quick optimization would be to make names a set, since list lookup is linear (O(n)) whereas set lookup is constant (O(1)):

names = set()
doubleList = []
for number, object1 in enumerate(allinstances):
    if object1.name not in names:
        names.add(object1.name)

0👍

I think a lot of your performance issues are coming from Django having to run ~25000 queries happening when you have a lot of records.

prefetch_related is what I would use here.

The first section of your code would look like the following:

def some_func(user):
    group1 = user.manytomanyfield1.all().prefetch_related('manytomanyfield2')
    allinstances = []
    for item in group1:
         allinstances += item.manytomanyfield2.all()
👤chrisg

Leave a comment