[Django]-Itertools.groupby in a django template

by

21👍

✅

I think that you’re right. I don’t understand why, but it looks to me like your groupby iterator is being pre-iterated. It’s easier to explain with code:

>>> even_odd_key = lambda x: x % 2
>>> evens_odds = sorted(range(10), key=even_odd_key)
>>> evens_odds_grouped = itertools.groupby(evens_odds, key=even_odd_key)
>>> [(k, list(g)) for k, g in evens_odds_grouped]
[(0, [0, 2, 4, 6, 8]), (1, [1, 3, 5, 7, 9])]

So far, so good. But what happens when we try to store the contents of the iterator in a list?

>>> evens_odds_grouped = itertools.groupby(evens_odds, key=even_odd_key)
>>> groups = [(k, g) for k, g in evens_odds_grouped]
>>> groups
[(0, <itertools._grouper object at 0x1004d7110>), (1, <itertools._grouper object at 0x1004ccbd0>)]

Surely we’ve just cached the results, and the iterators are still good. Right? Wrong.

>>> [(k, list(g)) for k, g in groups]
[(0, []), (1, [9])]

In the process of acquiring the keys, the groups are also iterated over. So we’ve really just cached the keys and thrown the groups away, save the very last item.

I don’t know how django handles iterators, but based on this, my hunch is that it caches them as lists internally. You could at least partially confirm this intuition by doing the above, but with more resources. If the only resource displayed is the last one, then you are almost certainly having the above problem somewhere.

👤senderle

27👍

Django’s templates want to know the length of things that are looped over using {% for %}, but generators don’t have a length.

So Django decides to convert it to a list before iterating, so that it has access to a list.

This breaks generators created using itertools.groupby. If you don’t iterate through each group, you lose the contents. Here is an example from Django core developer Alex Gaynor, first the normal groupby:

>>> groups = itertools.groupby(range(10), lambda x: x < 5)
>>> print [list(items) for g, items in groups]
[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]

Here is what Django does; it converts the generator to a list:

>>> groups = itertools.groupby(range(10), lambda x: x < 5)
>>> groups = list(groups)
>>> print [list(items) for g, items in groups]
[[], [9]]

There are two ways around this: convert to a list before Django does or prevent Django from doing it.

Converting into a list yourself

As shown above:

[(grouper, list(values)) for grouper, values in my_groupby_generator]

But of course, you no longer have the advantages of using a generator, if this is an issue for you.

Preventing Django from converting into a list

The other way around this is to wrap it in an object that provides a __len__ method (if you know what the length will be):

class MyGroupedItems(object):
    def __iter__(self):
        return itertools.groupby(range(10), lambda x: x < 5)

    def __len__(self):
        return 2

Django will be able to get the length using len() and will not need to convert your generator into a list. It’s unfortunate that Django does this. I was lucky that I could use this workaround, as I was already using such an object and knew what the length would always be.

👤Will Hardy

Leave a comment