1đź‘Ť
With your question you are entering in a conceptual/architectural domain rather than in a “this particular view of the data in my project is hard to represent in the template layer of django”. So I will try to give you the birds view (when flying and not on the ground) of the problem and you can decide for yourself.
From the first philosophy box in the django template language documentation it’s clearly stated that templates should have as little program logic as possible. This indicates that the representation of the data used in the template should be simple and totally adapted to the template you are trying to build (this is my interpretation of it). This approach indicates that you should have a layer responsible for intermediating the representation of your data (models or other sources) and the data that your template needs to achieve the final representation you want you users to see.
This layer can simple stay in your view, in viewXXX_organize_data, or in some other form respecting to a more complex/elaborated architecture (see DCI or Hexagonal).
In your case I would start by doing something like viewXXX_organize_data() where I would use the most appropriate data structures for the template you are trying to build, while keeping some independence from the way you obtain your data (through models other services etc).
You can even think of not using you model objects directly in the template and creating template specific objects to represent a certain view of the data.
Hope this helps you make a decision. It’s not a concrete answer but will help you for sure make a decision and then staying coherent all trough your app.