[Answer]-Django pagination and annotate

I ended up adding raw sql to improve the query.
this did the trick and now each pagination takes ~3 ms.