[Fixed]-Annotate queryset with count of filtered related objects

I might misunderstood your question but would this work?

User.objects.filter(Q(teammember__primary_contact=False) |
                    Q(teammember__isnull=True))