[Answer]-Django Related Field based filter

I think you want,

contacts = Contacts.objects.filter(user=request.user, phone_numbers__isnull=False)

i.e check for False rather than True, as you want Contacts where at least one PhoneNumber is there. If its not there it will be NULL.