[Answer]-Groups for M2M object based permissions

distinct() method removes duplicate categories from queryset. Try this:

Category.objects.distinct().filter(allowed_groups__in=user.groups.all())