[Django]-MultipleObjectsReturned with get_or_create

by

11đź‘Ť

âś…

As the name implies, get_or_create model.objects.get()s or model.objects.create()s.

It’s conceptually equivalent to:

try:
   model.objects.get(pk=1)
except model.DoesNotExist:
   model.objects.create(pk=1)

The source is where you find definitive answers to these types of questions. Hint: search def get_or_create.
As you can see, this function only catches DoesNotExist in the try/except.

def get_or_create(self, **kwargs):
    """
    Looks up an object with the given kwargs, creating one if necessary.
    Returns a tuple of (object, created), where created is a boolean
    specifying whether an object was created.
    """
    assert kwargs, \
            'get_or_create() must be passed at least one keyword argument'
    defaults = kwargs.pop('defaults', {})
    lookup = kwargs.copy()
    for f in self.model._meta.fields:
        if f.attname in lookup:
            lookup[f.name] = lookup.pop(f.attname)
    try:
        self._for_write = True
        return self.get(**lookup), False
    except self.model.DoesNotExist:
👤Yuji 'Tomita' Tomita

28đź‘Ť

Example code

Imagine you have the following model:

class DictionaryEntry(models.Model):
    name = models.CharField(max_length=255, null=False, blank=False)
    definition = models.TextField(null=True, blank=False)

and the following code:

obj, created = DictionaryEntry.objects.get_or_create(
    name='apple', definition='some kind of fruit')

get_or_create

In case you have not seen the code for get_or_create:

 # simplified
 def get_or_create(cls, **kwargs):
     try:
         instance, created = cls.get(**kwargs), False
     except cls.DoesNotExist:
         instance, created = cls.create(**kwargs), True
     return instance, created

about webservers…

Now imagine that you have a webserver with 2 worker processes that both have their own concurrent access to the database. 

 # simplified
 def get_or_create(cls, **kwargs):
     try:
         instance, created = cls.get(**kwargs), False # <===== nope not there...
     except cls.DoesNotExist:
         instance, created = cls.create(**kwargs), True
     return instance, created

If the timing goes right (or wrong depending on how you want to phrase this), both processes can do the lookup and not find the item. They may both create the item. Everything is fine…

MultipleObjectsReturned: get() returned more than one KeyValue -- it returned 2!

Everything is fine… until you call get_or_create a third time, “third time is a charm” they say.

 # simplified
 def get_or_create(cls, **kwargs):
     try:
         instance, created = cls.get(**kwargs), False # <==== kaboom, 2 objects.
     except cls.DoesNotExist:
         instance, created = cls.create(**kwargs), True
     return instance, created

unique_together

How could you solve this? Maybe enforce a constraint at the database level:

class DictionaryEntry(models.Model):
    name = models.CharField(max_length=255, null=False, blank=False)
    definition = models.TextField(null=True, blank=False)
    class Meta:
        unique_together = (('name', 'definition'),)

back to the function:

 # simplified
 def get_or_create(cls, **kwargs):
     try:
         instance, created = cls.get(**kwargs), False
     except cls.DoesNotExist:
         instance, created = cls.create(**kwargs), True # <==== this handles IntegrityError
     return instance, created

Say you have the same race as before, and they both did not find the item and proceed to the insert; doing so they will start a transaction and one of them is going to win the race while the other will see an IntegrityError.

mysql ?

The example uses a TextField, which for mysql translates to a LONGTEXT (in my case). Adding the unique_together constraint fails the syncdb.

django.db.utils.InternalError: (1170, u"BLOB/TEXT column 'definition' used in key specification without a key length")

So, no luck, you may have to deal with MultipleObjectsReturned manually.

possible solutions

  • It may be possible to replace the TextField with a CharField.
  • It may be possible to add a CharField which may be a strong hash of the TextField, that you can compute in pre_save and use in a unique_together.
👤dnozay

4đź‘Ť

Another situation that could cause MultipleObjectsReturned error with get_or_create() API seems to be if there are multiple threads calling this API at the same time with the same set of query parameters.

Solely relying on try… catch… to create a unique row in Python wouldn’t work. If you are trying to use this API, I think you should have a matching uniqueness constraint on the appropriate columns in the database.

See: https://code.djangoproject.com/ticket/12579

👤AdvilUser

2đź‘Ť

Warning

This method is atomic assuming that the database enforces uniqueness of the keyword arguments (see unique or unique_together). If the fields used in the keyword arguments do not have a uniqueness constraint, concurrent calls to this method may result in multiple rows with the same parameters being inserted.

https://docs.djangoproject.com/en/3.2/ref/models/querysets/#django.db.models.query.QuerySet.get_or_create

👤alfonsoolavarria

Leave a comment