[Django]-"Too many SQL variables" error in django with sqlite3

actually those limits are given here:
https://www.sqlite.org/c3ref/c_limit_attached.html#sqlitelimitvariablenumber

with an explanation:

Maximum Number Of Host Parameters In A Single SQL Statement

A host parameter is a place-holder in an SQL statement that is filled
in using one of the sqlite3_bind_XXXX() interfaces. Many SQL
programmers are familiar with using a question mark (“?”) as a host
parameter. SQLite also supports named host parameters prefaced by “:”,
“$”, or “@” and numbered host parameters of the form “?123”.

Each host parameter in an SQLite statement is assigned a number. The
numbers normally begin with 1 and increase by one with each new
parameter. However, when the “?123” form is used, the host parameter
number is the number that follows the question mark.

SQLite allocates space to hold all host parameters between 1 and the
largest host parameter number used. Hence, an SQL statement that
contains a host parameter like ?1000000000 would require gigabytes of
storage. This could easily overwhelm the resources of the host
machine. To prevent excessive memory allocations, the maximum value of
a host parameter number is SQLITE_MAX_VARIABLE_NUMBER, which defaults
to 999.

The maximum host parameter number can be lowered at run-time using the
sqlite3_limit(db,SQLITE_LIMIT_VARIABLE_NUMBER,size) interface.

https://www.sqlite.org/limits.html

Googling this error message brought me here so adding my solution.

In my case this error was caused by line:

Event.objects.all().delete()

Probably Django was trying to “DELETE WHERE id IN (?,?,?,…” and list of ids was to long.

Performance was not an issue for me so i solved this by:

while Event.objects.count():
    ids = Event.objects.values_list('pk', flat=True)[:100]
    Event.objects.filter(pk__in = ids).delete()

There is no way to bypass this, it’s an SQLITE limitation. Your query will run fine on MySQL however. But if you have to do this kind of query, you are most probably doing it wrong. Your mysql schema should be redone. And if you can’t, then you may want to break this query into 100 of small queries.

I hava found the same error when I run the line:

    Entry.objects.all().delete()

I solved the problem by

    while Entry.objects.count():
          Entry.objects.all()[0].delete()

I think it’s a better idea than others.

Note: I am not Django expert

The problem is that in your query

vals = Company.objects.filter(id__in=comp_ids)... 

you pass a bloated set of values and the underlaying engine cannot execute the query.
Fortunately sqlalchemy, that you execute in that line, is lazy. You an execute a SELECT and then filter the entries “manually” without big performance penalties.

It would look more-less like that:

# here you should do a proper session.query(CompanyTable)
vals_all = Company.objects

vals_filtered = (item for item in vals_all if item.id in comp_ids)
vals_ordered = sorted(vals_filtered, key=attrgetter('name'))
vals_final = [(v.id, v.name) for v in vals_ordered]

I had a similar problem, but when deleting just single row that had a large number of related rows in another table. Django was calling a cascading delete causing the problem, but since my code was only calling a single object delete, the solutions described in other answers could not be applied (unless I modified the Django library code, which I didn’t want to do).

My solution was to first delete the related objects via raw sql query, then use the normal Django delete() on the parent object, all wrapped in a transaction.

Before:

object.delete() # Django tries to delete a large number of related object and fails

After:

from django.db import connection

Class MyObject:
    def delete_related(self):
        cur = connection.cursor()
        sql = 'delete from my_table where ...'
        cur.execute(sql, params)

...

with transaction.atomic():
    object.delete_related() # so that Django won't try (and fail) on its own
    object.delete()