[Solved]-Django View Causes Psycopg2 Cursor Does/Does Not Exist Error

Usually, this error append when the models and the database are not compatible. Like if you changed a model adding a field but didn’t migrate it.

Be sure to check that all the applications of your site are in your INSTALLED_APP this error can be caused because the migrations doesn’t apply on a new non declared app.
Then

python manage.py makemigrations && python manage.py migrate

Each of the below mentioned solutions has its own cons.

• disable cursors: We’ll lose benefits of server side cursors (chunked resultset).
• wrap into transaction: This adds overhead of transaction and can decrease the query execution throughput on high traffic sites which uses lot of .iterator() querysets.
• extra connection: Developers have to remember to use separate database for .iterator() queryset.

So better approach could be to use two database settings. One for PgBouncer and second for direct db connection. (Both the database setting should point to same database in the backend) And route .iterator() queryset to use direct db connection based on transaction status.

Note: We should leave DISABLE_SERVER_SIDE_CURSORS=False (for both db settings) as PgBouncer supports server side cursors when iterator queryset is wrapped inside transaction.

DATABASE_URL: 'postgresql://django:xxx@localhost:7432/dbname'   # (pgbouncer connection)
DATABASE_URL_DIRECT: 'postgresql://django:xxx@localhost:6432/dbname' # (direct db connection)

inside settings.py

USE_PGBOUNCER = True
if USE_PGBOUNCER
    if 'migrate' not in sys.argv:
        # django app proccess
        DATABASES = {
            'default': dj_database_url.parse(config['DATABASE_URL']), # (pgbouncer connection)
            'direct_db': dj_database_url.parse(config['DATABASE_URL_DIRECT'])  # (direct db connection)
        }
     else:
        # django migration proccess
        DATABASES = {
            'default': dj_database_url.parse(config['DATABASE_URL_DIRECT'])  # (direct db connection)
        }
else:
    # not using pgbouncer.
    DATABASES = {
        'default': dj_database_url.parse(config['DATABASE_URL'])   # (direct db connection)
    }

while initilizing the django apps (inside AppConfig.ready())

from functools import wraps

from django.apps import AppConfig
from django.conf import settings
from django.db import transaction
from django.db.models.query import ModelIterable, ValuesIterable, ValuesListIterable, \
    NamedValuesListIterable, FlatValuesListIterable


class CommonAppConfig(AppConfig):
    name = 'app_name'

    def ready(self):

        if settings.USE_PGBOUNCER:
            direct_db = 'direct_db'. # DATABASE setting
            ModelIterable.__iter__ = patch_iterator_class(using=direct_db)(ModelIterable.__iter__)
            ValuesIterable.__iter__ = patch_iterator_class(using=direct_db)(ValuesIterable.__iter__)
            ValuesListIterable.__iter__ = patch_iterator_class(using=direct_db)(ValuesListIterable.__iter__)
            NamedValuesListIterable.__iter__ = patch_iterator_class(using=direct_db)(NamedValuesListIterable.__iter__)
            FlatValuesListIterable.__iter__ = patch_iterator_class(using=direct_db)(FlatValuesListIterable.__iter__)


def patch_iterator_class(using):
    def decorator(func):
        @wraps(func)
        def wrapper(self, *args, **kwargs):
            cxn = transaction.get_connection()
            if not self.chunked_fetch or cxn.in_atomic_block:
                # We are already in db transaction so use the same db connection (default) using
                # which db transaction was started to execute iterator query.
                # Or
                # We are neither in db transaction nor it is a chunked_fetch so continue over same db connection
                return func(self, *args, **kwargs)
            # We are not in any db transaction and it is chunked_fetch so redirect iterator query to use
            # direct_db connection to avoid cursor not found exception.
            self.queryset = self.queryset.using(using)  # redirect query to use direct db connection.
            return func(self, *args, **kwargs)
        return wrapper
    return decorator