[Fixed]-How to throttle Django error emails

by

16👍

✅

Using Gattster’s great answer as an example, I wrote a simple implementation based on django’s built-in cache functions.

# -*- coding: utf-8 -*-

from django.utils.log import AdminEmailHandler
from django.core.cache import cache


class ThrottledAdminEmailHandler(AdminEmailHandler):

    PERIOD_LENGTH_IN_SECONDS = 10
    MAX_EMAILS_IN_PERIOD = 1
    COUNTER_CACHE_KEY = "email_admins_counter"

    def increment_counter(self):
        try:
            cache.incr(self.COUNTER_CACHE_KEY)
        except ValueError:
            cache.set(self.COUNTER_CACHE_KEY, 1, self.PERIOD_LENGTH_IN_SECONDS)
        return cache.get(self.COUNTER_CACHE_KEY)

    def emit(self, record):
        try:
            counter = self.increment_counter()
        except Exception:
            pass
        else:
            if counter > self.MAX_EMAILS_IN_PERIOD:
                return
        super(ThrottledAdminEmailHandler, self).emit(record)

And the logging configuration has been changed in Django 1.9 as well, so in order for this handler to work, you need to configure logging as:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'fully.qualified.path.to.handler.ThrottledAdminEmailHandler'
        }
    },
    'loggers': {
        'django': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

where the change is only to change the name of the logger from django.request to django. If you look in the logging system documentation, this could probably be achieved in a cleaner (?) way by implementing a logging filter.

👤Dennis Golomazov

8👍

I limited emails to 10 per minute by doing the following. This uses a redis connection function unique to my install. I suggest modifying the incr_counter function to suit your needs. To be safe, use a direct redis or memcache connection for this and not any django.cache wrappers.

settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'error_email_limiter.handler.MyAdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
            },
        }
}

error_email_limiter/handlers.py

class MyAdminEmailHandler(AdminEmailHandler):
    def incr_counter(self):
        c = get_redis_connection()
        key = self._redis_key()
        res = c.incr(key)
        c.expire(key, 300)
        return res

    def _redis_key(self):
        return time.strftime('error_email_limiter:%Y-%m-%d_%H:%M',
                             datetime.datetime.now().timetuple())

    def emit(self, record):
        try:
            ctr = self.incr_counter()
        except Exception:
            pass
        else:
            if ctr >= 10:
                return
        super(MyAdminEmailHandler, self).emit(record)
👤Gattster

2👍

One option is to switch to something like ErrorStack for error-reporting. I wrote a django app to make it stupid-simple to integrate into your project.

👤Carl Meyer

1👍

I presume the database downtime was not intentional, in which case you probably should have put your Django process into some kind of maintenance mode or taken it offline?

Otherwise, the usual mail app is django-mailer which might have helped you simply because it stores outgoing mail in your database and thus would have failed 🙂

If you really need to rate limit it would be best to do so in your MTA. This could mean the ability to just shut down the part of the MTA process responsible for sending mail or something exotic like using this patch for qmail to throttle incoming connections as a means of fighting spam

👤Van Gale

0👍

maybe Nagios’ page on flapping is worth reading

👤thanos

Leave a comment