[Fixed]-Django server not sending logs to Logstash

by

6👍

✅

Looking at the configuration, logger “django.request” is set to level “DEBUG” and the handler “logstash” is set to level “INFO”. My guess is that the handler will not process DEBUG messages. I’m not sure though.

Set the same level for the logger and the handler to test that it works.

What level to use depends on what you want from your logs. In this case I guess level INFO would suffice.

If not already take a look at Django logging

NOTE: From comments it seems not to solve the problem but I hope it is useful anyway.

UPDATE:

I tried the following configuration and it catches 404 and 500 errors in the “debug.log”.

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
    'logfile': {
        'level': 'WARNING',
        'class': 'logging.FileHandler',
        'filename': os.path.join(PROJECT_DIR, 'debug.log'),
    },
},
'loggers': {
    'django.request': {
        'handlers': ['logfile'],
        'level': 'WARNING',
        'propagate': True,
    },
}}

With this test configuration the logstash handler should at least receive the message/logrecord. If no luck I suggest to try debug the logstash.TCPLogstashHandler and the
SocketHandler (inherited by TCPLogstashHandler) to make sure they receive the emitted record.

👤Daniel Backman

2👍

I faced the same issue but after scrambling around a lot, I made it work by changing django.request to django.server.
I got the idea when I used only django as a logger name in the python code and then found out the actual logger_name from log data stored in elasticsearch.
Below is the updated code

LOGGING = {
  'version': 1,
  'handlers': {
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'host': 'xx.xx.xx.xx',
            'port': 5959, # Default value: 5959
            'version': 0, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
            'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
            'fqdn': True, # Fully qualified domain name. Default value: false.
            'tags': ['django.request'], # list of tags. Default: None.
        },
  },
  'loggers': {
        'django.server': {  # Here is the change
            'handlers': ['logstash'],
            'level': 'DEBUG',
      }
  },
}

Refer to this for more detail on django logging
https://docs.djangoproject.com/en/1.11/topics/logging/#id3

👤Tasneem Haider

0👍

Your logging configuration is correct.
You need to mention index name in elasticsearch config part of logstash conf. Update your logstash config to 

input {
    tcp {
    port => 5959
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["xx.xx.xx.xx:9200"]
    manage_template => false
    index => "djangologs"
  }
}

If you are using google cloud or AWS, open port/update firewall rules.

👤wittyurchin

0👍

Thank you Tasneem,

This worked on my machine:

mysite/settings.py

LOGGING = {
  'version': 1,
  'handlers': {
      'logstash': {
          'level': 'DEBUG',
          'class': 'logstash.TCPLogstashHandler',
          'host': 'xx.xx.xx.xx',
          'port': 9600, # Default value: 5959
          'version': 1, # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
          'message_type': 'django',  # 'type' field in logstash message. Default value: 'logstash'.
          'fqdn': False, # Fully qualified domain name. Default value: false.
          'tags': ['django.request'],# list of tags. Default: None.
      },
  },
  'loggers': {
      'django.server': {
          'handlers': ['logstash'],
          'level': 'DEBUG',
      },
  },
}

logstash# cat logstash-django.conf

input {
  tcp {
    port => 9600
    codec => json
  }
}
output {
    elasticsearch {
      hosts => ["https://your.elasticsearch.com:port"]
      user => ["xx"]
      password => ["xx"]
      index => "djangotest"
      document_type => "%{type}"
    }
   stdout {
      codec => rubydebug {metadata => true }
  }
}
👤kujiy

Leave a comment