Introduction to Logging in Django

Sivasubramaniam Arunachalam

July 20, 2013

@sivaa_in

http://www.meetup.com/BangPypers/events/125755952/

Agenda

• Introduction to Logging

• Python/Django Logging

• Few Best Practices

• Demo

Your Application misbehaved

Next, what the user will do?

https://twitter.com/naitikvyas/statuses/221096865574293504

https://twitter.com/vijaypadiyar/status/352439670732369921

https://twitter.com/PaulAppleyard/status/42294624353656832

The Users

• End Users

• System Administrators

• Developers

• Support Team

The Questions

• What

• When

• How Much

• How to Control

• How to Correlate

to Log

What to Log?It’s an Art

Exceptions

• No Managed Exceptions

• Global Exception Handler

Events

• Security Incidents

• Life Cycle

• Resource usage

States

• Retry

• Time-Outs

Exceptions/Events alone will not help much

Debug Traces

• User Input

• System/API Calls

• Method Invocation

And much more…

• HTTP Requests

• DB Calls

• Threads

Logging in Django

Logging in DjangoPython• Standard Module

• Multiple Destinations

• Custom Filtering Options

import logging

• Loggers

• Handlers

• Filters

• Formatters

Loggers

• Entry Point

• Unique Name

• Log Level

Log Level

• DEBUG

• INFO

• WARNING

• ERROR

• CRITICAL

Log Record• Log Level

• Logger Name

• Message– Error Code

– Stack Trace

• Others

– Path, Line #, args, func

Handlers

• Output Mechanism – File

– Screen/Console

– HTTP/Mail

– Queue

• Log Level

Default Handlers

• StreamHandler

• SocketHandler

• DatagramHandler

• SysLogHandler

• NTEventLogHandler

• MemoryHandler

• FileHandler– Default

– Rotating

– TimedRotating

– Watched

• NullHandler

• SMTPHandler

• HTTPHandler

Handler 02Logger ‘X’

Handler N

Handler 01

Handler E-MailPayment Logger

Handler File

Handler SMS

Handler E-MailPayment Logger

Handler File

Handler SMS

ERROR

Filters

• Conditional Filtering

• Changing Log Levels

• Chaining

Logger 02Filter ‘X’

Handler 01

Logger 01

Log Record will be Dropped, If

• Log Level is lower than– Logger’s Log Level

– Handler’s Log Level

• Filter Chain returns a False

Formatters

• The output text format

• Python Formatting String

http://docs.python.org/2/library/logging.html#logrecord-attributes

logging.method()• logger.log()

• logger.critical()

• logger.error()

• logger.warning()

• logger.info()

• logger.debug()

• logger.exception()

Django Extensions - Loggers

1. django

2. django.request

3. django.db.backends

django.request

• 4xx -> WARNING

• 5xx -> ERROR

• Contexts– HTTP Status Code (Response)

– Entire Request Object

django.db.backends

• SQLs -> DEBUG

• Only when DEBUG = True

• Contexts– SQL Statement

– Duration

– Parameters

Django Extensions – Handlers

• AdminEmailHandler

AdminEmailHandler

• E-Mails -> Site Admins

• Contexts– Request Object

– Stack Trace

• include_html = True/False– Full Trace/Local Vars/Settings

Django Extensions – Filters

• CallBackFilter

• RequireDebugFalse

• RequireDebugTrue

CallBackFilter

• Call Back Functions

• Called with Log Record

• From 1.4

RequireDebugFalse

• DEBUG = False

• From 1.4

RequireDebugTrue

• DEBUG = True

• From 1.5

settings.LOGGING

disable_existing_loggers

• False - Merge

• True - Ignore

• From 1.5

SENTRY

Best Practices

self.logger = logging.getLogger(type(self).__name__)

No Single Logger For All Classes

Project Level Formatters

Avoid “print” in development. Use “stdout” with a “StreamHandler”

Don’t Catch -> Log -> Re-raiseLet the caller do that

Something terribly wrong? Don’t use logging alone. Raise a Exception to the caller.

. Logging is NOT a way to deal with exceptions

Different Log File for each Environment

LOG_ROOT = '/var/log/<project>/<env>/'

Un-handled Exceptions should be stored separately

Sensitive information’s in Logs

Size of the Log Files

• Text Editor Problems

• Archive via Scripts

• Compress to Save Space

Only Integers1300604499194,4,192168001002,20600,1001,2,500000

Be aware of cluster nodesserver<id>_node<id>_<logname>.log

is_debug()

I/O Performance

Batch Logging

No Repeated Logs & Stack Traces

Infrastructure Failures

End Users -> No Stack Trace Please!

Centralized Logging

Thank You!siva@sivaa.in

bit.ly/sivasubramaniambit.ly/sivaa_in

References

http://css.dzone.com/articles/best-practices-python-logginghttp://architects.dzone.com/articles/deferred-logging-file-handlerhttp://architects.dzone.com/articles/high-performance-and-smarterhttp://java.dzone.com/articles/dark-art-logginghttp://architects.dzone.com/articles/think-ahead-think-logginghttp://www.shutupandship.com/2012/02/how-python-logging-module-works.htmlhttp://java.dzone.com/news/application-logging-what-when