Logging Best Practices

Dubna 2012

Benedicto Fernandez

Software Engineer

CERN / GS-AIS

Only useful for debugging/development?

Contents

• Logging libraries

• Logging levels

• Where to log?

• Performance

• How to access?

• What to log?

What are we talking about?

First Logging: System.out.println

• Easy

• Quick

• No dependencies

• Redirect to file or null in Production– Log everything or nothing

System.out.println(“Start method1”);

ExampleStart method1

Before retrieve information from DB

After retrieve information from DB

Updating information

Before save information in the DB

After save information in the DB

Finish method1

Start method2

Before retrieve information from DB

Error retrieving information from DB: NullPointerException

Next evolution:Logging Libraries

• Log4j, Java Logging…log.info(“Start method1”);

• Provide– Time of the message– The class, the method, the line– …

• Print easily the stacktrace• Log to files, databases, system log

mechanisms

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method206:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Logging levels (I)

• Log4j– Trace– Debug– Info– Warning– Error– Fatal

• Java Logging– Finest– Finer– Fine– Config– Info– Warning– Severe

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2

06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer

06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2

06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException

java.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Example06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer

06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2

06:05:56 07.11.2011 INFO (Class1.java:211) Start method2

06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB

06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerExceptionjava.lang.NullPointerException

at pkg.Class1.method2(Class1.java:225)

at pkg.Class1.methodN(Class1.java:458)

at pkg2.ClassZ.methodUpdate(ClassZ.java:458)

at …

at java.lang.Thread.run(Thread.java:595)

Logging levels (II)

log4j.logger.ch.cern.ais=DEBUG

log4j.logger.ch.cern.ais.package.Class1=TRACE

log4j.logger.org.springframework=ERROR

log4j.logger.org.springframework.ws=INFO

Where to log?

• Text files (Plain text, XML,…)• Binary files• Database• Syslog (Windows & Unix/Linux)• Network (TCP & UDP)

Where to log?

• Text files (Plain text, XML,…)• Binary files• Database• Syslog (Windows & Unix/Linux)• Network (TCP & UDP)

Always to a Text file (+ something else)?

Performance

• "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is " +

String.valueOf(entry[i]));

Performance

• "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is "

+ String.valueOf(entry[i]));

• Solutionif (logger.isDebugEnabled()) {

logger.debug("Entry number: " + i + " is "

+ String.valueOf(entry[i]));

}

Next evolution:Logging Libraries v2.0

• Simple Logging Facade (SLF4J)– Logback

• Parameterized messageslogger.debug("Entry number: {} is {}“, i

, String.valueOf(entry[i]));

Multiple users (I)

• Different users working simultaneously

• Log messages mixed for different users

• Difficult to trace user’s actions

Multiple users (I)

• Different users working simultaneously

• Log messages mixed for different users

• Difficult to trace user’s actions

Log User ID

Multiple users - Example06:05:55 07.11.2011 INFO (Class1.java:41) User=bfernan; message=‘Start method1’;06:05:55 07.11.2011 DEBUG (Class1.java:44) message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 DEBUG (Class1.java:46) message=‘After retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:50) message=‘Updating information’;

06:05:55 07.11.2011 DEBUG (Class1.java:94) message=‘Before save information in the DB’;

06:05:56 07.11.2011 DEBUG (Class1.java:96) message=‘After save information in the DB’;

06:05:56 07.11.2011 INFO (Class1.java:100) User=bfernan; message=‘Finish method1’;

06:05:56 07.11.2011 INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’;06:05:56 07.11.2011 DEBUG (Class1.java:220) message=‘Before retrieve information from DB’;

06:05:56 07.11.2011 ERROR (Class1.java:243) message=‘Error retrieving information from DB: NullPointerException’;

Multiple users (II)

• Mapped Diagnostic Contexts (MDC)public void serviceMethod1(String userName,…) {

String userName = getLoggedUserName();

MDC.put( “userName”, userName);

}

• Configuration…PatternLayout=%T User=%X{userName};message=‘%m’;%n

Multiple users (III)public void doFilter(ServletRequest request, ServletResponse response,

    FilterChain chain) throws IOException, ServletException {

    HttpServletRequest req = (HttpServletRequest) request;        Principal principal = req.getUserPrincipal();    // Please note that we could have also used a cookie to     // retrieve the user name

    if (principal != null) {      String username = principal.getName();      MDC.put(USER_KEY, username);    }

    try {      chain.doFilter(request, response);    } finally {      MDC.remove(USER_KEY);    }  }

Multiple users (III)public void doFilter(ServletRequest request, ServletResponse response,

    FilterChain chain) throws IOException, ServletException {

    HttpServletRequest req = (HttpServletRequest) request;        Principal principal = req.getUserPrincipal();    // Please note that we could have also used a cookie to     // retrieve the user name

    if (principal != null) {      String username = principal.getName();      MDC.put(USER_KEY, username);    }

    try {      chain.doFilter(request, response);    } finally {      MDC.remove(USER_KEY);    }  }

Multiple users - Example06:05:55 07.11.2011 INFO (Class1.java:41) User=bfernan; message=‘Start method1’;

06:05:55 07.11.2011 DEBUG (Class1.java:44) User=bfernan; message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 DEBUG (Class1.java:46) User=bfernan; message=‘After retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:50) User=bfernan; message=‘Updating information’;

06:05:55 07.11.2011 DEBUG (Class1.java:94) User=bfernan; message=‘Before save information in the DB’;

06:05:56 07.11.2011 DEBUG (Class1.java:96) User=bfernan; message=‘After retrieve information in the DB’;

06:05:56 07.11.2011 INFO (Class1.java:100) User=bfernan; message=‘Finish method1’;

06:05:56 07.11.2011 INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’;

06:05:56 07.11.2011 DEBUG (Class1.java:220) User=xxxxxxx; message=‘Before retrieve information from DB’;

06:05:56 07.11.2011 ERROR (Class1.java:243) User=xxxxxxx; message=‘Error retrieving information from DB: NullPointerException’;

Multiple Connections Same User (I)

• “Account” shared by 2+ end users

• Different – Session ID– Remote Address

user1

Multiple Connections Same User (I)

• “Account” shared by 2+ end users

• Different – Session ID– Remote Address

Log Session ID

Log Remote IP

user1

Multiple Connections Same User (I) - Example

06:05:55 07.11.2011 INFO (Class1.java:41) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘Start method1’;

06:05:55 07.11.2011 DEBUG (Class1.java:44) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:211) IP=61.82.23.9;Session=00000002; User=bfernan;message=‘Start method2’;

06:05:55 07.11.2011 DEBUG (Class1.java:220) IP=61.82.23.9;Session=00000002; User=bfernan;message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 DEBUG (Class1.java:46) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘After retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:50) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘Updating information’;

06:05:56 07.11.2011 DEBUG (Class1.java:94) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘Before save information in the DB’;

06:05:56 07.11.2011 DEBUG (Class1.java:96) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘After retrieve information in the DB’;

06:05:56 07.11.2011 ERROR (Class1.java:243) IP=61.82.23.9;Session=00000002; User=bfernan;message=‘Error retrieving information from DB: NullPointerException’;

06:05:56 07.11.2011 INFO (Class1.java:100) IP=192.168.2.99;Session=00000001; User=bfernan;message=‘Finish method1’;

Multiple Connections Same User (II)

• What about tabs?– Same Remote IP– Same Session ID

Multiple Connections Same User (II)

• What about tabs?– Same Remote IP– Same Session ID

Log Transaction ID

Multiple Connections Same User (I) - Example

06:05:55 07.11.2011 INFO (Class1.java:41) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘Start method1’;

06:05:55 07.11.2011 DEBUG (Class1.java:44) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:211) IP=192.168.2.99;Session=00000001; Transaction=00000002;User=bfernan;message=‘Start method2’;

06:05:55 07.11.2011 DEBUG (Class1.java:220) IP=192.168.2.99;Session=00000001; Transaction=00000002;User=bfernan;message=‘Before retrieve information from DB’;

06:05:55 07.11.2011 DEBUG (Class1.java:46) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘After retrieve information from DB’;

06:05:55 07.11.2011 INFO (Class1.java:50) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘Updating information’;

06:05:56 07.11.2011 DEBUG (Class1.java:94) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘Before save information in the DB’;

06:05:56 07.11.2011 DEBUG (Class1.java:96) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘After retrieve information in the DB’;

06:05:56 07.11.2011 ERROR (Class1.java:243) IP=192.168.2.99;Session=00000001; Transaction=00000002;User=bfernan;message=‘Error retrieving information from DB: NullPointerException’;

06:05:56 07.11.2011 INFO (Class1.java:100) IP=192.168.2.99;Session=00000001; Transaction=00000001;User=bfernan;message=‘Finish method1’;

How to access the information? v1.0

• View the file

– Human readable format

• System utils (grep command)

$ grep '.*ERROR.* User bfernan .*' log_cra-ws.log

– Easy for regext (Ex: key=value;)

$ grep ‘.*ERROR.*;User=bfernan;.*’ log_cra-ws.log

Cluster of Servers

• Application distributed several Servers

• 1 log file for each server

Which one contains the error?

How to access the information? v2.0

• Forward log messages to a Server– Log4j Server

• UI Tools– Apache Log4j Chainsaw– MindTree Insight

Analytic Tools

• Splunk– Indexer– Search Engine– “Pseudo” real-time– Alerts

Example

Example

Example

What to log?

• User requests (start & end)

• Service methods calls (start & end)

• Calls to external systems ( before & after)

• Errors

• Any relevant state

Conclusions (I)

• Log everything useful, not only debug

information

• Log to text files, human readable format

• Use a template parseable with regexp (Ex:

key=value;)

Conclusions (II)

• Use the appropriate level

• One single action per line

• Be careful with the performance

• Be ready to profit of it

Why not logging standards?

“A good job can save a day of work”

Спасибо!

Benedicto Fernandez

Software Engineer

CERN / GS-AIS