Logging and Tracing for your Microservices – Log4j ... · .consulting .solutions.partnership...

.consulting .solutions .partnership

Logging and Tracing for your Microservices –Log4j, Zipkin and Spring SleuthAlexander Schwartz, Principal IT ConsultantJAX DevOps – 11 April 2018

Logging and Tracing for your Microservices

© msg | April 2018 | Logging and Tracing for your Microservices | Alexander Schwartz 2

Searching for Errors1

Context Information with Log4j2

Context Information for Web Applications3

Tracing across Services and Interfaces4

Tools for Development and Production5

Recap – What to use and where to start6

About me – Principal IT Consultant @ msg Travel & Logistics


15 year Java

7 years PL/SQL

7 yearsconsumer finance

3,5 years online banking

1 wife

2 kids600+

Geocaches

@ahus1de









Searching for Errors

An error occurred


That doesn’t work! What’s in the logs?

Searching for Errors

Welcome to Logging Hell!


Log:

Code:

for (Invoice i : invoiceRespository.findAll()) {

i.calculateTotal();

}

07:26:00.595 d.a.t.d.Invoice ERROR - can't load item ID 4711


8© msg | April 2018 | Logging and Tracing for your Microservices | Alexander Schwartz







Context Information with Log4j

Brief history of Log4j


Features:API for logging, implementation for different log formats,asynchronous logging, re-configuration at runtime, filters

Availability and License:• Apache Projekt with Apache License 2.0

History:• 1.2 since May 2002 (2015 end of life)• 2.0 since July 2014• 2.11 since March 2018


Logging with context information – log4j 1.x and slf4j


for (Invoice i : respository.findAll()) {

MDC.put(INVOICE_ID, Long.toString(i.getId()));

try {

i.calculateTotal();

} finally {

MDC.remove(INVOICE_ID);

}

}


Logging with context information – log4j 2.x


for (Invoice i : respository.findAll())

try (final CloseableThreadContext.Instance c

= CloseableThreadContext.put(

INVOICE_ID, String.valueOf(i.getId()))) {

i.calculateTotal();

}

}


Adding context information to the log


Configuration:

Log output:

<PatternLayout pattern="%d{HH:mm:ss.SSS} %X %-5level ..."/>

08:39:42.969 {invoiceId=1} ... - can't load item ID 4711









Context Information for Web Applications

Filters for more context information in the log


@Provider

public class RequestFilter implements ContainerRequestFilter {

@Override

public void filter(ContainerRequestContext containerRequestContext) {

MDC.put("http.url",

containerRequestContext.getUriInfo().getRequestUri().toString());

MDC.put("http.method", containerRequestContext.getMethod());

Principal principal =

containerRequestContext.getSecurityContext().getUserPrincipal();

if(principal != null) {

MDC.put("user", principal.getName());

} else {

MDC.put("user", "anonymous");

}

}

}


Filters for more context information in the log


@Provider

public class ResponseFilter implements ContainerResponseFilter {

@Override

public void filter(ContainerRequestContext containerRequestContext,

ContainerResponseContext containerResponseContext) {

MDC.remove("http.url");

MDC.remove("http.method");

MDC.remove("user");

}

}


Log output for Web Applications


08:52:54.276 {http.method=GET,

http.url=http://localhost:8080/api/startBillingRun, invoiceId=1,

user=Theo Tester} ERROR d.a.t.d.Invoice - can't load item ID 4711

Further fields that might of of interest:• Client-IP-Address• Part of the Session-ID• Browser name (User Agent)









Tracing across Services and Interfaces

Tools for Development and Production


One user requests writes log entries on three different servers


Concept Dapper and its application in Zipkin


pass trace-ID andadditionally a span-ID

assign Trace-ID


How it works


Additional HTTP Header:

GET /api/callback HTTP/1.1

Host: localhost:8080

...

X-B3-SpanId: 34e628fc44c0cff1

X-B3-TraceId: a72f03509a36daae

...

09:20:05.840 {

X-B3-SpanId=34e628fc44c0cff1,

X-B3-TraceId=a72f03509a36daae,

..., invoiceId=1} ERROR

d.a.t.d.Invoice - can't load

purchase ID 4711

Additional Informationen in the Log:


What you need


Do-it-yourself:Servlet-Filter to read information from incoming requestWrapper for calls to remote systems to pass on context information

For Spring-Boot:Add Spring Sleuth [slyth] as a dependency

Building blocks for other projects:Zipkin‘s “brave” libraries for client- and server-side handling










What is Zipkin?


Core Functionality:• Distributed Tracing • Collect context and timing information• Central store for collected information to analyze errors, timing issues and dependencies

(in production usually only a small percentage of the requests is traced)

Concepts:• Re-implementation of the concepts of Google Dapper, that is an internal closed-source tool at Google• First used as part of Finagle at Twitter, now Open Source


Zipkin Server receives information and provides a Web UI


For each trace-/span-ID:• start-/end time client• start-/end time server• tags and logs


Web UI Zipkin



Adding more information


@Component

public class BillingResource {

@Autowired

private Tracer tracer;

public String startBillingRun() {

/* ... */

Span span = tracer.getCurrentSpan();

span.logEvent("log a line");

tracer.addTag("tag", "value");

/* ... */

}

}


Zipkin Browser-Plugin for Chrome


Add trace-IDto HTTP Headersat client

Direct link to traces view in a new tab


Zipkin Browser-Plugin for Chrome



Bonus: Per-Request-Debugging-Logging


• Zipkin and Sleuth pass on the Header X-B3-Flags, that can encode additional information• Zipkin-Chrome-Plugin passes the value “1” (debug=true)• A Servlet-Filter can pick up this value and pass it to the MDC

(for example set X-B3-Flags-debug to true)• The following log4j2-Konfiguration activates the trace level for requests that have this HTTP-Header

<DynamicThresholdFilter key="X-B3-Flags-debug" onMatch="ACCEPT"

defaultThreshold="warn" onMismatch="NEUTRAL">

<KeyValuePair key="true" value="trace"/>

</DynamicThresholdFilter>









Recap

What to use and where to start


Use … to get …

Thread Context / Diagnostic Context Context information in all dependent calls (useful for iterations in master/detail structures)

@ahus1de

Recap



Use … to get …


Filter for Web requests Context information about the user and the called URL

@ahus1de

Recap



Use … to get …



Trace-Header á la Dapper with Zipkin/Brave/Sleuth

Correlation of logs with trace-ID and span-ID in development and production, including per-request debug logs

@ahus1de

Recap



Use … to get …





Zipkin-Server Searchable traces including timing and dependencies for development and production

@ahus1de

Recap



Use … to get …






Zipkin Browser Plugin Externally provided trace-IDs, direct search in development and production, enforced traces (if disabled by default in production or limited to only a percentage of the requests)

@ahus1de

Recap



Use … to get …






Zipkin Browser Plugin Externally provided trace-IDs, direct search in development and production, enforced traces (if disabled by default in production or limited to only a percentage of the requests)

@ahus1de

Links


Log4j 2 – insbesondere NDC/MDC https://logging.apache.org/log4j/2.x/manual/thread-context.html

Zipkin, Bravehttps://github.com/openzipkin/zipkinhttps://github.com/openzipkin/brave

Spring Sleuthhttps://github.com/spring-cloud/spring-cloud-sleuth

@ahus1de

Dapperhttp://research.google.com/pubs/pub36356.html

Sample Project (plus links to articles)https://github.com/ahus1/logging-and-tracing

.consulting .solutions .partnership

Alexander SchwartzPrincipal IT Consultant

+49 171 [email protected]

@ahus1de

msg systems ag (Headquarters)Robert-Buerkle-Str. 1, 85737 IsmaningGermany

www.msg.group

Date post:	23-May-2020
Category:	Documents
Upload:	others
View:	19 times
Download:	0 times

Logging and Tracing for your Microservices – Log4j ... · .consulting .solutions.partnership...

Documents