Confidential. Internal use only

Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracingGo Language NYC Meetup - January 19, 2017

Yuri Shkuro, Staff Engineer, Uber Technologies

Confidential. Internal use only

Agenda

1. What is distributed tracing

‐ Quick intro

2. Why should you care

‐ Demo

3. Why OpenTracing

‐ Code examples

4. Jaeger @ Uber

Confidential. Internal use only

What Is Distributed Tracing(show of hands)

Distributed Context Propagation

A

B

CD

E

{context}{context}

{context}{context}

Unique ID → {context}

1. Assign a unique identifier to each request at the edge service

2. Store it in a context object, along with other metadata

3. Propagate the context across process boundaries (in-band)

4. Baggage is arbitrary K/V5. Capture timing, events, tags and

collect them out of band (async)6. Re-assemble the call tree from the

storage for the UI

Edge service

Confidential. Internal use only

Why Should You Care?(demo)

https://github.com/uber/jaeger/tree/master/examples/hotrod

Confidential. Internal use only

Recap

1. Discover architecture

2. View request timeline & errors

3. Find sources of latency

4. Highly contextualized logging

5. Baggage propagation

6. Compute usage attribution

Tracing is

WHAT’S THE CATCH?

● Instrumentation is time consuming

● Vendor lock-in (backend and data model)

● Inconsistent semantics across languages

● Not suitable for open source frameworks

Tracing Instrumentation Has Been Too Hard

A vendor-neutral open standard for distributed tracing.

• Addresses instrumentation problem• Open source, no vendor lock-in• Consistent semantics across languages• Available in 7+ programming languages

Who should care:

• Cloud-native / microservice applications• OSS frameworks, esp. IPC and control flow• Tracing and/or monitoring systems

http://opentracing.io

An official project of

OpenTracing

Announced v1.0 spec in August 2016

Tracer Implementations: Zipkin, Jaeger, Hawkular, LightStep, Appdash, and a few smaller tracing systems

Supported Frameworks: GRPC, YARPC, Dropwizard, Flask, Django, Go-kit, ...

Some Companies using OpenTracing:

A young, fast growing project

OpenTracing Approach

TRACER

● StartSpan(operationName) Span

● Inject(SpanContext, Format, Carrier)

● Extract(Format, Carrier) SpanContext

SPAN

● Context() SpanContext

● SetTag(key string, value interface{})

● Log(...)

● Finish()

● SetBaggageItem(key, value string)

● BaggageItem(key string) string

API: Simple, yet expressive

● Off the shelf instrumentation● Start / Finish● Inject / Extract● Tags● Logs● Baggage

Code Examples

Jaeger (yā′gər - hunter)

Host or Container

Application

Instrumentation

OpenTracing API

jaeger-client

jaeger-agent (Go)

jaeger-collector(Go)

memory queue

Data Store(Cassandra)

jaeger-query(Go)

jaeger-ui(React)

Control Flow

TraceReporting

Thrift overTChannel

Control Flow Trace ReportingThrift over UDP

Adaptive Sampling

Confidential. Internal use only

Thanks!

References:

http://uber.github.io/jaeger

http://opentracing.io - @opentracing