© 2 0 2 0 S P L U N K I N C .

Abstraction Protraction Monitoring in a Kubernetes Environment

Dave McAllister

Feb 13th, 2020

© 2 0 2 0 S P L U N K I N C .

Abstraction adds

Infrastructure to your

Monitoring concerns

© 2 0 2 0 S P L U N K I N C .

In the beginning...

The world was a simpler place

• Server

• App(s)

• Logs

Server

Applications

Logs

© 2 0 2 0 S P L U N K I N C .

Then stuff got interesting1

• Microservices

• Orchestration

• containerd

• Massive Scale

• CI/CD

And so on

(1) “May you live in interesting times”

© 2 0 2 0 S P L U N K I N C .

“It’s Turtles all the Way Down”

https://xkcd.com/1416/

TL; DR:

► Abstractions make life “simpler”

► Abstractions make digging harder

© 2 0 2 0 S P L U N K I N C .

Orchestration

• Containers change the way we develop, deploy, and maintain apps

• Orchestration - a tool that automates the deployment, management, scaling, networking, and availability of apps

• Container Orchestration - a tool that automates the deployment, management, scaling, networking, and availability of container-based apps

Kubernetes is container orchestration at large

© 2 0 2 0 S P L U N K I N C .

Tools add Complexity

• Kubernetes - Container Orchestration - Master - Worker Nodes - Pods

• Containers • App • Sidecar

© 2 0 2 0 S P L U N K I N C .

© 2 0 2 0 S P L U N K I N C .

But Monitoring? Logs?

• Early stacks were simple and static

• But orchestration changed that

© 2 0 2 0 S P L U N K I N C .

So What?

• Monitoring needs to change

• Apps, containers, orchestrators

• “So where is it?”

• Logging needs to change

• Ephemeral

• Scaling as needed

Image C

redit t

o G

eek a

nd P

oke b

log:

htt

p:/

/geekandpoke.t

ypepad.c

om

/.a/6

a00d

83

41

d3

df5

53

ef0

128

76

4d

3a

ef9

70c-p

i

© 2 0 2 0 S P L U N K I N C .

So what about K8s?

© 2 0 2 0 S P L U N K I N C .

Monitor each layer explicitly • Collect logs and monitor for

• Errors

• Failures

• Performance issues

• At each layer

• Pod

• Container

• Orchestrator

Kubernetes Control Plane

API Server

etcd

Scheduler

Controller Manager

Kube-State-Metrics

Kublets

Container Metrics

Logs

© 2 0 2 0 S P L U N K I N C .

Monitor your orchestrated environment

• Centralize your logs and metrics

• Your environment is still decentralized

• Getting things aggregated gives you a fighting chance

• Centralize your logs and metrics

• Your environment is still decentralized

• Getting things aggregated gives you a fighting chance

© 2 0 2 0 S P L U N K I N C .

Simplify, simplify, simplify

• It’s complex enough with 17 tools doing different things

• Don’t do this

• Let your monitoring needs drive your tools

• Responsiveness matters

• In debug

• In deployment

• On scale

© 2 0 2 0 S P L U N K I N C .

Data consistency makes life better

• Ensure data consistency across all the layers

• accurate timestamps

• consistent units of measurement (such as milliseconds vs. seconds)

• collecting a common set of metrics and logs across applications and components

© 2 0 2 0 S P L U N K I N C .

Account for ephemeral containers

• Things start and stop all the time

• Grab the logs and metrics before they’re gone

• The thing you don’t see will get you

• Similar cluster and group activities

© 2 0 2 0 S P L U N K I N C .

Monitor without touching the apps

• DaemonSet or sidecar model

• Use a DaemonSet with fluentd

• Use a DaemonSet with Prometheus monitoring

• Use instrumented libraries

© 2 0 2 0 S P L U N K I N C .

• Be clear as crystal

• Keep services as small as necessary

• Make orchestration itself observable

• Find places not to orchestrate

© 2 0 2 0 S P L U N K I N C .

Summary

• Orchestration and containers change the game

• There are new rules and new ways to apply

• Microservices and orchestration go hand in hand

• You can hurt yourself

• Tread lightly

© 2 0 2 0 S P L U N K I N C .

Closing Thoughts

• We pay for ease of use in additional complexity • Emerging technology need attention just like apps • Keep an eye on on the truths in your logs

“A distributed system is one in which the failure of a

computer you didn't even know existed can render

your own computer unusable.”

– Leslie Lamport, 1987

Thank You

© 2 0 2 0 S P L U N K I N C .

davemc@splunk.com

https://www.linkedin.com/in/davemc