Date post: | 21-Jan-2018 |
Category: |
Engineering |
Upload: | dave-mangot |
View: | 108 times |
Download: | 0 times |
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Observability in DevOps with Modern Distributed Applications
Dave Mangot Head of Site Reliability Engineering, SolarWinds Cloud @davemangot
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
About Me
• 20+ years in systems engineering, have led DevOps and operational maturity transformations at multiple companies
• Big companies: Cable & Wireless, Salesforce, SolarWinds
• Small Companies: Terracotta, Tagged, Librato
• International DevOps contributor
@davemangot
5/9/16 2
© 2017 SolarWinds Worldwide, LLC. All rights reserved.
SolarWinds company overview: Quick facts
• Founded in 1999, over 18 years in the industry
• Over 250,000 customers in 170 countries, including 425 of the Fortune 500®
• Global headquarters in Austin, Texas, with 2,300 employees worldwide
• SolarWinds was taken private by private equity investment firms Thoma Bravo and Silver Lake Partners in February 2016
3
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.4
but we wouldn’t be upset if you did :)
You do not need to purchase these solutions from SolarWinds!
Disclaimer:
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 5
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 6
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 7
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 8
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 9
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 10
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 11
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 12
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 13
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
DevOps
14
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
DevOps
15
DevOps is a collaboration between Development, Operations and other teams with the recognition that we are tasked with achieving common business goals.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 3 Ways of DevOps
16http://itrevolution.com/the-three-ways-principles-underpinning-devops/
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 3 Ways of DevOps
17
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Feedback
18
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
2nd Way of DevOps
The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made. The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
19http://itrevolution.com/the-three-ways-principles-underpinning-devops/
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.20
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.21
If it’s not monitored, it doesn’t exist
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Uptime Monitoring
22
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Uptime Monitoring
23
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 24
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 25
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 26
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 27
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
https://blog.gigaspaces.com/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/
28
Latency matters. Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.29
You can’t improve what you don’t measure
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Observability
30https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.31
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging (events)
32
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging
33
tail -f /var/log/messages
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 34
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 35
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Application Development/Troubleshooting
36
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Setup
37
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Logging
38
• Application Development
• Troubleshooting
• Deployments
• Search Alerts
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Shipping
39
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Search Alerts
40
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Papertrail
41
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 42
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 43
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Metrics (aggregatable)
44
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Infrastructure Metrics
45
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Host Metrics
46
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.47
root 23706 1 0 Nov18 ? 00:00:00 /bin/sh -c start-stop-daemon --start --quiet --pidfile /var/run/appoptics/appoptics-snapteld.pid —chuid appoptics --exec /opt/appoptics/sbin/snapteld --oknodo --config /opt/appoptics/etc/config.yaml --pidfile /var/run/appoptics/appoptics-snapteld.pid
Host Metrics
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Host Metrics
48
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Application Metrics
49
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Kafka
50
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Custom Metrics
51
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Metric Submission
52
api = librato.connect('email', 'token')
q = api.new_queue()
q.add('temperature', 22.1, tags={'location': 'downstairs'})
q.add('temperature', 23.1, tags={'location': 'upstairs'})
q.submit()
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Custom Metrics
53
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
54
● Latency
● Traffic
● Errors
● Saturation
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
55
Latency
The time it takes to service a request.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
56
Traffic
How much demand is being placed on your system.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
57
Errors
The rate of requests that fail either explicitly, implicitly, or by policy.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
The 4 Golden Signals
58
Saturation
How “full” your service is.
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Tracing (request scoped)
59
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 60
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
61
Web
App
Database
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
62
Web
App
Database
X-Trace: 123654
X-Trace: 123654
X-Trace: 123654
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 63
WHAT IS DISTRIBUTED TRACING?
HELP!No, help ME!
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
export APPOPTICS_SERVICE_KEY="795fXXXXXXXe5f5b17c7:serviceA"
-javaagent:/usr/local/lib/appoptics-agent.jar
64
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
65
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
66
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Distributed Tracing
67
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.68
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED. 69
© 2017 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Thanks!
70
@davemangot