Post on 27-Jul-2015
transcript
Copyri
ght
Life of an Event
A never ending tool chain
26-03-2015
Arnold van Wijnbergen
Copyri
ght
2
Arnold van Wijnbergen@BSMConsultant / Arnold.van.Wijnbergen@gmail.com
•15 years working experience
with automating IT
•DevOps Evangelist with
great passion about
Architecture and
Automation Tooling
•Full time Tooling Geek
Automation
Monitoring
Event management
Deployment
#IlovIT ☁
Copyri
ght
3
AND WHO ARE YOU ?
Dev, Ops or just DevOps
Copyri
ght
4
Do you recognize this situation?
Organization is having monitoring but
always missing the correct data and
triggers.
Outage top 10 week 12
Copyri
ght
5
REALITY CHECK
IT is missing some crucial
parts for monitoring
Copyri
ght
6
Most monitoring is reactive and based
on what we know
Monitoring must be proactive and
based on what we see as unusual
behavior
Copyri
ght
7
Here the story begins …
This requires a different approach,
which fits the DevOps culture
Copyri
ght
8
Next statements will place the
context to DevOps
By applying the three Ways
/Underpinning Principles of DevOps
(Gene Kim)
• Systems Thinking
• Amplify Feedback Loops
• Culture of Continual
Experimentation And Learning
Copyri
ght
9
ITOPS IS MISSING CONTROL
BUT
THIS IS A DEV&OPS CHALLENGE
Statement 1
Copyri
ght
10
Statement 2
DEV BUILDS BUSINESS VALUE
OPS MUST SUPPORT BUSINESS VALUE
Development started with Agile
Scrum approach.
DevOps will fill in these gaps for
Operations with use of Continuous
Monitoring to support the
Continuous Delivery pipeline
Copyri
ght
11
BUT
Statement 3
DEV LIKE MANY EVENTS
OPS LIKE FEW EVENTS
We need best of both worlds
Copyri
ght
12
BUT
Statement 4
DEV DOES TESTING
OPS DOES MONITORING
Align both disciplines
Close your feedback loop here
Reuse code for Quality
Assurance and Continuous
Improvement…
Copyri
ght
13
BUT
Statement 5
All resources within your solution
are involved. Operations must be
able to pinpoint the outage,
understand and improve the whole
solution to increase resilience
Monitoring is NOT only about
infrastructure and solving the problem
Copyri
ght
14
Now that we know the
DevOps part of the story
How should we fill in the
Monitoring part of the story ?
Copyri
ght
15
Application
Infra
structure
Business
Activity
Transaction
Data
Application
Runtime
Middleware
OS
Virtualization
Server
Storage
Network
Automation glue
Monitoring all Resources …
• Their characteristics and
variables need to be measured
as an indicator to build up an
expected health state.
• Send out events, which give you
specific information about a
resource, such as availability
and software faults.
• Generate metric data, which
give you time-series analyse on
a resource their performance.
But individual resources don’t
give you a reliable overview of
your solution/enterprise state
!!!
Customer SolutionFirst the WHAT ?
Copyri
ght
16
Just like a starry sky all events (like stars) are inter-related
to each other like (the constellation)
And sometimes a star falls down, which breaks the chain
Copyri
ght
17
Now that we talked you through Space
we will take the last steps how to
implement Continuous Monitoring
Copyri
ght
18
Introduce a central event
handler which …
• Will become your big data
lake
• Will contain events and
alerts ( actionable events )
• Is used for dash boarding
• Trend analyses over time
• Application logging should
contain ARM points or
UUID for traceability
• Can escalate events to
alerts over time by
complex correlation.
• Should feed your IT chain
dashboards for deeper
chain analytics.
• At least 30 days of raw
operational data
First step – Centralize your event data
Data Lake
Infra
Monitoring
User
Experience
Monitoring
Application
Monitoring
Copyri
ght
19
IT Value Chain
Business Application Business Application
Load balancer Load balancer
CMS
CMDB
Populate chain
&
Application dependencies
Employee is boarding a new customer
Confirmation
by post fails
SYMPTOM
SYMPTOM
SYMPTOM
Print queue
offline
CAUSE
Get to know your IT
Value Chains by …
• Use CMS tools for
discovering your IT
landscape with their
dependencies and to
keep configuration
management up-to-
date.
• Giving the DevOps
teams the possibility
to see the business
impact, track down
all specific events,
deepdive and identify
the cause event and
other symptom
events within one or
more related
business
applications.
Second step – Introduce IT Chain Monitoring
Copyri
ght
20
Introduce a analytics system
to ….
• Analyze all your
monitoring historical data,
like events, metrics and
health states for the
machine learning
algorithms.
• These pre-processing will
find patterns to
automatically produce a
model for post processing
on production data
streams.
• After one of the patterns is
matched a prediction by
an early warning alert will
send out.
Third step – Put Analytics on your monitoring
data
Predictive Analytics Engine
EventsMetricsHistorical
Data sources
Prediction
Pre-processing Patterns
Post-processingModel
Produce model
Algorithms
Input
Output
Copyri
ght
21
Fourth step – Review your feedback loop for your implemented
Continuous Monitoring toolchain
IT FactoryIT Delivery
Production
environment
SYSTEM Actor
End users
PEOPLE Actor
IT Management
Support desk
PEOPLE Actor
IT Management Tool chain
SYSTEM Actor
DevOps Engineers
PEOPLE Actor
Delivery Pipeline
Copyri
ght
22
The Life of an Event
CONCLUSION
EVERY EVENT HAS A PURPOSE FOR
EXISTENCE AND DURING HIS
LIFETIME MULTIPLE INTER-
RELATIONS WILL EXIST. THIS
BEHAVIOUR WILL HELP US TO LEARN
AND DETECT PATTERNS
AT LAST BE PREDICTIVE FOR THE
FUTURE
Copyri
ght
23