Date post: | 23-Jan-2015 |
Category: |
Technology |
Upload: | virtual-campus |
View: | 601 times |
Download: | 3 times |
www.s-cube-network.eu
S-Cube Learning Package
Analyzing Business Process Performance Using KPI Dependency Analysis
University of Stuttgart (USTUTT), TU Wien (TUW)
Branimir Wetzstein, USTUTT
Learning Package Categorization
S-Cube
Adaptable Coordinated Service Compositions
Adaptable and QoS-aware Service Compositions
Analyzing Business Process Performance Using KPI Dependency Analysis
Learning Package Overview
Problem Description
KPI Dependency Analysis
Discussion
Conclusions
Let’s Consider a Scenario (1)
Assume we have implemented a business process as a
service orchestration
It is a reseller process which interacts with external services
of customer, suppliers, bank, shipper, and internal services
such as the warehouse etc.
Let’s Consider a Scenario (2)
We are interested in measuring the performance of the
business processes (time, cost, quality, customer satisfaction)
This is done by defining Key Performance Indicators (KPIs)
which specify target values on key metrics based on business
goals
– KPI target value function maps metric value ranges to KPI classes
(e.g., “good”, “medium”, “bad”)
Some typical KPI metrics in our scenario
– Order Fulfillment Lead Time
– Perfect Order Fulfillment (in time and in full)
– Customer Complaint Rate
– Availability of the reseller service
Let’s Consider a Scenario (3)
In the first step, KPIs are monitored at process runtime for a
set of process instances (what?)
If monitoring of KPIs shows unsatisfying results, we want to
be able to analyze and explain the violations (why?)
That is not trivial as a KPI often depends on many influential
factors measured by lower-level metrics
Purchase Order Process
Order
Fulfillment
Lead Time
PPMs
Avail.in Stock,
Customer,
Products, …
QoS
Service
Availability,
Response Time
is measured by
Learning Package Overview
Problem Description
KPI Dependency Analysis
Discussion
Conclusions
Architectural Overview (1)
Architectural Overview (2)
Model and deploy the business process (e.g., in WS-BPEL )
Define and monitor a set of KPIs and potential influential
metrics
– Event-based monitoring based on CEP
– Supporting in particular both process events and QoS events and their
correlation
Train a decision tree (KPI Dependency Tree) from monitored
data
– Gather monitored data from Metrics DB
– Classify the monitored process instances according to their KPI class
– Use Decision Tree Learning Algorithms to learn the dependencies of
the KPI and the lower-level metrics
Background: Event-Based Monitoring
In order to be used for analysis, runtime data needs to be
monitored
Event-based monitoring is an often-used idea to implement
this
Basic principle:
Register for and receive some lifecycle events from the service
composition and use Complex Event Processing (CEP) to extract,
correlate and aggregate monitoring data from raw event data
Can be used to monitor both QoS and domain-specific data
Background: Monitoring of Service Orchestrations
Our monitoring approach for service orchestrations supports:
– Process Performance Metrics (PPMs) based on process events (BPEL event model)
– QoS metrics based on QoS events provided by QoS monitors
– Correlation of Process events and QoS events
– Metric calculation based on Complex Event Processing (ESPER)
Process Engine
Lis
tener
Dashboard
QoS Monitor
Complex Event
Processing
Metric
definitions
Metrics
Database
Service
Event
Event
KPI Dependency Analysis - Motivation
So far we are able to monitor metrics and find out which KPI targets are
violated (what?)
In the next step, we want to explain the violations (why?)
That is not trivial as a KPI often depends on many influential factors
measured by lower-level metrics
Typically, such an analysis is done manually (if at all) by a business
analyst using OLAP queries on a data warehouse
that is very cumbersome and time-consuming
we want to “discover” the problems in an automated way
therefore we can use data mining techniques
In particular, we construct a classification problem and use existing
classification learning techniques (decision trees)
Background: Machine Learning and Data Mining
Automated discovery of interesting patterns from large
amounts of data (stored typically in data warehouses)
– Manual discovery (e.g., by using OLAP queries) could take days or
weeks
Functionalities include:
– Mining of association rules, correlation analysis
– Classification and Prediction our focus here!
– Clustering
– Time-series analysis
– Graph mining and text mining
Interdisciplinary field using techniques from machine learning,
statistics, pattern recognition, data visualization
Background: Classification Learning (1)
Given: a (historical) dataset containing a set of instances described in
terms of:
– A set of explanatory (a.k.a. predictive) categorical or numerical attributes
– a categorical target attribute (a.k.a. class)
Goal: based on the historical dataset (“supervised learning”) create a
classification model which helps…
– explaining the dependencies between the class and the explanatory attributes
in history data (interpretation)
– making predictions about future data; i.e. based on future explanatory attribute
values predict the class (prediction)
Some Classification Learning techniques:
– Decision Trees
– Classification Rules
– Support Vector Machines
Background: Classification Learning (2)
Training Phase
Test Phase
Classification
Algorithm
Training Data
Test Data
New Data
Classification
Model
Prediction Phase Interpretation Phase
Predicted Class
Explanatory
Attribute
Values
Knowledge
Background: Decision Tree Learning
A1
< 2 > 4
2 < x <4
yes no yes no
< 3 > 3 < 1 > 1
C2
20/1
C1
50
C3
20
C3
10/1
C4
80/2
C1
30
C1
5
A2 A2
A4 A3
A non-leaf node represents
an explanatory categorical
or numeric attribute;
Outgoing edges represent
conditions on the parent
explanatory attribute values
A leaf node represents a
target attribute class .
A path shows which
attribute values lead to a
certain class. The leaf node
shows the corresponding
number of instances from
the training set.
KPI Dependency Analysis
The KPI class of a process instance (alt. choreography instance, activity
instance, business object, …) depends on a set of influential factors
(PPMs and QoS metrics)
For finding out those dependencies, we use classification learning:
– The data set consists of a set of (historical) process instances; for each
process instance the KPI class and a set of metrics is evaluated
– The KPI is the target attribute which maps values of the underlying metric to
categorical values (KPI classes)
– The potential influential lower-level metrics are the explanatory attributes
(predictive variables)
– Goal: Based on a set of monitored instances, create a classification model
(decision tree) which identifies recurring relationships among the explanatory
attributes which describe the instances belonging to the same KPI class
The decision tree (KPI dependency tree) can be used to explain KPI
classes of past process instances and also to predict the class of process
instances for which only the values of some of the lower-level metrics are
are known
Defining KPIs
The KPI Definition includes:
– KPI metric definition (e.g., order fulfillment time)
– A set of categorical values defining the KPI classes, at least 2 (e.g.,
“green”, “yellow”, “red”)
– Target value function mapping KPI metric values to KPI classes (e.g.,
m < 2 days green, 2 days < m < 4 days yellow, otherwise red)
The KPI metric is specified for a monitored entity type:
– Process Instance (e.g., duration of a reseller process instance)
– Activity Instance (e.g., duration of the supplier service invocation)
– Choreography Instance (e.g., duration
– Service endpoint (e.g., availability)
– Set of Process Instances per day (e.g., average duration)
Generating Metric Definitions
A set of metric definitions (representing potential influential factors) can be
automatically generated
We support rules to generate automatically the following metrics:
– Service invocation:
- availability and response time of invoked service (both for synchronous
and asynchronous invocations (invoke-receive))
– WS-BPEL invoke activity (other basic activities not interesting for long-running
processes):
- execution time of the activity (i.e. the time between starting and finishing
the activity)
- If part of a loop, in addition:
- Average/minimum/maximum execution time per process instance
- number of executions per process instance
– For every branching activity, fault, compensation and event handlers, we
generate a metric representing the branch that has been executed
Metrics based on process variable data elements are created manually
Data Preparation and Learning
Create a KPI Analysis Model
– Select KPI + a set of potential influential factors
Gather metric values of monitored entity instances and create a training
set:
– Each monitored entity instance with ist KPI class and influential metric values
maps to a row in the training set
A decision tree is learned (e.g., using the J48 algorithm)
KPI Dependency Analysis
KPI Analysis Model
Metric: Order Fulfillment Lead Time
Analyzed Time window: last 2 months
Instances: Filter: customerType=“gold“
Target < 5 days green
Value: >= 5days red
Metric Set: M={orchestration.all, qos.all}
Algorithm: Classification Tree -J48
Monitor Model
Choreo. Order Fulfillment Lead Time,
Level: Delivery Time Shipment,…
Orch. Order In Stock, Delivery Time
Level: Supplier, Order amount,
Packaging time…
KPI
Supplier
Deliv. Time
Order
In Stock
Process
Availability …
Red 28 h No 1,00 …
Green N/A yes 0,84 …
Red 32 h
No
0,9
…
……. ……. ……. ……. …
Delivery Time
Shipment
Delivery Time
Supplier
Order In
Stock?
Order In
Stock?
Delivery Time
Supplier
< 2 > 4 2 < x <4
yes no yes no
< 3 > 3 < 1 > 1
green
20/1
green
50
red
20
red
10
red
80/2
green
30
green
5
Design time
Runtime
Service Process Infr. Av., response time
Level: banking service, …
Designate a
Metric as KPI
Metric
values
Decision
Tree
Learning
KPI
definition
Prototype Implementation
Prototype is based on…
Apache ODE (BPEL execution
engine)
– Publishes events to JMS topics
– Standalone QoS monitor evaluates
QoS metrics of services
Monitoring Tool
– Based on ESPER CEP Framework
– Metrics DB in MySQL
– Bam Dashboard as Java Swing
Application
Process Analyzer
– Uses WEKA Machine Learning
toolkit
Learning Package Overview
Problem Description
KPI Dependency Analysis
Discussion
Conclusions
© Philipp Leitner
Experimental Results
Generated tree for KPI = order fulfillment time (J48 algorithm)
Contains the expected influential metrics in a satisfactory manner and
produce suitable results ‘out of the box’
In our setting on a standard laptop computer a decision tree generation
based on 1000 instances takes about 30 sec
Experimental Results: Drill-Down
Generated tree for KPI = “order in stock” (J48 algorithm)
Here, we perform “drill-down” analysis by setting the metric “order in stock”
as KPI
We want to understand which factors have an influence on whether the
order cannot be processed from stock
Experiment Results: Differences between Algorithms
We have experimented with J48 and ADTree and generated trees for
different numbers of process instances (100, 400, 1000)
– ADTree algorithm produces bigger trees than J48 (third column: number of leaves and
nodes) for the same number of instances. However, it also reaches a higher precision
(last column: correctly classified instances).
– Both algorithms show very similar results concerning the displayed influential metrics.
Typically there is only one or at most two (marginal) metrics which differ.
Experiment Results: Tree Size
Trees are getting bigger with the number of process instances
– J48 generated for 400 instances a tree with 11 nodes, for 1000
instances a tree with 18 nodes, while the precision improved only by
1%
– When the tree gets bigger, factors are shown in the tree which have
only marginal influence and thus make the tree less readable
(‘Displayed Metrics’ shows how many distinct metrics are displayed in
the tree)
Experiment Results: Tree Size (2)
To improve the readability…
– usage of parameters has lead to only marginal changes in our
experiments (for example, J48 -U with no pruning). The only
parameter that turned out useful to reduce the size of the tree was
‘reduced error pruning’ (J48 -R)
– Another option, in the case of too many undesirable (marginal)
metrics, is to simply remove those metrics from the potential influential
factor metric set and repeat the analysis
Some Important Earlier Work
We were not the first ones to have similar ideas
Important earlier work includes:
Castellanos, M., et al., 2005. iBOM: a platform for intelligent business operation management. In: Proceedings of the 21st international conference on data engineering (ICDE005). Washington, DC: IEEE Computer Society, 1084–1095
M. Castellanos, F. Casati, U. Dayal, and M.-C. Shan, “A Comprehensive and Automated Approach to Intelligent Business Processes Execution Analysis,” Distributed and Parallel Databases, vol. 16, no. 3, pp. 239–273, 2004.
Main Advances Over Earlier Work
The S-Cube approach to KPI analysis based on event logs
improves on earlier work in some important aspects:
– KPI Dependency Analysis incorporates both process-level metrics and
QoS metrics
– Semi-automated generation of potential influential metric definitions for
WS-BPEL processes
– Many different algorithms can be used for analysis
- Courtesy of the WEKA backend
Discussion - Advantages
The KPI Dependency Analysis based on decision trees has a
number of clear advantages …
– Simplicity – the principle approach is relatively easy to understand; the
generated trees can be understood also by non-IT users
– Efficiency – the analysis of influential factors is “automated”; the
traditional approach is to manually pose analysis questions by using
OLAP queries over data marts which is much more time-consuming
– Proven in the real world – machine learning is by now a proven
technique that has been successfully applied in many areas
Discussion - Disadvantages
… but of course the approach also has some disadvantages.
– Bootstrapping problem – the approach assumes that some recorded
historical event logs are available for training
– Necessary domain knowledge – in order to define the potential
influential metric set some domain knowledge is necessary
– Availability of monitoring data – one of the basic assumptions of the
approach is that all necessary data can be monitored (if this is not the
case the approach cannot be used)
Learning Package Overview
Problem Description
KPI Dependency Analysis
Discussion
Conclusions
© Philipp Leitner
Summary
Classification learning based techniques can be used to
explain performance problems in service compositions
Steps:
1. Define a KPI and a set of potential influential metrics
2. Monitor all metrics for a set of process instances
3. Train a decision tree from historical event log
The created KPI dependency tree explains the dependencies
of the KPI classes and a set of lower level process metrics
and QoS metrics
Further S-Cube Reading
Wetzstein, Leitner, Rosenberg, Brandic, Dustdar, and Leymann. Monitoring and Analyzing Influential Factors of Business Process Performance. In Proceedings of the 13th IEEE international conference on Enterprise Distributed Object Computing (EDOC'09). IEEE Press, Piscataway, NJ, USA, 118-127.
Wetzstein, Branimir; Leitner, Philipp; Rosenberg, Florian; Dustdar, Schahram; Leymann, Frank: Identifying Influential Factors of Business Process Performance Using Dependency Analysis. In: Enterprise Information Systems. Vol. 5(1), Taylor & Francis, 2010.
Kazhamiakin, Raman; Wetzstein, Branimir; Karastoyanova, Dimka; Pistore, Marco; Leymann, Frank: Adaptation of Service-Based Applications Based on Process Quality Factor Analysis. In: Proceedings of the 2nd Workshop on Monitoring, Adaptation and Beyond (MONA+), co-located with ICSOC/ServiceWave 2009.
Leitner, Wetzstein, Rosenberg, Michlmayr, Dustdar, and Leymann. Runtime Prediction of Service Level Agreement Violations for Composite Services. In Proceedings of the 2009 International conference on Service-Oriented Computing (ICSOC/ServiceWave'09), Springer-Verlag, Berlin, Heidelberg, 176-186.
Leitner, Michlmayr, Rosenberg, and Dustdar. Monitoring, Prediction and Prevention of SLA Violations in Composite Services. In Proceedings of the 2010 IEEE International Conference on Web Services (ICWS '10). IEEE Computer Society, Washington, DC, USA, 369-376.
Acknowledgements
The research leading to these results has
received funding from the European
Community’s Seventh Framework
Programme [FP7/2007-2013] under grant
agreement 215483 (S-Cube).
© Philipp Leitner