Date post: | 30-Mar-2015 |
Category: |
Documents |
Upload: | dandre-cleeton |
View: | 215 times |
Download: | 0 times |
Automatic Configuration of Internet Services
Wei Zheng, Ricardo Bianchini, and Thu Nguyen
Department of Computer ScienceRutgers University
Motivation Downtime and management are expensive for services
• Downtime can cost millions of dollars per hour [IW00]
• Management can account for 50—80% of IT budgets [Brown05]
Operator mistakes are a major source of downtime
• Cause up to 36% of service failures [Oppenheimer03]
Misconfiguration is the most common type of mistake
• More than 50% of service failures [Oppenheimer03]
• 24 out of 42 mistakes in a multi-tier service [Nagaraja04]
Understanding Misconfigurations Misconfigurations occur frequently for two main
reasons:
• Servers and services are becoming increasingly complex
• Operators modify servers’ configs often, as service evolves via software and hardware upgrades, node additions and removals
Example: Apache
• One config file has 240+ lines relating to performance, structure, modules, and others
• Config files must change every time a node is added or removed in the 2nd tier
Our Work Idea: Automatically generate server config files as
the system and its workload evolve
Goals: Reduce config complexity and operator effort, and generate optimized config files efficiently
Contributions:
• Infrastructure for automatic configuration of services
• Heuristics for efficient parameter tuning
• Quantitative evaluation for a realistic service
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
Key Observations Behind ACI Most of each server’s config doesn’t change
Node additions and removals require tedious and mistake-prone reconfigurations
Changes to a config parameter affect only a few other parameters
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
Changes to a config parameter affect only a few other parameters
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
• Network of membership daemons
Changes to a config parameter affect only a few other parameters
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
• Network of membership daemons
Changes to a config parameter affect only a few other parameters
• Parameter dependency graph and parameter-tuning heuristics
Global View of ACI (3-tier Service)
Local View of ACI
Daemon
Scripts + templates
Config
files
Node i
Global membership info + tuned parameter values
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
Tuning Parameters Tune parameters for some metric, as service
evolves
Evolution may cause current config to behave poorly• Ex: when throughput is the metric, more memory may
allow more threads and higher throughput
Problem• Hard to know what parameters are affected• Hard to select new values for the parameters
Parameter Dependency Graph Idea: Constrain the search space with a parameter
dependency graph and value ranges
Vertex = parameter, directed edge = dependency
After each change, ACI traverses the graph to find the affected parameters and tune them experimentally• Ex: if Apache node is upgraded, make Apache
parameters “sources” and tune reachable parameters
Overhead of generating graph can be amortized
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
A depends on B
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
A depends on B B doesn’t depend on A
Generating the Graph (cont.) An evolution may change the bottleneck tier
Dependencies may be affected by provisioning of tiers• Ex: if throughput is the metric, different bottleneck tiers
may lead to different dependencies
Run dependency checking once per tier, each time forcing the tier to become the bottleneck
Dependency graph is the union of these results
Tuning = Traversing the Graph In a cycle, tune all member vertices at the same
time
In an acyclic chain, tune parents first assuming best values for ancestors and current values for descendants
A singleton is tuned in isolation
Select values using Simplex algorithm and value sets
# experiments increases with connectivity, value set size
Summary
Find important parameters
Find dependencies
Tune parameters
Normal execution
Evolution
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
Evaluation: Performance Tuning 3-tier auction (Apache, Tomcat, MySQL) on 9
nodes; 24 performance parameters
Management complexity and mistake-elimination results in the paper
Efficiency: Number of throughput-tuning experiments after DB machine upgrade, 2 application server removal
Online Auction Dependencies
Performance Tuning Approaches Exhaustive: O(RN) runs; R = parameter range and
N = # parameters = 24; infeasible
Dependency graph + exhaustive: N = 9; infeasible
Simplex
ACI = Dependency graph + Simplex
Performance Tuning Efficiency
ACI needs 2 service changes to amortize the graph overhead
EvolutionTuning Approach
Throughput (reqs/sec)
Number of Experiments
DB node upgrade
Best prior values
343 -
Simplex search
372 299
ACI 377 821 + 73
Removal of 2 application
servers
Best prior values
91 -
Simplex search
114 220
ACI 110 0 + 35
Amortizing the Graph Overhead Does dependency graph change when service
evolves?
DB node upgrade• Graph does not change
Removal of 2 application server nodes• Graph does not change
Upgrade MySQL server version• Graph is a subset of original one
Conclusions Proposed automatic configuration infrastructure
Proposed notion of parameter dependency graph
Quantified benefits experimentally for realistic service
ACI reduces config complexity, eliminates misconfigs, and produces high-performance services efficiently
Questions?
http://vivo.cs.rutgers.edu/