1
Modeling the Performance ofUltra-Large-Scale Systems
King Chun Foo, Zhen Ming Jiang, Bram Adams, Ahmed E. Hassan, Ying ZouQueen’s University, Canada
Kim Martin, Parminder FloraResearch In Motion
Using Layered Simulations
2[Slide courtesy of Randy Shoup, eBay Distinguished Architect]
ULS are Everywhere, and they’re HUGE!
4
1. Performance Testing should happen …
developmentprocess
here?
here!
5
2. One Model to Rule Them All?
1. How do response time differ between these two mechanism?
6
3. Different Stakeholders, Different Interests
End User Developer System Engineer
System Integrator
Sales Rep
8
An Analogy: The “4+1” View Model
Scenario
Logical View
Process View
Development View
Physical View
9
The “3+1” View Model for Performance Modeling
Scenario
WorldView
PhysicalView
Component
View
ComponentLayer
A Layered Simulation Model!
10
WorldViewLayer
PhysicalLayer
11
Case Study 1Bottleneck Detection in RSS Cloud
• RSS blogging service providers (e.g., Wordpress)• Evaluation of two notification delivery mechanisms:
– Periodic Pull– Real-time Push
Q1. How does response time differ?
Q2. How does resource usage(e.g., CPU, memory) look like?
ComponentLayer
The RSS Cloud Model
12
WorldViewLayer
PhysicalLayer
13
CPU is the Bottleneck!
14
Case Study 2Evaluating a Distributed Monitor
1. How do response time differ between these two mechanism?
16
Configuration Score
DISCRETIZEfuzzy values
LESS is MORE
17
Distributed vs. Centralized
0.1Hz
0.2Hz
0.3Hz
0.1Hz0.2Hz0.3Hz
faster “cheaper”
lower resource usage
18
Keeping the Model Up-to-Date
• (What-if) Experimenting with different configuration/deployment scenarios
• (Digging deeper) Adding new “layers”, as system is understood better
• (Design evolution) Updating the simulation models as the system evolves
QUESTIONS!
20
21
Backup Slides
FOUR CHALLENGESThe Current Practice of Performance Verification
22
1. Too Late in the Development Lifecycle
• Design changes are not evaluated until after code is written
– Happens at the last stage of a delayed schedule
23
Requirements
Design
Implementation
Verification
Maintenance
2. Lots of Data
• Industrial case studies have 2000> counters
• Time consuming to analyze
• Hard to compare more than 2 tests at once
24
3. No Documented Behavior
25
• Analysts have different perceptions of performance regressions
• Analysis may be influenced by– Analyst’s knowledge– Deadline
4. Heterogeneous Environments
• Multiple labs to parallelize test executions– Hardware and software may differ– Tests from one lab may not be used to analyze
tests from another lab
26
Categorize Each Challenge
27
Design Implementation
ImplementationImplementation
AT THE DESIGN LEVELPerformance Verification
28
Evaluate Design Changes through Performance Modeling
• Analytical models are often not suitable for all stakeholders– Abstract mathematical and statistical concepts
• Simulation models can be implemented with support of existing framework– Visualization– No systematic approach to construct models that
can be used by different stakeholders
29
Layered Simulation Model
30
Physical layerComponent layer
World view layer
Can the current infrastructure support the projected growth of users?
Investigate threading model
Hardware resource utilization
Case Studies
• We conducted two case studies– RSS Cloud
• Show the process of constructed the model• Derive the bottleneck of the application
– Performance monitor for ULS systems• Evaluate whether or not an organization should re-
architect the software
• Our model can be used to extract important information and aid in decision making
31
