Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | bruce-lester |
View: | 218 times |
Download: | 1 times |
Flexible Scheduling in Middleware for Distributed Rate-Based Real-Time
Applications
Christopher D. GillDissertation Supervisors: Dr. Ron K. Cytron, Dr. Douglas C. Schmidt
Department of Computer ScienceWashington University, St. Louis, MO
Tuesday, December 18, 2001
Historical Challenges
Motivation for Studying Adaptive MiddlewareTrends
•Many mission-critical distributed applications require real-time QoS guarantees– e.g., combat systems, online trading, telecom
•Building QoS-enabled applications manually is tedious, error-prone, & expensive
•Conventional middleware does not support real-time QoS requirements effectively
•Building distributed systems is hard•Building them on-time & under budget is even harder
•Hardware keeps getting smaller, faster, & cheaper
1 1Proxy
service
Service
service
AbstractService
service
Client
•Software keeps getting larger, slower, & more expensive
New Challenges
Overview of Research AreasTechnical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Customizable Middleware
Nimble Adaptation
Performance Awareness
Inclusive Systems Approach
These technical challenges raise crucial systems issues in both theoretical and empirical dimensions
These technical challenges raise crucial systems issues in both theoretical and empirical dimensions
Motivating Applications
Boeing Bold Stroke Middleware Infrastructure Platform
•Operations well defined
•Event-mediated middleware solution
•Previous-generation systems static
•Next-generation systems dynamic and adaptive
Domain CompanyAvionics mission computing Boeing, RaytheonMass storage devices SUTMYN, StorTekMedical Information Systems Siemens, GESatellite Control LMCO COMSATTelecommunications Motorola, Lucent, Nortel,
Cisco, SiemensMissile & Radar Systems LMCO Sanders, RaytheonSteel Manufacturing Siemens ATDBeverage Bottling Automation Krones AG
Limitations With Existing Approaches
Historically, each application has solved scheduling on its own
• Tedious, error-prone• Costly over system lifecycles• Single-paradigm approaches
Current middleware lacks hooks for key domain-specific features, e.g.:• Optimized integration with higher level managers• Hybrid static-dynamic scheduling strategies• Strategies built from primitive elements• Adaptive domain-specific optimizations
Research Contributions
Systems Architecture and Framework• Extends open-source middleware
Patterns for Adaptive Scheduling• Capture design experience and solutions
Empirical Evaluation of Adaptive Scheduling• Practical benefits for real-world applications
Research Approach: the Kokyu Flexible Middleware Scheduling/Dispatching Framework
Application specifies characteristics• e.g., criticality, periods, dependencies
Application specifies characteristics• e.g., criticality, periods, dependencies
Dispatcher is (re)configurable• Multiple priority lanes• Queue, thread, timers per lane• Starts repetitive timers once• Looks up lane on each arrival
Dispatcher is (re)configurable• Multiple priority lanes• Queue, thread, timers per lane• Starts repetitive timers once• Looks up lane on each arrival
Scheduler
sub-graph
ratetuples
WCET propagation
selectedrates
rate propagation
propagatedrates
tuplevisitor
operationvisitors Rate and priority
assignment policy
DispatcherDispatching configuration
RMS
LLF
mandatory
optionallaxity
static
static
timers
Scheduler assigns rates & priorities per topology, scheduling policy
• Defines necessary dispatch configuration
Scheduler assigns rates & priorities per topology, scheduling policy
• Defines necessary dispatch configuration
Implicit projection• Of specific scheduling policy into
generic dispatch infrastructure
Implicit projection• Of specific scheduling policy into
generic dispatch infrastructure
Greater Utilization with Criticality Isolation
Technical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware
Nimble Adaptation
Performance Awareness
Inclusive Systems Approach
Safety: Meet Critical Deadlines
Static scheduling cannot isolate critical operation deadlines in overload
Dynamic Static
same point in execution
Solution: Support for Hybridizing Static & Dynamic Scheduling Heuristics
Abstract mapping: operation characteristics OS & middleware primitives
• Generalizes RMS, EDF, LLF, MUF, RMS+LLF, RMS+EDF, …
• Arbitrary composition of primitives• Drives factory-driven dispatching
module (re)configuration at run-time (work in progress)
DispatcherDispatching configuration
laxity
laxity
critical non-critical
MUF
No One Strategy is Optimal
Technical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware Dispatching composed from primitive elements
Supports tailored “fit” of scheduling/dispatching
Nimble Adaptation
Performance Awareness
Inclusive Systems Approach
Case for Multi-Paradigm Scheduling
Dynamic1Dynamic2Static
)
Solution: Compose of Scheduling Heuristics from Dispatching Primitives
Dispatcher
laxity
laxity
mandatory optional
Gives Fine Grain Control over Feasibility / Performance Trade-Off
laxity
staticstatic
static
staticfeasibility
performance
MUF RMS+LLF
Improving the Bound on Adaptive Rescheduling
Technical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware Dispatching composed from primitive elements
Supports tailored “fit” of scheduling/dispatching
Nimble Adaptation Integrated rate/priority selection mechanisms
O(n2)/O(n log n)
O(n log n)/O(n) bound on adaptation
Performance Awareness
Inclusive Systems Approach
Benefits of Resource Adaptation in Middleware
Adaptation Preserves Feasibility in Changing Environment• Skillfully done, it also improves performance
Middleware Offers Portable, Robust Solutions• Can add new managers to a system, E.g., QuO, RTARM
However, a new Manager (e.g., RTARM) may be a Black Box• Determines the number of scheduler calls• Determines the inputs to each call
Need Optimizations to Ensure Nimble Adaptation
Adaptation Time (RTARM as a Black Box)
n = # target region operationsO(log n) or O(n) fits measured time / callNumber of calls is O(n)Time to adapt thus O(n log n) or O(n2)Either way, we can do better
Solution: Integrated Rate & Priority Selection
Skillfully integrate rate and priority mechanisms
At least as good overall: O(log n) per operation: comparison sort
Better in special cases: O(1) per operation: radix sort
Retains Modular Policy while Supporting Optimization
Scheduler
sub-graph
ratetuples
WCET propagation
selectedrates
rate propagation
propagatedrates
tuplevisitor
operationvisitors Rate and priority
assignment policy
Unobtrusive Monitoring and Control FeedbackTechnical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware Dispatching composed from primitive elements
Supports tailored “fit” of scheduling/dispatching
Nimble Adaptation Integrated rate/priority selection mechanisms
O(n2)/O(n log n)
O(n log n)/O(n) bound on adaptation
Performance Awareness Time and space efficient data collection and storage framework
Provides run-time observable info for control, post-analysis
Inclusive Systems Approach
Solution: Kokyu Real-Time Metrics Monitoring Framework
EM
BE
DD
ED
BO
AR
DS
RE
MO
TE
WO
RK
ST
AT
ION
SH
AR
ED
ME
MO
RY
METRICS CACHE
REMOTE LOGGER
METRICS MONITOR
RTARM
DOVEBrowser (Java)
STO
RA
GE
QuO
OPERATIONS
PRO
BE
S
DISPATCHER
PROBES
FRAME MANAGER
Feedback to Higher-LevelResource Managers
Logging and Visualization
Customized Data Collection
Consistent View of Time
Co-Scheduling Control Elements with ApplicationTechnical Challenge
Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware Dispatching composed from primitive elements
Supports tailored “fit” of scheduling/dispatching
Nimble Adaptation Integrated rate/priority selection mechanisms
O(n2)/O(n log n)
O(n log n)/O(n) bound on adaptation
Performance Awareness Time and space efficient data collection and storage framework
Provides run-time observable info for control, post-analysis
Inclusive Systems Approach
Decision lattice joining a priori analysis with empirical measurement
Towards run-time reflective and adaptive policy selection
Inclusive Systems Approach
Dispatcher
low
medium
high
App! App?Control! Control?
Goal: co-scheduling of control elements with the application
Problem: separate treatment of control elements and application
Solution: Use Empirical & A Priori Information to Co-Schedule Resource Mgrs & Applications
Preserve Assurances butOptimize Performance
DispatcherLLF {App!, Control!,?, App?}
RMS {App!, Control!,?, App?}
RMS {App!, Control!,?}LLF {App?l}
RMS {App!, Control!}LLF {Control?, App?}
RMS {App!}LLF {Control!,?, App?}
LLF {App!, Control!,?}LLF {App?}
LLF {App!, Control!}LLF {Control?, App?}
LLF {App!}LLF {Control!,?, App?}
feasibility
performance
feas
ibil
ity
per
form
ance
feas
ibil
ity
per
form
ance
feas
ibil
ity
per
form
ance
feas
ibil
ity
per
form
ance
feas
ibil
ity
per
form
ance
feas
ibil
ity
per
form
ance
feasibility
performance
feasibility
performance
feasibility
performance
Towards an Adaptive Control Decision Lattice
Experimental Test-BedApplication• Research Version of Operational Flight Program for AV-8B Aircraft• Added navigation route computations to ramp non-critical load• Added critical and non-critical computations to inject execution time jitter
Middleware• The ACE ORB (TAO)• TAO Real-Time Event Channel• Kokyu Framework: Scheduling, Dispatching, and Metrics
Operating System and Hardware• VxWorks RTOS on the PPC boards• 200 MHz Motorola PPC 604 card• two 100 MHz Dy4-177 PPC 603 cards• Dy4-783 memory mapped display processor• Commercial VME-64 chassis with all boards• Switched Ethernet, MIL-STD-1553 MUX Bus on one Dy4-177 card
Popular Scheduling Strategies
Rate Monotonic Scheduling (RMS)• Assigns thread priorities by rate• Operations at each priority handled in FIFO order
RMS + Minimum Laxity First (MLF)• Critical operations managed as in RMS• Non-critical operations managed in single lowest priority• Non-critical operations handled in minimum laxity (slack time) order
Maximum Urgency First (MUF)• Thread priority per criticality level• Operations in each priority level handled in laxity order
Experimental Benchmarks
Measure Performance of Heuristics over Execution Time Jitter & Load
• Each numbered operating region is stable• Transitions between operating regions: changes in SRT load, HRT+SRT jitter • Performed using realistic hardware, OS, middleware, OFP application
Region 7 Performance (MUF does Better)
Region 8 Performance (RMS+MLF does Better)
Adaptation is Beneficial: Best Strategy Differs
L ML MHH L ML MHH L ML MHH
Mining Adaptation Clues
Refining Adaptation Clues
Adaptation over Scheduling Heuristics
A Map of the Best Performing Heuristic over Execution Jitter & Load• RMS performs best if system is under-loaded (theory predicts this)• RMS+MLF performs best in overload if jitter is very high or very low• MUF performs best if jitter is moderate
A Basis for Adaptive Control• Run-time observable measure correlates with performance: operation latency• A simple automaton could be constructed
My Contribution: A Unified Middleware Approach
Real QoS problems require both theoretical and empirical perspectives• Scheduling theory generalized over OS/middleware primitives heuristic space• Empirical study of specific heuristic (sub-)spaces is crucial• Analogy: theory/studies of Ethernet behavior: bin-exp-backoff vs. congestion collapse
Technical Challenge Research Approach
ResearchImpact
Efficient and Safe Systems
Arbitrary strategies that hybridize static/dynamic scheduling/dispatching
Increased utilization, critical operations still meet their deadlines
Customizable Middleware Dispatching composed from primitive elements
Supports tailored “fit” of scheduling/dispatching
Nimble Adaptation Integrated rate/priority selection mechanisms
O(n2)/O(n log n) O(n log n)/O(n) bound on adaptation
Performance Awareness Time and space efficient data collection and storage framework
Provides run-time observable info for control, post-analysis
Inclusive Systems Approach
Decision lattice joining a priori analysis with empirical measurement
Towards run-time reflective and adaptive policy selection
Research Impacts and Collaborations
Topics Publications, Systems, & Middleware
Kokyu
Middleware Framework
• Journal of Real-Time Systems, 2001• IEEE Proceedings special issue (submission in progress)• Boeing: ASTD, ASFD, WSOA, Bold Stroke (SEC, MoBIES)• Distinct open-source framework (Kokyu) – early 2002
Integrated Middleware Resource Management
• With Boeing, Honeywell: DASC, 1999• With Boeing, BBN: ICDCS, 2001• With Boeing: DASC, 2001• WORDS 2002• Boeing: ASTD, WSOA, Bold Stroke
Real-Time Metrics & Visualization Infrastructure
• DASC, 1999• DASC, 2000• Boeing: ASFD, WSOA, ASTD II, Bold Stroke (in progress)
Future Research ObjectivesTopics Problems, Approaches, & Investigations
Extremely Small Footprint DRE Firm/Soft/
Middleware
• Dispatch/comm/addressing in micro-niches (downward scalability)• Interesting design tensions between time/space/power/…• “Just enough” middleware: e.g., from Jini-like backbone to an ORB• DARPA ITO NEST Program: OEP middleware
Advanced Techniques for QoS Mechanism Instantiation
• Composing middleware scheduling/dispatching points end-to-end• Heterogeneous: multiple layers and paths• Multi-dimensional resource management: memory, network, CPU• Discover/apply good heuristics and domain-specific optimizations• Empirical/theoretical study/construction of adaptive decision lattices• AOP, domain-specific type systems
Coordinated Multi-Layer Multi-Agent QoS Management
• Integration, cooperation and co-design• Resource managers, schedulers, dispatchers, feature sets• Across hardware, firmware, OS, middleware, application layers• Toward generalized techniques, patterns, and a “complete” theory and practice of QoS composition for real-world systems
• E.g., real-time + anytime + adaptive control + decision-aiding + power-awareness + footprint-awareness + …
Concluding Remarks
Empirical Evaluation• Validates adaptive/hybrid scheduling approach• Quantifies costs/benefits of discrete alternatives• Powerful when combined with theoretical view
–“Mining” technique for problems & properties
Composable Scheduling/Dispatching• Enables domain-specific optimizations, especially when design decisions are aided by empirical data
Heuristic Space Experiments• Offer a quantitative blueprint for adaptation
Open-Source Code• All software described here that is uniquely a part of my research will be made available in the ACE_wrappers distribution
–Kokyu framework (early 2002)–Dispatching for new TAO Event Channel
ThanksMentors•Dr. Douglas C. Schmidt, Dr. David L. Levine, and Dr. Ron K. Cytron
Colleagues and Collaborators•Faculty and Staff of WU CS and CoE•Dr. Douglas Niehaus•Mr. David Sharp, Mr. Bryan Doerr, Mr. Don Winter, Dr. David Corman, Dr. Doug Stuart, Mr. Brian Mendel, Mr. Greg Holtmeyer, Mr. Pat Goertzen, Ms. Jeanna Gossett, Ms. Amy Wright, Mr. Jim Urness, Mr. Tom Venturella, Mr. Russ Wolter
•Mr. Kenneth Littlejohn (AFRL), Dr. Gary Koob (DARPA ITO)•Dr. Rakesh Jha, Mr. John Shackleton, Mr. Nigel Birch•Dr. Joseph Loyall, Dr. Richard Schantz, Dr. John Zinky, Dr. David Bakken•Dr. Ebrahim Moshiri, Mr. Malcolm Spence, Mr. Kevin Stanley
Friends and Family•Members of the DOC Group•WU CS Graduate Students•My wife Barb and son Paul•My Mom Dr. Helen Gill, Dad Mr. David Gill and sister Ms. Sarah E. Gill•My parents and siblings-in-law
Additional Questions ?These slides:
http://www.cs.wustl.edu/~cdgill/cdgill_defense.ppt http://www.cs.wustl.edu/~cdgill/cdgill_defense.pdf
Contact Information:
http://www.cs.wustl.edu/~cdgill
Backup Slides
Number of Calls is Linear
Time in Each Call Appears Linear
Problem: Limitations with Existing Approaches to Time Frame Management
Previously ad hoc• Used different time types
–I.e., gethrtime ( ) vs. gettimeofday ( )
• Fortunately mapped to same clock–Brittle & non-portable solution
• No logical time support end-to-end• No common point of reference for time frames across subsystems
Application
DispatcherUpcall
Adapter
QuO RT-ARM
???
10 32
0 1
01 Hz
2 Hz
4 Hz
Harmonic Frame Rates
Start EndId
Solution: Distinct Manager for Physical & Logical Time Frames
FRAME MANAGER
EMBEDDED BOARD
DISPATCHER
START END IDRATE
0 50 020HZ
0 100 010HZ
0 200 05HZ
50 100 120HZ
0 100 010HZ
0 200 05HZ
100 150 220HZ
100 200 110HZ
0 200 05HZ
150 200 320HZ
100 200 110HZ
0 200 05HZ
200 250 420HZ
200 300 210HZ
200 400 15HZ
250 300 520HZ
200 300 210HZ
200 400 15HZ
300 350 620HZ
300 400 310HZ
200 400 15HZ
350 400 720HZ
300 400 310HZ
200 400 15HZ
400 450 820HZ
400 500 410HZ
400 600 25HZ
ADAPTER RTARM QuO
20HZ45
0
20H
Z45
0 50010H
Z
6005HZ
WSOA:
Singleton Frame Manager• Consistent low-level service
• Register rates with Frame Manager
• Frame start, end, id for each rate
• Timer-based call to update frames
• Dispatcher gets deadline for period–I.e., for laxity, deadline queues
• Upcall adapter gets deadline for cancellation, metrics decisions
• Higher-level resource managers can also use Frame Manager for adaptation decisions
–E.g., RT-ARM, QuO• IDL interface allows remote calls
20HZ
10HZ
5HZ
Problem: Limitations with Existing Adaptive Feedback Collection Approaches
Solutions originally ad hoc• Tedious, error prone, brittle
Lack of support for embedded, RT constraints
• Stringent memory limitations• Narrow windows of harvest time• Shared memory (e.g., VME) is attractive
• However, C++ semantics can cause problems
–Virtual functions–Native pointers–Heap allocation –Address space mapping
SH
AR
ED
ME
MO
RY
METRICS CACHE
SH
AR
ED
ME
MO
RY
METRICS CACHE
Process A Process B
Solution: Metrics Collection Within Embedded & Real-Time Constraints
WSOA:Shared Memory Capability• Offset-based smart pointers, • Strategized allocators• Supports run-time cache sizing and use from any address space
• System instrumented with C++ inline probe macros: write to a singleton metrics cache
Cache is Usable from Process Owned Memory
• Upcall Adapter• Metrics Monitor, Logger DOVE
SH
AR
ED
ME
MO
RY
METRICS MONITOR
METRICS CACHE
DIS
PA
TC
HE
R
20H
z D
eque
ue
20Hz E
nqueue
BEGINSUSPENDRESUME
END
Process Tile
BEGINEND
EMBEDDED BOARDS
UPCALL ADAPTER
OPERATION
BE
GIN
B
SUSP
EN
D
S
RE
SUM
E
R
EN
D
EBEGIN
B
SUSPEND
S
RESUME
R
END
E
B
EN
D
BE
GINE
20Hz EnQ
20Hz DQ
Proc Tile