Towards Self-Tuning Databases
Sang-Won Lee([email protected])
SKKU VLDB Lab.( http://oralab.skku.ac.kr/~oralab )
Nov. 1, 2002
SKKU VLDB Lab.Toward Self-Tuning Databases
2
Contents
• Introduction
- Why DB tuning?
• Current Database Tuning Paradigm
• Toward Self-Tuning Databases
SKKU VLDB Lab.Toward Self-Tuning Databases
3
What is DB Tuning?
• Car tuning- Goals? - What? - How?
SKKU VLDB Lab.Toward Self-Tuning Databases
4
DB = IT Core Platform in the 21st Century?
• ERP/CRM/SEM/BSC
• DW/OLAP/Data Mining
• Web Log Analysis
• GIS/XML/Mobile
• Bio-informatics
• ....
• Ubiquitous Databases
SKKU VLDB Lab.Toward Self-Tuning Databases
5
Why DB Tuning in the 21st Century?
• DB performance influences all aspects of your business- DB performance management is one of the most important
success factors in almost every IT projects- “ 미래 웹경쟁력은 데이터처리능력에” (Scott McNealy, Sun
CEO, 전자신문 )
• Database tuning is not a technical issue!
• Database tuning is a business issue!
DB tuning can improve your productivity
DB tuning can increase your revenue/profit
DB tuning can increase your ROI(Return on Investment)
SKKU VLDB Lab.Toward Self-Tuning Databases
6
Why DB Tuning in the 21st Century?(2)
• More data- doubling every 9 month
• More users- from CEO to sales representatives
• More complex query- e.g. OLAP/data mining query
• Faster responses- e.g. web response time impacts revenue!- e.g. real time personalization
Without DB tuning, you need double-up investment on DB server and software every 6 month!
SKKU VLDB Lab.Toward Self-Tuning Databases
7
Why DB Tuning in the 21st Century?(3)
• Data growth vs. computer speedup
** Moore’s Law -- # of transistors/chip doubles every 18 months (1965)
0
500
1000
1500
2000
2500
3000
3500
Year
Pe
tab
yte
s
DiskSalesMoore'sLaw
SKKU VLDB Lab.Toward Self-Tuning Databases
8
Why DB Tuning in the 21st Century?(4)
• Disk sales doubling every 9 months - Greg Papadopoulos, Sun CTO- similar results from Winter VLDB survey (http://www.wintercorp.c
om)
• Time to process all your data doubles every 18 months!- You need DPA, not DBA!
- Database Performance Administrators- BUT! they are extremely expensive!!
• Maximize your DB performance!! (DBMax)Maximize your PRODUCTIVITYMaximize your REVENUE/PROFITMaximize your ROI
SKKU VLDB Lab.Toward Self-Tuning Databases
9
Tuning Activities for Databases
Systems (H/W, O/S)Systems (H/W, O/S)
Oracle ServerOracle Server DatabaseDatabaseApplicationsApplications
SQLSQL
Oracle Database Architecture
Components for Tuning Check Lists for Tuning Criterial for Tuning
ApplicationsLogic / Server Connection / APPS Config etc Response Time, CPU usage
SQLAccess Path / Join Order / Join Methods / Parallelization / Efficient Parsing etc. SQL Tuning
Elapsed Time, Plan, Trace, Performance Statistics
Oracle Database ServerSGA Size / Server Configuration / Initial Paramaters Setting Performance Statistics, Waiting
Event, CPU usage
Logical & Physical Database Design
Physical DB Design / IO Balancing / De-Fragmentation / Removing Chaining etc.
Performance Statistics, Waiting Event, Disk I/O
Systems (H/W, O/S)Addign CPUs / Kernel Parameter Setting / RAID Config. etc CPU Usage, Memory Usage,
Disk I/O Traffic
SKKU VLDB Lab.Toward Self-Tuning Databases
10
Current DB Tuning Paradigm
1. (Logical) DB design DB Modeller
2. H/W set-up and configuration HW experts
3. Application development and test Developers
4. System open DPA
5. System operation
6. Data increases
7. Unpredicted performance problems DPA
MonitoringDrill-down
to BottleneckRoot Cause
AnalysisTuning
SKKU VLDB Lab.Toward Self-Tuning Databases
11
Why DB Monitoring in the 21st Century?
• DB tuning vs. DB monitoring
- DB Monitoring DB Tuning
- E.g.: Tuning 10 hour SQL into 10 minute SQL is not any more special skill
• Why DB monitoring is so difficult in the 21st century?
- e.g. ERP, DW/OLAP, CRM etc
- packaged and tool-based
- DB schema is too complex
- SQL is unknown
- data size is unpredictable
- .....
SKKU VLDB Lab.Toward Self-Tuning Databases
12
Why DB Monitoring in the 21st Century?(2)
• Usually, thousands of SQL statements are running concurrently in a database
• Cho & Lee’s rule
- “When database performance is low, less than 5% SQLs are using more than 95% database resource”
• A key issue in DB monitoring
- To rapidly identify the problematic SQL statements!
- Without tool, this task is extremely difficult
SKKU VLDB Lab.Toward Self-Tuning Databases
13
Why Self-Tuning Databases?
• War of TCO(Total Cost of Ownership) in 21st century
- Major costs of IT infra. ownership
- H/W, S/W, Human Cost
- Moore’s law: H/W, S/W cost ↓
- BUT! Human cost ↑
- cf. 1950s: telephone industry
- manual switching by operators automatic switching system
SKKU VLDB Lab.Toward Self-Tuning Databases
14
Why Self-Tuning Databases?(2)
SKKU VLDB Lab.Toward Self-Tuning Databases
15
Why Self-Tuning Databases?(3)
SKKU VLDB Lab.Toward Self-Tuning Databases
16
Why Self-Tuning Databases?(4)
• Commercial DBMSs are expontially complicated
- Even 10-year veteran Oracle DBAs do not know 50% of all the new features of Oracle’s latest release
- Worsely, it would take more than 1 week for them to understand the performance behavior of a single new feature
- The Gap between human DB learning curve and new features becomes larger
- “Crossing this chasm” seems to be impossible!!
SKKU VLDB Lab.Toward Self-Tuning Databases
17
Industry Cases: IBM DB2
• SMART project: Self-Managing And Resource Tuning
- database part of IBM’s Autonomic Computing Initiatives: IBM Autonomic Computing
- Features
1. Up and Running: “configuration advisor”
- 98.4% of experts; outperform average DBAs
2. Design: “index/design advisor”
3. Maintenance: “LEO(LEarning Optimizer)”
4. Problem Determination and Correction: “health center”
5. Availability and Disaster Recovery: “recovery expert”
SKKU VLDB Lab.Toward Self-Tuning Databases
18
Industry Cases: IBM DB2
SKKU VLDB Lab.Toward Self-Tuning Databases
19
Industry Cases(2): Oracle9i
• Oralce 9i Manageability: more conservative approach
1. Self-Manageability(more focused)- space, memory, processor, operations
2. Self-Tuning- Dynamic memory management: buffer cache, run-time SQ
L memory(sort/hash area)
Oracle Memory Model
Sort Area, Hash Area, Bitmap Index Creation/Merge
• Pre-Oracle9i
- Fixed PGA size
- Sort(Hash)_Area_Size
• Oracle9i
- PGA_Aggregate_Target
- Dynamically allocated according to the needs
SKKU VLDB Lab.Toward Self-Tuning Databases
20
Industry Cases(2): Oracle9i
050
100150200250300350400450500
1 2 4 6 8 10 12 16 20
Auto
Manual - 5MB
Manual - 15MB
Avg. Response time in seconds
Number of Users
SKKU VLDB Lab.Toward Self-Tuning Databases
21
Industry Cases(2): Oracle9i
0
1,000
2,000
3,000
4,000
5,000
6,000
1 2 4 6 8 10 12 16 20
Manual - 5M
Manual - 15M
Auto
Total Memory Usage in MB
Number of Users
SKKU VLDB Lab.Toward Self-Tuning Databases
22
Industry Cases(3): MS SQL Server
• AutoAdmin: Self-Tuning and -Administering Databases
“We achieve this by enabling databases to track the usage of their systems and to gracefully adapt to application requirements. Thus, instead of applications having to track and tune databases, databases actively auto-tunes itself to be responsive to application needs.”
- See AutoAdmin homepage for details
- http://www.research.microsoft.com/research/dmx/autoadmin/
- “Rethinking Database System Architecture: Toward a Self-Tuning RISC-style Database System”
SKKU VLDB Lab.Toward Self-Tuning Databases
23
Industry Cases(4): XM
• DBMax for Oracle(http://www.ex-em.com)- Not Self-Tuning Databases
- Domestic Oracle Performance Monitoring Tool- We plan to incorporate intelligent recommendation wizards for D
B tuning- Application logic- Batch program scheduling- Automatic problem tracking down- Configuration setting- Dynamic memory adjustment- SQL tuning: based on “Query Graph” methodology
- remedy for optimizer’s internal “size estimation cost formula” & “attribute value independence assumption”
SKKU VLDB Lab.Toward Self-Tuning Databases
24
References
• Gerhard Weikum et al., “Self-Tuning Database Technology and Information Services: from Wishful Thinking to Viable Engineering” VLDB 2002, Ten-Year Award
• Gerhard Weikum et al, “The COMFORT Automatic Tuning Project”, Information Systems 19(5), 1994
• Gerhard Weikum et al, “Special Issues on Self-Tuning Database and Applications,” Data Engineering Bulletine, Jun., 1999
• IBM’s Autonomic Computing Perspectives: http://www.resarch.ibm.com/autonomic/manifesto/autonomic_computing.pdf
• Guy Lohman et al, “Toward Automomic computing with DB2 Universal Database,” SIGMOD Record, Sep., 2002
• MS AutoAdmin Homepage: http://www.research.microsoft.com/
• Oracle Manageability: http://technet.oracle.com/products/manageability/content.html
SKKU VLDB Lab.Toward Self-Tuning Databases
25
Let’s DBMax!