Presentation Title Up to 3 Lines of Text. Lorem Ipsum Delorum.
Lorem Vida Vada Ium Delorum.Speaker Daniel Zilio IBM Canada
Ltd
Session Code: <D07> Wed, May 14, 2014 (10:30 AM – 11:30 AM) |
Platform: DB2 LUW
#IDUG
Objectives
• Cover a complete end-to-end scenario with tools from the LUW AESE
bundle to show how to monitor poorly performing set of statements,
tune them as well as re-monitor and compare the tuned performance
to the original performance
• Understand how to find long running complex queries • Understand
how to analyze execution plans without re-explain
and obtain recommended tuning improvements all in the same
browser
• Exhibit advanced features in the tuning tool such as: virtual
index analysis
• Point to best practices for monitoring and tuning a workload in
DB2 LUW
2
#IDUG
DB2 Tools Supporting Peformance Monitoring and Tuning • Monitor and
baseline performance
• InfoSphere Optim Performance Manager (OPM) V5.3 * • Analyze query
access plans and identify tuning possibilities
• InfoSphere Optim Query Workload Tuner (OQWT) V4.1.0.1 * •
Implement tuning recommendations
• InfoSphere Optim Query Workload Tuner V4.1.0.1 *
* Included in DB2 AESE (and AWSE in 10.5 or later)
#IDUG
4
clients
Data pruning Configurable retention
Administration Monitoring
Interactive Dashboards
Health summary
Performance overview
Detailed dashboards
Real-time History
Monitored Servers Monitored Clients
TCP/IP and DRDA
DB2 9.5- Attachment
Snapshots SQL connection
TCP/IP and DRDA
#IDUG
#IDUG
OQWT 4.1.0.1 Key Features for DB2 LUW for v9.5 and above
Access Path Graph
Access Path Graph
Visual Plan Hint
Visual Plan Hint
Query Report Query Report
Index Advisor Index Advisor
Query Advisor Query Advisor
Statistics Advisor Statistics Advisor
Query AdvisorsQuery Tools
Access Plan Explorer
Access Plan Explorer
Table Organization Advisor
Table Organization Advisor
new
new
#IDUG
OQWT Workload Tuning • OQWT automates tuning and tunes the workload
as a whole
• A workload in OQWT is just a collection of statements
• Workload tuning speeds up analysis • Analyzes multiple queries at
once
• Workload tuning considers tradeoffs in tuning multiple statements
together • Statistics tradeoff: CPU costs vs. query savings •
Indexing tradeoff: query speed vs. index maintenance costs • Table
organization advisor: query speed vs. table maintenance costs •
Focus on biggest impact statements
#IDUG
Performance monitoring and tuning example
• Step 1: Run workload and monitor with OPM • Step 2: Investigate
overview dashboard to see breakdown of metrics and
isolate which ones should be further investigated • Step 3: View
health summary dashboard to view high level indicators of
areas having issues • Step 4: Proceed to dashboard with more
details (in example look at I/O
breakdown • Step 5: Drill down into SQL dashboard
• Find SLOW queries (e.g., the SQL with worst response/resource
time) • Analyze execution time breakdown including IO and
data/index breakdown for
worst queries • Step 6: Tune the problem statements using OQWT •
Step 7: Rerun workload, re-monitor and show performance
comparison
reports
#IDUG
Step 1: Run the workload statements • Identify statements to serve
as the workload
• Represents workload on the system • Potentially can be the
statements already in package cache
• Run the statements • Run using DS 4.1.0.1 Integrated Query
Environment (SQL Script view)
• Have OPM monitor performance the workload execution
#IDUG
• Get more context of system activity • Set baselines using the
time slider to set reference point • View significant deviations
from the baseline
12
Set baselines
Time slider controlSee deviations. View range. Hover to see
values.
OPM Overview dashboard Key-performance-indicators (KPIs) will tell
you what‘s going on
Alerts for critical areas will tell you immediately if something is
critical and needs further attention
Real time or any time views with automated aggregation and
retention management
Historical information lets you go back in time and see how a
problem arose, mitigate problems first and do causal analysis
later, compare to prior time periods
#IDUG
Overview Dashboard • Isolate the portion of execution time that
dominates • Can look at this related to output from health
summary
#IDUG
Overview Dashboard (continued)
• Further details can be viewed in the dashboard related to
Locking, I/O and SQL processing
• Analysis can point to issues such as relatively high physical
reads
#IDUG
24x7, always-on monitoring Web based dashboards
available anywhere Key Performance Indicators (KPIs) for server and
response
time metrics Workload-specific templates State, threshold, and
user-defined alerts Cross-database summary, alert history, alert
details Customized email, SMTP, SNMP notifications Drill through to
detailed data for alerted condition
Flagged as an issue. Double
click to expand to
details
#IDUG
Alert Details for I/O • Show details such as the type and time of
the alert • One can dig into a graphical view of some metrics over
time
#IDUG
Step 4: Guided Flow to Problem Diagnosis
• Drill down into relevant dashboard • View alerted condition
context and analyze
#IDUG
I/O Dashboard • Bufferpool information can be analyzed • Such
issues as low hit ratio and breakdown of I/O time spent
such as data pages over index pages can be easily seen
#IDUG
Dashboard Analysis • From the overview, one can detect a resource
issue • For I/O dashboard, one can determine the bufferpool leading
to most I/O
such as rows read and physical pages • You can drilldown to the
tablespace and table contributing most I/O
#IDUG
Step 5: SQL Dashboard Analysis • Isolate statements leading to
performance issues • In this case, more physical pages over many
statements
of the workload are from data page access • Use OQWT to determine
if better indexes can be chosen
• Top executions or summary • Sort by CPU time, I/O time,
rows
read, etc • Real time or historical • Partition or member views •
SQL statement details • Tune with Query Workload Tuner • In-context
WLM reports • What changed with Configuration
Manager • Actions for single and multiple
statements • Stored procedure correlation
#IDUG
Long running complex query identification • SQL dashboard can be
used to determine slowest statement • Analysis of time breakdown
for that statement can be made • Tuning of an individual statement
can be made as well
#IDUG
Step 6: Tune a set of statements (SQL Dashboard)
• The “Tune All” button will pass the statements to Optim Query
Workload Tuner • Optim Query Workload Tuner will tune statements as
a whole
Launch Optim Query Workload Tuner
#IDUG
OPM-OQWT Integration
• Tune and review recommendation without Data Studio Client •
Embedded query tuning functions packaged with OPM web console •
Support ad-hoc tuning and retune a query • Support access plan
graph
• Preserve Tune and Tune All launch points in OPM • SQL dashboard •
EI dashboard • Locking dashboard • OPM Report
• Separate menus to tune using Data Studio Client and OPM Web
Console • If your statement source is not from OPM such as an
activity event monitor, stored
procedure, etc., then one would use the Data Studio client
• Persist tuning jobs and results • Support both DB2 z/OS and DB2
LUW.
#IDUG
#IDUG
#IDUG
#IDUG
#IDUG
#IDUG
Tune from Data Studio Client • From OPM, workload statements copied
to Capture
view of OQWT
• Capture of a workload can be from a wide set of sources:
• File • Package cache • Package from the catalog • SQL procedure •
View, Trigger or SQL UDF • OPM repository • Event monitor
• Optionally filter statements to target worst set of
statements
• E.g., tune longest running statements and/or statements that are
most frequent
• Filters can include: • Pruning statements based on runtime
information • Capping the number of statements in total
#IDUG
Workloads in OQWT • The set of statements can be saved as an OQWT
WORKLOAD
• Saved workloads can be re-tuned in the future to determine if new
changes to the system require further performance tuning
• Single query analysis and advisors can be invoked
#IDUG
Invoking the New Advisor • For a workload defined in the MANAGE
tab, select the advisor
from the INVOKE view • Select items to recommend to execute the
related advisor(s) • Outputs the result in the Review view
#IDUG
Summary for each object used in the workload
View and run RUNSTATS recommendations
#IDUG
RUNSTATS execution from the Advisor • The RUNSTATS statements can
be run from the Statistics
Advisor • One can choose to also save the statistics to the
statistics
profile for auto runstats
Workload Index Advisor (WIA) • Improve performance by accounting
for missing useful indexes, such as:
• Foreign key indexes • Indexes for filtering, screening, and
ordering • Indexes to support index only access • Multi-column
foreign key indexes for zigzag joins in DB2 10 for LUW
Create recommended Indexes with statistics collection
View affected statements and estimated performance gain for the
selected index
Summary for each object used in the workload
#IDUG
Creating recommended indexes from WIA • The creation script can
be
viewed and run • It includes statistics
collection on index creation • Affected statements per
index can also be displayed
#IDUG
Workload Test Candidate Indexes
• Allow user to run test candidate indexes against a workload •
Supports disabling of existing indexes for workload and
single
query • Can be invoked stand alone or from WIA • Users can
also:
• Virtually add and modify candidate indexes • Modify candidate
index statistics
#IDUG
#IDUG
Review Results Details • Similar to the index advisor output •
Displays new and existing indexes
#IDUG
Workload Access Plan Comparison • Review tab only displays most
recent workload test candidate index results
• Same for all workload advisors • Comparison can be done in many
cases:
• Compare plans before and after tuning is done such as new
statistics or indexes were added • Compare plans with virtual
indexes to plans before virtual indexes were attempted • Compare
plans before and after a DB2 migration • Compare plans before and
after binding of static SQL • Compare the same or different
workloads on same or different systems (e.g., compare the workload
on a
production and test system) • Compare plans to a workload whose
plans were saved at a time the system was running well (for
baseline
comparisons to the newer state of the DB system)
#IDUG
be invoked
© 2013 IBM Corporation42
Single Query Access Plan Compare Details Combine plan structure
comparison and access plan graph comparison
#IDUG
Access Plan Details
• View the query plan in more details • Includes view of access
plan using web based explain
#IDUG
IBM Silicon Valley Laboratory – DB2 11 for z/OS ESP QCERT
Simplified operations with less clicks
* IE v9 or above, Firefox, Chrome are recommended
#IDUG
IBM Silicon Valley Laboratory – DB2 11 for z/OS ESP QCERT
More powerful advanced search to help locate the problematic
plans
#IDUG
Step 7: Monitor the tuned SQL statements in OPM • Rerun and monitor
the workload after tuning • Verify that the tuning has fixed the
performance issue
#IDUG
• SQL analysis • Top SQL • Top Package • SQL comparison
• Resource analysis • Disk space consumption • Table • Workload
Manager
#IDUG
SQL Comparison Report • Compare workload execution before and after
tuning • The report can show execution time, I/O, and CPU time
comparison
#IDUG
#IDUG
• Workload Table Organization Advisor • Determines which row
organized tables to convert to column organization • Estimates
query workload performance improvement based on the
recommendation
• Workload Test Candidate Table Organization feature • Allows
user-driven what-if analysis using a subset of row organized tables
• Determines the estimated workload performance improvement if they
are converted
#IDUG
Improve Statistics Quality for Complex Operations Statistical Views
Recommendations
• Two issues for statistics estimation in the DB2 for LUW optimizer
• Data skew and join predicate filter [mis]calculation
• WHERE DS.storekey = S.storekey and DS.store_sales > … • By
default, each value of S.storekey would match with an equal number
of rows in DS but not true with skew
• Overloaded dimension in a star join • Time dimension may contain
20 years of data but the transaction table in a data warehouse
keeps only the last 5 quarters of
data. Uniform distribution is a wrong assumption after the
join
• A Statistical View is considered a key solution in solving these
problems • CREATE view sv_store_dailysales as • (SELECT s.* FROM
store s, daily_sales ds WHERE s.storekey = ds.storekey) • ALTER
VIEW sv_store_dailysales ENABLE QUERY OPTIMIZATION • RUNSTATS on
table db2dba.sv_store_dailysales WITH DISTRIBUTION
• Statistics on the views will be used in query planning (such as
join result size) • Assists LUW in determining join sizes and how
many rows feed operators after joins • Extends Workload Statistics
Advisor to collect expression and join information • Determines
which statistical views to create and potentially which existing
statistical
views to modify, and what statistics to collect on the views
#IDUG
Statistical Views Recommendation Details • Recommended statsviews
are listed in the summary table • Generates DDL and RUNSTATS
statements • Affected statements per view are shown
Generates execution
#IDUG
Statistical Views Advisor and DB2 LUW 10 • For DB2 LUW 10, the
Workload Statistical Views
Advisor handles new statsview features in DB2 LUW 10
• Statistical RI constraint • More than 2 tables in the CREATE for
compound
statsviews • Expressions in the CREATE and RUNSTATS • Column groups
in the RUNSTATS
#IDUG
Workload Design Advisor
• Allows for the selection of: • MQTs • Clustering • DPF
partitioning • Indexes associated with
these new recommendations
• Users can limit the space occupied by these constructs as in the
index advisor
#IDUG
OQWT Single Query Tuning • Single query tuning features sometimes
makes sense
• View access plans and sort operators in the plan explorer to
focus on problem areas of the statement
• Run statistics advisor to determine if any missing statistics
make sense • Run index advisor to determine if an index could be
added • Manually run the test candidate indexes feature to see new
index performance
improvement • Compare plans between current explain and explain
from index additions • Advanced:
• Try the Optimization Profile Editor to try out plan hints to
change a plan
• Setup single query as a file and capture as a workload • Run the
workload statistical views advisor to get recommendations
#IDUG
Advisor Recommendations
Visual Plan Hints
• View operators with associated measures in one area
• Order operators based on measures to identify hot spots
• View operator details
• View flow of query operator processing
• See optimizer rationale
Performance Manager v5.3 •
https://www.ibm.com/developerworks/community/wikis/home?
• Tuning and monitoring database system performance: •
https://www.ibm.com/developerworks/community/wikis/home?
lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Tuni
ng%20and%20Monitoring%20Database%20System%20Performance
• Writing and tuning queries for optimal performance •
https://www.ibm.com/developerworks/community/wikis/home?
lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Writ
ing%20and%20Tuning%20Queries%20for%20Optimal%20Performance
for DB2 LUW
Speaker Daniel Zilio IBM Canada Ltd
Session Code: <D07> Wed, May 14, 2014 (10:30 AM – 11:30
AM)
#IDUG
Speaker Daniel Zilio IBM Canada Ltd
Session Code: <D07> Wed, May 14, 2014 (10:30 AM – 11:30 AM) |
Platform: DB2 LUW
1
2
#IDUG
Objectives
• Cover a complete end-to-end scenario with tools from the LUW AESE
bundle to show how to monitor poorly performing set of statements,
tune them as well as re-monitor and compare the tuned performance
to the original performance
• Understand how to find long running complex queries • Understand
how to analyze execution plans without re-explain
and obtain recommended tuning improvements all in the same
browser
• Exhibit advanced features in the tuning tool such as: virtual
index analysis
• Point to best practices for monitoring and tuning a workload in
DB2 LUW
2
2
3
#IDUG
DB2 Tools Supporting Peformance Monitoring and Tuning • Monitor and
baseline performance
• InfoSphere Optim Performance Manager (OPM) V5.3 * • Analyze query
access plans and identify tuning possibilities
• InfoSphere Optim Query Workload Tuner (OQWT) V4.1.0.1 * •
Implement tuning recommendations
• InfoSphere Optim Query Workload Tuner V4.1.0.1 *
* Included in DB2 AESE (and AWSE in 10.5 or later)
4
#IDUG
4
clients
Data pruning Configurable retention
Administration Monitoring
Interactive Dashboards
Health summary
Performance overview
Detailed dashboards
Real-time History
Clients
1-slide overview of OPM 5.3 - brief talk about the highlights of
OPM - admittedly VERY brief, but a good slide anyway.
OPM is a db2 database performance monitor
-Centralized – collects db2 monitor data from across the
enterprise
-Supports multiple versions of DB2 LUW
-Automatic collection and storage of perf data over time
-Alerts built-in, and customizable/extendable
-Robust web-based online viewing of perf data with rich reporting
facilities from canned to customized
-Integration and awareness of other tooling such as Query Tuner and
OCM for example
NOTE TO SPEAKER: Common question about OPM – does it use agents?
Answer: No, it does not require any agent to be installed on
monitored db server. It uses all DB2 functionality to
monitor.
(compare/contrast to OQCR, which DOES use an agent-like component,
the “S- TAP”. Not better/worse, just different. OQCR has different
mission than OPM)
5
#IDUG
Monitored Servers Monitored Clients
TCP/IP and DRDA
[Use this version if no 9.5 or early databases to monitor] Let me
give you an overview of the architecture as a backdrop to the
discussion
InfoSphere Optim Performance Manager consists of a console server
and a repository server component. •The Repository server collects
the data from the monitored database and stores it in a DB2
repository database. It also support aggregation and pruning of the
performance data in the repository.. -The Console server reads the
data from the repository database and displays it in the Web UI on
users request. -The OPM servers must be co-located with the DB2
performance repository. [The performance repository may not be a
DPF or pureScale instance]
[click]
For monitoring DB2 V9.7 databases the repository server connects to
the monitored database and issues SQL statements to collect the
in-memory metrics. Some metrics are also still collected via
snapshots UDFs. The user specifies how often to collect the data.
Additionally depending on the configuration some event monitors may
be created on the monitored database, for example to collect
locking information.
[click]
The main user interface is the web UI. It provides a series of
dashbaords for viewing all the data in the performance
repository.
[click]
When deploying Extended Insight, users can optionally deploy the
Extended Insight client (aka Data Tools Runtime Client) on the
monitored client envirnments. These are installed on the systems
where your database applications are runningand support Java, CLI
and .Net applications clients. The Extended Insight clients send
data about executed SQL statements and transactions to the
repository server.
In parallel, the repository server collects data server execution
data for SQL and transactions (9.7 only) from the monitored
database and maps them to the data sent by the clients. This way
you get a complete picture about the transaction and statement
response times of your application. To collect the Extended Insight
server data, Performance Manager starts the package cache and unit
of work event monitors on the monitored database to get the SQL and
transaction details in addition to the listing of all statements
within the package cache. Note that EI client and server data
collection is independnet and you can activate one without the
other.
[click]
Other tools such as Optim Query WOrkload TUner, Data Studio, or
reporting products may also connect to the repository to access
performance metrics.
6
#IDUG
DB2 9.5- Attachment
Snapshots SQL connection
TCP/IP and DRDA
[Use this version if there are 9.5 database to monitor] Let me give
you an overview of the architecture as a backdrop to the
discussion
InfoSphere Optim Performance Manager consists of a console server
and a repository server component. •The Repository server collects
the data from the monitored database and stores it in a DB2
repository database. It also support aggregation and pruning of the
performance data in the repository.. -The Console server reads the
data from the repository database and displays it in the Web UI on
users request. -The OPM servers must be co-located with the DB2
performance repository. [The performance repository may not be a
DPF or pureScale instance]
[click]
For monitoring DB2 V9.7 databases the repository server connects to
the monitored database and issues SQL statements to collect the
in-memory metrics. Some metrics are also still collected via
snapshots UDFs. The user specifies how often to collect the data.
Additionally depending on the configuration some event monitors may
be created on the monitored database, for example to collect
locking information. For monitoring DB2 V9.1 or DB2 V9.5 database
the repository server uses instance attachments to get the snapshot
data in addition to connections for starting event monitors.
[click]
The main user interface is the web UI. It provides a series of
dashbaords for viewing all the data in the performance
repository.
[click]
There is also a legacy Performance Expert client. It is typically
used by customers who have migrated from DB2 Performance Expert,
the predecessor product to InfoSphere Optim Performance Manager. [
Or by users monitoring DB2 9.1 or 9.5 database who want to retain
long term data in a Performance Warehouse for reporting. ]
[click]
When deploying Extended Insight, users can optionally deploy the
Extended Insight client (aka Data Tools Runtime Client) on the
monitored client envirnments. These are installed on the systems
where your database applications are runningand support Java, CLI
and .Net applications clients. The Extended Insight clients send
data about executed SQL statements and transactions to the
repository server.
In parallel, the repository server collects data server execution
data for SQL and transactions (9.7 only) from the monitored
database and maps them to the data sent by the clients. This way
you get a complete picture about the transaction and statement
response times of your application. To collect the Extended Insight
server data, Performance Manager starts the package cache and unit
of work event monitors on the monitored database to get the SQL and
transaction details in addition to the listing of all statements
within the package cache. Note that EI client and server data
collection is independnet and you can activate one without the
other.
7
#IDUG
8
#IDUG
OQWT 4.1.0.1 Key Features for DB2 LUW for v9.5 and above
Access Path Graph
Access Path Graph
Visual Plan Hint
Visual Plan Hint
Query Report Query Report
Index Advisor Index Advisor
Query Advisor Query Advisor
Statistics Advisor Statistics Advisor
Query AdvisorsQuery Tools
Access Plan Explorer
Access Plan Explorer
Table Organization Advisor
Table Organization Advisor
new
new
Key features that will be added later this year in the OQWT 4.1
release include the Access Plan Explorer (a key item for fighting
competitors such as Quest), the Visual Plan Hint, and the Testing
of Index candidates (virtual indexes!). Additionally, the Workload
advisors allow you to do key tuning work on groupings of SQL
Query tuner tools:
============= Query Formatter
The query is formatted for readability, showing the query structure
The predicates are ordered according to your specification.
Query Annotation Display cost estimates and statistical information
about the predicates, and additional annotations such as data
skew.
Access Plan Graph Shows the visual explain Provides cost and
cardinality estimates for each step of the process Provides actual
cardinality and compares with estimates (if data server
supports)
Access Plan Explorer Tabular and hierarchical (tree) form of
explain
Summary Report
HTML form of analysis result which can be shared with other
DBAs
Statistics Advisor Provides information about missing, obsolete, or
conflicting statistics that might have a negative effect on the
access plan that the optimizer chooses to execute the query For
example, SA recommends which column group statistics to
collect
Provides a RUNSTATS command that you can run on to collect and
repair the statistics. Save RUNSTATS commands to statistics profile
table which will be use by automatic statistics collection
Index Advisor The index advisor recommends indexes that you might
create to improve the performance of the query, and provides DDL
scripts that you can run to create the recommended indexes Index
recommendations will be consolidated: For example, index (A, B, C)
will be merged with index (A, B)
Query advisor The query advisor uses a set of rules to evaluate how
the query is written and provides best-practice suggestions for
writing queries For example, there may be a missing join predicate
due to a multi-column primary- foreign key relationship
Access path advisor The access path advisor examines the access
plan chosen by the optimizer and identifies certain common access
path issues For example, a warning will be issued for a table scan,
or a hash join may not be used due to data type mismatch of the
join columns The warnings provided by the access path advisor can
help you to understand where to look for trouble in the access plan
graph.
NEW FEATURES: ============
Visual Plan Hint (to create Optimization Profiles)
Workload Support
Workload = a set of queries [+ runtime metrics, including execution
count and CPU usage]
Capture from various sources: package cache, catalog, OPM, SQL SP,
flat file, XML file, view/trigger, Routine Editor
New Workload Advisors:
Integration with OPM
The Access Plan Explorer adds key plan representation abilities. It
allows the viewing of a plan in either a tabular or tree
representation. Customers familiar with using Quest will like this
function.
Virtual Indexes can be reviewed in a what-if scenario by using the
Testing Candidate indexes feature. It will not only show which
indexes (existing and planned) are to be used, but also will give
you estimates of disk space usage and performance gains with the
new set of indexes.
The Visual Plan Hint allows you to create an optimization profile
to guide the DB2 Optimizer for query execution.
The Workload Advisors added in the 3.1.1 release add key features
and function not in the current tooling (DB2ADVIS or Data
Studio).
A workload is defined as a grouping of queries and will include the
runtime metrics of the SQL.
You also have the ability to capture the workload SQL from a number
of important sources.
From within OQWT you have the ability to invoke the advisors and
you have multiple options for the advisor recommendations: view,
save, or implement.
Key integration with Optim Performance Manager (OPM) is also
provided with the ability to tune a workload obtained/identified
from the OPM dashboard.
These advisors are discussed in more detail on the following
pages.
Workload = a set of queries [+ runtime metrics, including execution
count and CPU usage]
Capture from various sources: package cache, catalog, OPM, SQL SP,
flat file, XML file, view/trigger, Routine Editor
Actions for a specific workload:
Consolidate/Explain workloads
Create a new workload by further filtering of statements
Invoke advisors
Workload Statistics Advisor
Workload Index Advisor
Integration with OPM Tuning a workload identified in OPM
dashboard
9
#IDUG
OQWT Workload Tuning • OQWT automates tuning and tunes the workload
as a whole
• A workload in OQWT is just a collection of statements
• Workload tuning speeds up analysis • Analyzes multiple queries at
once
• Workload tuning considers tradeoffs in tuning multiple statements
together • Statistics tradeoff: CPU costs vs. query savings •
Indexing tradeoff: query speed vs. index maintenance costs • Table
organization advisor: query speed vs. table maintenance costs •
Focus on biggest impact statements
OQWT does try and focus on statements that both have highest number
of executions with highest cost so that the recommendations
provided will have the biggest impact in the resulting performance
improvements
The queries can be captured from a set of sources and can be
tailored to target certain items such as queries that are using a
certain table, or queries from the package cache or event
monitor
BENEFIT of OQWT: Speed up analysis, optimize design, and balance
resource usage
10
#IDUG
Performance monitoring and tuning example
• Step 1: Run workload and monitor with OPM • Step 2: Investigate
overview dashboard to see breakdown of metrics and
isolate which ones should be further investigated • Step 3: View
health summary dashboard to view high level indicators of
areas having issues • Step 4: Proceed to dashboard with more
details (in example look at I/O
breakdown • Step 5: Drill down into SQL dashboard
• Find SLOW queries (e.g., the SQL with worst response/resource
time) • Analyze execution time breakdown including IO and
data/index breakdown for
worst queries • Step 6: Tune the problem statements using OQWT •
Step 7: Rerun workload, re-monitor and show performance
comparison
reports
11
#IDUG
Step 1: Run the workload statements • Identify statements to serve
as the workload
• Represents workload on the system • Potentially can be the
statements already in package cache
• Run the statements • Run using DS 4.1.0.1 Integrated Query
Environment (SQL Script view)
• Have OPM monitor performance the workload execution
12
#IDUG
• Get more context of system activity • Set baselines using the
time slider to set reference point • View significant deviations
from the baseline
12
Set baselines
Time slider controlSee deviations. View range. Hover to see
values.
OPM Overview dashboard Key-performance-indicators (KPIs) will tell
you what‘s going on
Alerts for critical areas will tell you immediately if something is
critical and needs further attention
Real time or any time views with automated aggregation and
retention management
Historical information lets you go back in time and see how a
problem arose, mitigate problems first and do causal analysis
later, compare to prior time periods
You can get an at-a-glance view of the activity on a database in
the Overview dashboard. It provides a higher level view of system
activity: CPU distribution overall as well as within DB2, workload
information, and specific tabs for examining locking, I/O, and SQL
processing.
From here you can set baselines using the time slider to select the
baseline timeframe. The time slider controls the range of
performance data that you view on the dashboards and reflects all
the data stored in the performance repository.
A baseline refines a range of activity against which you want to
compare, typically a range that you would consider “normal”.
Performance Manager then calculates norms from this range. Note
that you want to select a range that is wide enough to represent
normal variations, but not too wide as to be meaningless.
Once a baseline is set, the at-a-glance values are automatically
compared to the baseline and significant deviations indicated.
Deviations are flagged when current values out outside of 2
standard deviations or lower or higher than the minimum or maximum
in the baseline interval. Deviation are flagged by color and the
diamond to the left of the bar. You can see the normal range in
grey (dark grey is 1 std dev, light grey is 2 std dev) and hovering
pops up the specific baseline values.
[Note: The at a glance dashboard is only available for DB2 9.7 and
later monitored databases. For 9.5, only the KPI overview dashboard
is available]
13
#IDUG
Overview Dashboard • Isolate the portion of execution time that
dominates • Can look at this related to output from health
summary
14
#IDUG
Overview Dashboard (continued)
• Further details can be viewed in the dashboard related to
Locking, I/O and SQL processing
• Analysis can point to issues such as relatively high physical
reads
15
#IDUG
24x7, always-on monitoring Web based dashboards
available anywhere Key Performance Indicators (KPIs) for server and
response
time metrics Workload-specific templates State, threshold, and
user-defined alerts Cross-database summary, alert history, alert
details Customized email, SMTP, SNMP notifications Drill through to
detailed data for alerted condition
Flagged as an issue. Double
click to expand to
details
InfoSphere Optim solutions provides proactive notification of
emergent problems before they impact your business. •Monitor any
DB2 workload, 24 x 7, to assure that nothing is missed. Getting
proactive notification means that monitoring must be active 24x7.
To support always-on monitoring, you want to minimize monitoring
overhead on the monitored databases. We exploit state-of-the-art
DB2 9.7 in-memory metrics for performance monitoring. Any DB2
environment can be monitored from single instance SMPs to 100+
partition warehouses, or pureScale deployments •Web based
dashboards make data available anywhere, anytime. IT staff does not
have to go into the office to do problem diagnosis. •Key
performance indicators (KPIs) that support both database server and
response time measurements form the basis of alerting so you get
notified before emerging bottlenecks become business inhibitors.
•Performance Manager comes with built-in templates for common
workloads to get up and running quickly. Templates are available
for Cognos, Information Server, SAP, and WebSphere workloads or for
more generic OLTP, BI, or pureScale deployments. Templates include
default values for which metrics to collect, collection frequency,
and key performance indicator thresholds and alerts •You can view a
cross-database summary and alerts to se where problems are or get
specific notifications to identify problems. As shown here in the
background, the health summary provides instant visual indicators
of the health of all monitored databases with standard red, yellow,
and green light indicators. You can look at aggregate alerts to
help determine which database to focus. You can use the single
click display of these alerts to see if any of these alerts are
occurring around the same time to see if the problem is a single
spike issue or if it is a new steady-state level which will need to
be corrected long term.
•You can also get email notification, integrate SNMP alerts with
system management filtering or routing software, or incorporate
custom integrations with ticketing or other systems •From any
alert, you can display more details about the alert and then drill
down to detailed diagnostic dashboards for each of these areas. For
example, this screen shot indicates a locking issue. Clicking on
the red icon opens the alert list where you can get more detail on
the alert condition.
16
#IDUG
Alert Details for I/O • Show details such as the type and time of
the alert • One can dig into a graphical view of some metrics over
time
17
#IDUG
Step 4: Guided Flow to Problem Diagnosis
• Drill down into relevant dashboard • View alerted condition
context and analyze
18
#IDUG
I/O Dashboard • Bufferpool information can be analyzed • Such
issues as low hit ratio and breakdown of I/O time spent
such as data pages over index pages can be easily seen
19
#IDUG
Dashboard Analysis • From the overview, one can detect a resource
issue • For I/O dashboard, one can determine the bufferpool leading
to most I/O
such as rows read and physical pages • You can drilldown to the
tablespace and table contributing most I/O
Once the table leading to I/O issues is selected, you can select
SHOW SQL which will open the SQL dashboard to analyze which
statements using that table are the main contributors to the I/O
issues.
20
#IDUG
Step 5: SQL Dashboard Analysis • Isolate statements leading to
performance issues • In this case, more physical pages over many
statements
of the workload are from data page access • Use OQWT to determine
if better indexes can be chosen
• Top executions or summary • Sort by CPU time, I/O time,
rows
read, etc • Real time or historical • Partition or member views •
SQL statement details • Tune with Query Workload Tuner • In-context
WLM reports • What changed with Configuration
Manager • Actions for single and multiple
statements • Stored procedure correlation
21
#IDUG
Long running complex query identification • SQL dashboard can be
used to determine slowest statement • Analysis of time breakdown
for that statement can be made • Tuning of an individual statement
can be made as well
22
#IDUG
Step 6: Tune a set of statements (SQL Dashboard)
• The “Tune All” button will pass the statements to Optim Query
Workload Tuner • Optim Query Workload Tuner will tune statements as
a whole
Launch Optim Query Workload Tuner
23
#IDUG
OPM-OQWT Integration
• Tune and review recommendation without Data Studio Client •
Embedded query tuning functions packaged with OPM web console •
Support ad-hoc tuning and retune a query • Support access plan
graph
• Preserve Tune and Tune All launch points in OPM • SQL dashboard •
EI dashboard • Locking dashboard • OPM Report
• Separate menus to tune using Data Studio Client and OPM Web
Console • If your statement source is not from OPM such as an
activity event monitor, stored
procedure, etc., then one would use the Data Studio client
• Persist tuning jobs and results • Support both DB2 z/OS and DB2
LUW.
24
#IDUG
25
#IDUG
26
#IDUG
28
#IDUG
29
#IDUG
30
#IDUG
Tune from Data Studio Client • From OPM, workload statements copied
to Capture
view of OQWT
• Capture of a workload can be from a wide set of sources:
• File • Package cache • Package from the catalog • SQL procedure •
View, Trigger or SQL UDF • OPM repository • Event monitor
• Optionally filter statements to target worst set of
statements
• E.g., tune longest running statements and/or statements that are
most frequent
• Filters can include: • Pruning statements based on runtime
information • Capping the number of statements in total
31
#IDUG
Workloads in OQWT • The set of statements can be saved as an OQWT
WORKLOAD
• Saved workloads can be re-tuned in the future to determine if new
changes to the system require further performance tuning
• Single query analysis and advisors can be invoked
32
#IDUG
Invoking the New Advisor • For a workload defined in the MANAGE
tab, select the advisor
from the INVOKE view • Select items to recommend to execute the
related advisor(s) • Outputs the result in the Review view
Note for this release sthe Table Organization advisor cannot be run
at the same time as the index, MQT, MDC, or partitioning advisor
since their recommendations may conflict.
33
#IDUG
Summary for each object used in the workload
View and run RUNSTATS recommendations
Provides advice on: Missing statistics Conflicting statistics
Out-of-date statistics Base tables and materialized query tables
(MQTs)
Simplifies use
34
#IDUG
RUNSTATS execution from the Advisor • The RUNSTATS statements can
be run from the Statistics
Advisor • One can choose to also save the statistics to the
statistics
profile for auto runstats
Workload Index Advisor (WIA) • Improve performance by accounting
for missing useful indexes, such as:
• Foreign key indexes • Indexes for filtering, screening, and
ordering • Indexes to support index only access • Multi-column
foreign key indexes for zigzag joins in DB2 10 for LUW
Create recommended Indexes with statistics collection
View affected statements and estimated performance gain for the
selected index
Summary for each object used in the workload
Simplifies use Consolidate indexes and provide a single
recommendation
Displays existing index information and usage
Show affected SQL due to index recommendations to provide feedback
as to benefit per index
Enables what-if analysis
Provides DDL to create indexes to run immediately or save
Provides additional enhancements on top of db2advis
Avoid giving index recommendations with little improvements
Avoid giving “wide” index recommendations
Reduce the number of recommendations by consolidating indexes
Recommends indexes for zigzag join for DB2 LUW 10 and consolidates
taking jumpscan access into account as well in DB2 LUW 10
Capture of static SQL is more easily done with OQWT including using
the static SQL to run the index advisor
Finds redundant existing indexes to drop/alter
36
#IDUG
Creating recommended indexes from WIA • The creation script can
be
viewed and run • It includes statistics
collection on index creation • Affected statements per
index can also be displayed
37
#IDUG
Workload Test Candidate Indexes
• Allow user to run test candidate indexes against a workload •
Supports disabling of existing indexes for workload and
single
query • Can be invoked stand alone or from WIA • Users can
also:
• Virtually add and modify candidate indexes • Modify candidate
index statistics
38
#IDUG
39
#IDUG
Review Results Details • Similar to the index advisor output •
Displays new and existing indexes
40
#IDUG
Workload Access Plan Comparison • Review tab only displays most
recent workload test candidate index results
• Same for all workload advisors • Comparison can be done in many
cases:
• Compare plans before and after tuning is done such as new
statistics or indexes were added • Compare plans with virtual
indexes to plans before virtual indexes were attempted • Compare
plans before and after a DB2 migration • Compare plans before and
after binding of static SQL • Compare the same or different
workloads on same or different systems (e.g., compare the workload
on a
production and test system) • Compare plans to a workload whose
plans were saved at a time the system was running well (for
baseline
comparisons to the newer state of the DB system)
41
#IDUG
be invoked
© 2013 IBM Corporation42
Single Query Access Plan Compare Details Combine plan structure
comparison and access plan graph comparison
Need explanation for these.
Access Plan Details
• View the query plan in more details • Includes view of access
plan using web based explain
Information Management for System z
2014 WW Tech Sales Boot Camp - Jan 27th to Jan 31st 44
44
#IDUG
IBM Silicon Valley Laboratory – DB2 11 for z/OS ESP QCERT
Simplified operations with less clicks
* IE v9 or above, Firefox, Chrome are recommended
Information Management for System z
2014 WW Tech Sales Boot Camp - Jan 27th to Jan 31st 45
45
#IDUG
IBM Silicon Valley Laboratory – DB2 11 for z/OS ESP QCERT
More powerful advanced search to help locate the problematic
plans
46
#IDUG
Step 7: Monitor the tuned SQL statements in OPM • Rerun and monitor
the workload after tuning • Verify that the tuning has fixed the
performance issue
47
#IDUG
• SQL analysis • Top SQL • Top Package • SQL comparison
• Resource analysis • Disk space consumption • Table • Workload
Manager
Performance Manager includes out-of-the-box reporting against the
retained performance metrics. Users can report over any timeframe
and can schedule, browse, print, export, present, and share their
analysis and results. Each interactive report is like multiple
reports in one. Reports typically provide a graphical view with
zoom in and drill down capabilities to additional detail or through
to a contextually launched SQL report.
Reports can be broadly categorized into three main area: General
reporting, SQL analysis, and Resource consumption
General reports include:
•The Cross Database Overview report shows key performance data for
multiple monitored databases for a specified period of time. You
get an overview of how these monitored databases are performing in
one report. If you notice that a particular database has a problem,
you can analyze that database further. •The Performance Overview
Report which shows the overall health of the monitored database for
all major metrics. Users can view the complete database system or
only parts of it. •The database or database manager configuration
report contains details about capacity management, communications,
logging and recovery, and database management including detail
about partition or member level configuration and configuration
changes. •The Database Connection Report provides an overview of
the active database connections for a given time frame. The report
displays key performance indicators, such as lock wait times,
physical and logical reads and writes, and other connection
statistics. This report can identify applications that are not
performing well or applications that are causing problems in other
database applications. DBAs can drill down into a specific
connection to view complete identification details, timing
information, SQL activity, locks, cache, buffer pool, sorts, and
agent-related activity.
For SQL Analysis there are 3 main reports: •The Top SQL Report
shows top resource consumers, both long running large queries and
frequently run short queries. You can select the criteria by which
the data is organized and then drill into specific statements, see
response time patterns in histograms or performance trends. When
you identify queries that need to be tuned, you can launch Query
Tuner right from the report. •The Top SQL Package Report allows you
to identify top resource consumers by package name in a specified
time interval. DBAs can look at a graphic representation of the
workload by day to identify heavy duty, critical, or rogue
packages. Then you can select a specific package and drill through
to the SQL report for additional detail. •The SQL Baseline
Comparison Report lets you compare top SQL statements in the same
database regarding the maximum improvement, or regression, or both,
within two time frames. Users can drill down for a detailed
analysis of a specific SQL statement. The report includes the
complete statement text, general statement information, response
time analysis, sort performance, I/O activity, and buffer pool
activity. Similarly, you can drill through to the SQL report for
additional detail.
•For analyzing resources there are the Disk Space Consumption
Report, Tables usage report, and Workload manager report •The Disk
Space Consumption report provides an overview of the current disk
space usage by table space. You can analyze information about
growth rates to plan for future disk space requirements or table
space configuration changes. You drill down into the details of a
given table space, such as configuration, containers and tables,
container layout, and data skew. •The Table Usage Report lets you
identify "hot" tables or fast growing tables that might cause disk
contention or that might need reorganization. •The Workload Manager
Reports provide detail on workload configurations and drill through
to specific workloads or Service SuperClass, SubClass, and Work
Action Sets. They include details on queue time, execution time,
lifetime, and histograms for advanced tuning of Workload Manager
configurations.
48
#IDUG
SQL Comparison Report • Compare workload execution before and after
tuning • The report can show execution time, I/O, and CPU time
comparison
49
#IDUG
50
#IDUG
• Workload Table Organization Advisor • Determines which row
organized tables to convert to column organization • Estimates
query workload performance improvement based on the
recommendation
• Workload Test Candidate Table Organization feature • Allows
user-driven what-if analysis using a subset of row organized tables
• Determines the estimated workload performance improvement if they
are converted
Introduce in InfoSphere Optim Query Workload Tuner 4.1 are the
workload table organization advisor and the workload test candidate
table organization feature. The Workload table Organization
Advisor
Determines which row organized tables should be converted to column
organization
Estimates query workload performance improvement based on the
recommendation
The Workload Test Candidate Table Organization feature
Supports user-driven what-if analysis based on row-organized tables
that they select
It determines the estimated workload performance improvement for
this input set of tables if they are converted to column
organization
In the screen shot you can see that Query Tuner allows the user
to
•View tables recommended for conversion, the rationale for
conversion (not for not being converted), and any warnings
associated with the conversion. For example Warnings are provided
when the conversion will lose or change certain table properties
like indexes will be removed, Enforced RI or check constraints will
change to NOT ENFORCED, partitioning and clustering removed,…
•View statements affected
•Test candidate column-organized tables and get estimated
improvements
•Export table data to a file
52
#IDUG
Improve Statistics Quality for Complex Operations Statistical Views
Recommendations
• Two issues for statistics estimation in the DB2 for LUW optimizer
• Data skew and join predicate filter [mis]calculation
• WHERE DS.storekey = S.storekey and DS.store_sales > … • By
default, each value of S.storekey would match with an equal number
of rows in DS but not true with skew
• Overloaded dimension in a star join • Time dimension may contain
20 years of data but the transaction table in a data warehouse
keeps only the last 5 quarters of
data. Uniform distribution is a wrong assumption after the
join
• A Statistical View is considered a key solution in solving these
problems • CREATE view sv_store_dailysales as • (SELECT s.* FROM
store s, daily_sales ds WHERE s.storekey = ds.storekey) • ALTER
VIEW sv_store_dailysales ENABLE QUERY OPTIMIZATION • RUNSTATS on
table db2dba.sv_store_dailysales WITH DISTRIBUTION
• Statistics on the views will be used in query planning (such as
join result size) • Assists LUW in determining join sizes and how
many rows feed operators after joins • Extends Workload Statistics
Advisor to collect expression and join information • Determines
which statistical views to create and potentially which existing
statistical
views to modify, and what statistics to collect on the views
Statistical Views in DB2 for LUW are a powerful feature allowing
you to gather statistics on views and allowing better execution
plans. The Workload Statistics Advisor – Statistical Views feature
extends the Workload Statistics Advisor to provide key information
to DBAs on statistical views which would help the execution of the
workload provided. This slide provides some key points about
WSVA.
NOTE: Statistical views are like MQTs except they do no store data
or have indexes, but just store the statistics on the select clause
objects and match queries to use those statistics
53
#IDUG
Statistical Views Recommendation Details • Recommended statsviews
are listed in the summary table • Generates DDL and RUNSTATS
statements • Affected statements per view are shown
Generates execution
Summary table with sortable columns
for new and existing statsviews
Note that the user can highlight each statistical view which may be
a new one (state = PROPOSED) or an existing one. For the
highlighted view, the Details section below the list of statistical
views will indicate which statements that view assists and which
base tables are used in constructing that view.
A user can check the checkboxes in the view list and view the SQL
scripts that would include the CREATE VIEW, ALTER VIEW, and
RUNSTATS statements needed to create or modify statistical views. A
user can then choose to save the script to a file or clipboard, or
RUN the script as well as RUN the script and save the RUNSTATS to
the LUW statistics profile.
This advisor also will show the list of existing statistical views
that are UNUSED by the workload.
54
#IDUG
Statistical Views Advisor and DB2 LUW 10 • For DB2 LUW 10, the
Workload Statistical Views
Advisor handles new statsview features in DB2 LUW 10
• Statistical RI constraint • More than 2 tables in the CREATE for
compound
statsviews • Expressions in the CREATE and RUNSTATS • Column groups
in the RUNSTATS
55
#IDUG
Workload Design Advisor
• Allows for the selection of: • MQTs • Clustering • DPF
partitioning • Indexes associated with
these new recommendations
• Users can limit the space occupied by these constructs as in the
index advisor
56
#IDUG
OQWT Single Query Tuning • Single query tuning features sometimes
makes sense
• View access plans and sort operators in the plan explorer to
focus on problem areas of the statement
• Run statistics advisor to determine if any missing statistics
make sense • Run index advisor to determine if an index could be
added • Manually run the test candidate indexes feature to see new
index performance
improvement • Compare plans between current explain and explain
from index additions • Advanced:
• Try the Optimization Profile Editor to try out plan hints to
change a plan
• Setup single query as a file and capture as a workload • Run the
workload statistical views advisor to get recommendations
57
#IDUG
Advisor Recommendations
Visual Plan Hints
• View operators with associated measures in one area
• Order operators based on measures to identify hot spots
• View operator details
• View flow of query operator processing
• See optimizer rationale
Operator details area
Sort by measures
The Access Plan Explorer adds key plan representation abilities. It
allows the viewing of a plan in either a tabular or tree
representation. Customers familiar with using Quest will like this
function.
Provides a tabular or tree representation of a query plan
Easy to spot the most “expensive” operator
Summary table contains rows of operators and their properties
All columns are sortable in the summary table (e.g. sort by
cardinality or total query cost in descending order)
Additional database statistics or query/plan properties will be
displayed automatically without additional clicking
Provide added recommendations to existing indexes:
Drop indexes which already covered by other existing indexes
Recommend indexes to be changed from non-unique to unique since
they contain the same index keys as existing unique indexes
Recommend indexes to be changed to unique indexes with extra
include columns.
59
#IDUG
Performance Manager v5.3 •
https://www.ibm.com/developerworks/community/wikis/home?
• Tuning and monitoring database system performance: •
https://www.ibm.com/developerworks/community/wikis/home?
lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Tuni
ng%20and%20Monitoring%20Database%20System%20Performance
• Writing and tuning queries for optimal performance •
https://www.ibm.com/developerworks/community/wikis/home?
lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Writ
ing%20and%20Tuning%20Queries%20for%20Optimal%20Performance
#IDUG
for DB2 LUW
Speaker Daniel Zilio IBM Canada Ltd
Session Code: <D07> Wed, May 14, 2014 (10:30 AM – 11:30
AM)
60
Objectives
Slide 4
Slide 6
Slide 7
OQWT 4.1.0.1 Key Features for DB2 LUW for v9.5 and above
OQWT Workload Tuning
Step 1: Run the workload statements
Step 2: Database Overview Dashboard – At a Glance
Overview Dashboard
Alert Details for I/O
I/O Dashboard
Dashboard Analysis
Step 6: Tune a set of statements (SQL Dashboard)
OPM-OQWT Integration
Launch Single-Query Tuning from SQL Dashboard
View Single-Query Recommendation
Manage Tuning Jobs and Results
Tune from Data Studio Client
Workloads in OQWT
Workload Index Advisor (WIA)
Workload Test Candidate Indexes
Review Results Details
Reporting
Statistical Views Recommendation Details
Workload Design Advisor
Best Practices in performance monitoring and tuning:
References
