VERITAS Indepth™ for DB2 V2.5

Product Evaluation

By Philip K. Gunning Principal Consultant

Certified Advanced Database Administrator DB2 UDB V8.1

Gunning Technology Solutions, LLC

February 12, 2004

1

Background In today’s Internet age, businesses are challenged to provide customers with access to customer accounts, order status, and products and services offered 24 hours a day, 7 days per week. In this “On Demand” economy, businesses have chosen to use Relational Database Management Systems (RDBMS) (in our case, we will focus only on DB2™ Universal Database (UDB) for Linux, UNIX, and Windows) as the back-end for Internet based applications due to their robustness, reliability, and high availability. As such, application performance management products are needed to enable IT professionals to deliver and maintain top performing and highly available “On Demand” applications. VERITAS Indepth™ for DB2 software meets these requirements. 1. Introduction to VERITAS Indepth™ for DB2 V2.5 Gunning Technology Solutions, LLC principal Philip K. Gunning recently conducted an evaluation of VERITAS Indepth™ for DB2 V2.5 (formerly Precise/Indepth for DB2). VERITAS Indepth™ for DB2 is an application performance management software product that supports DB2 UDB running on AIX, Solaris, HP-UX, Linux and Windows NT/2000. The product uses a client /server architecture to deliver performance information about applications and DB2 database resource activity. Refer to Figure 1-1 for an overview of the VERITAS Indepth™ architecture and components.

Figure 1-1. VERITAS Indepth™ for DB2 components and architecture As illustrated in Figure 1-1, Indepth™ for DB2 consists of the following components:

• Client – Is a Windows-based client which consists of a Graphical User Interface (GUI) that is used to install the server-side lightweight agents and to display application and database performance information

• Listener Agent – Listens for requests from the user interface and starts the appropriate process to handle the request. It also handles management requests for the agent manager, such as starting or stopping an agent.

2

• Collector Agent -- Is a UNIX™ (currently Solaris, Linux, HP-UX and AIX) or Windows NT/2000™ lightweight process that monitors and collects information (using DB2 Snapshots) about DB2 activity and resource usage data. The collector agent uses a sampling technique that results in very little monitoring overhead. This is different than other products on the market that typically use event monitors to gather DB2 event data. The collector agent collects DB2 resource data for the following DB2 data elements:

Sessions Programs SQL statements OS users DB2 users Client machines Modules Actions SQL statement access plans Estimates of DB2 resource consumption

• EMC Symmetrix™ and IBM ESS Shark™ (SharkPoint) Agents -- Collects file system, disk,

storage subsystem and DB2 tablespace resource related data providing correlated information to the I/O consumption during SQL statement execution. This facilitates the identification and tuning of I/O bottlenecks.

• Interpoint (SAP) Agent -- The Indepth™ Interpoint agent records SAP application component and

SQL origin data for SAP applications at a very fine level of granularity enabling SAP application problems to be quickly pinpointed and solved.

• Historical Files -- Server side flat files that are used to store short-term performance data for

display and analysis with the Indepth™ for DB2 user interface prior to file data being loaded into the Indepth™ for DB2 Performance Warehouse (data is available in these files until overwritten when the user-defined maximum file size is reached. Prior to that point, data is loaded into the Performance Warehouse, based on user-specified intervals).

• Batch Controller Agent – Controls the batch process used to summarize data from the historical

files and to prepare the files for the loader to archive information into the Performance Warehouse. The Performance Warehouse loader batch controller component is used to load the data from historical files into the Performance Warehouse.

• Performance Warehouse – A collection of DB2 tables that contain historical application and

database performance related data, SQL statements, and DB2 resource activity data (the Performance Warehouse need not reside in the same instance as the monitored production database, but needs to be cataloged so it can be accessed by the Performance Warehouse loader).

Refer to Figure 1-2 for an overview of the Indepth™ for DB2 Client User Interface.

3

Figure 1-2. VERITAS Indepth™ for DB2 Client graphical user interface The Indepth™ for DB2 GUI consists of several toolbars, tabs, and workspaces. I found the GUI very intuitive and easy to use. Data is displayed and analyzed using the concept of a “workspace.” There are six workspaces and we will discuss how each allows you to identify and solve performance problems in detail in this paper. The user interface is also used to install the server-side code and agents, to configure the database monitoring and collection intervals, and for scheduling the recurring load of the Indepth™ for DB2 Performance Warehouse. A key component of Indepth™ for DB2 setup is the Agent Manager. The Agent Manager is configured by selecting the Tune option from the Tune toolbar, and then selecting the Agent Manager option. Refer to Figure 1-3 for an overview of the Agent Manager collector agent setup process.

4

Figure 1-3. Indepth™ for DB2 Agent Manager collector setup The agent manager collector properties tab contains the following attributes:

• Number of samples per second – this attribute is used to specify the number of samples (which consist of various DB2 snapshot commands) to be taken per second.

• Time slice size – The default time slice size is 15 minutes providing 4 intervals per hour. This means that data will be summarized for presentation in 15-minute time intervals. You can specify a combination of time slice and sampling rate to fit the needs of your database environment.

• Number of historical files – specifies how many OS files will be used to store monitored data (these files are used by the Recent Activity Workspace and will be described in more detail under that section).

• Maximum size of each historical file – specifies the maximum size of each historical file used to store monitored data and indirectly determines how much data can be stored before the data is wrapped to the beginning.

• Number of samples per hour – specifies how frequent the “get snapshot for bufferpools on <database name>” command is issued.

• Keep recent activity for at least – this attribute is used to control how long data is retained in the historical files for display in the Recent Activity workspace.

NOTE: We used a TPC-D database and the DB2 provided SAMPLE database and scripts to generate the activity and workloads for this paper. The operating system was AIX 5.1 and we used DB2 V8.1 FP2. We also used Windows XP as the client operating system. Indepth for DB2 operates in the same way across all supported environments (AIX, Solaris, Linux, HP-UX, Windows NT/2000).

5

2. Current Activity Workspace The current activity workspace is selected by a left mouse-click on the current activity workspace. This is how all workspaces are activated on the client. The current activity workspace displays current connections and includes data such as userid, token, locks held, lock status, unit of work status, and associated SQL. The current activity workspace can be used for point-based monitoring to view current database activity and take action if necessary. For example, this workspace can be used to resolve locking issues involving current applications. Refer to Figure 2-1 for an example of how to use the current activity workspace to resolve locking issues.

Figure 2-1. Current Activity Workspace As illustrated in Figure 2-1, the session ID with application handle 32 is tagged with a red lock icon in the workspace tree. Under the Current session tab in the workspace is the session data. As indicated in Figure 2-1, the application is in a lock wait status. By selecting the Lock tab as shown in Figure 2-2, you can tell it is waiting on locks held by agent id 480.

6

Figure 2-2. Lock tab (Identification of lock holder) Since we know the agent id of the lock holder, we can drill down (Figure 2-3) to see why it is holding the locks.

7

Figure 2-3. Session details on lock holder Figure 2-3 shows the details of application handle 480, the lock holder that is in a UOW waiting state. As shown in Figure 2-4, by selecting the Text tab, you can see that this is an update statement

8

Figure 2-4. SQL Statement text involved in lock contention Since it has the locks it needs, further investigation is required to determine the exact nature of the problem. It turned out that the user had submitted the update statement from the command line processor with the auto-commit option turned off. The application was actually finished but the locks had not been released. Lock contention problems like this are fairly common in the real world and a tool is necessary to assist DBAs to quickly identify and resolve locking problems. I found the Current Activity workspace to be easy to use and valuable in identifying and resolving lock contention problems. 3. Recent Activity Workspace The recent activity workspace is primarily where current production performance data is analyzed and examined. In enables point-in-time analysis of performance data. The data available for analysis in this workspace comes from data contained in the Indepth™ historical files, prior to them either wrapping around, or being loaded into the Indepth™ Performance Warehouse (which is typically loaded on a nightly basis, but can be specified to load sooner or later based on a user defined interval and schedule). The amount of data available for viewing in the Recent Activity workspace, will depend on the number and size of the historical files – once they are filled they wrap around to the beginning, so you’ll want to use a trial and error procedure to size the historical files to hold data for the period of interest for your environment. The data from the historical files offer the most granular amount of detailed information for use in the Recent Activity workspace. To review current production activity, production DBAs will spend most of their time using this workspace to review current performance, to identify suboptimal SQL, investigate application problems, identify the most active tables and tablespaces, and review bufferpool performance activity and access patterns.

9

Figure 3-1. Recent Activity Workspace Figure 3-1 shows a summary of activity for a 15-minute time slice within the chosen historical time unit. It should be noted that at this point, Recent Activity reads the raw historical files that contain collected performance data. The amount of the data available for viewing in this workspace is controlled by the total size of the historical files. Therefore, proper sizing of the historical files is important to allow for the full analysis capability of the desired interval. A standard practice for most businesses is to size the historical files to contain data for about a month. This enables comparative performance analysis between month-ending processes from within this workspace. You should use a trial and error process to size these files to hold data for the period you need in your environment. You can also use the Recent Activity workspace to analyze SQL and launch explains, which depict in a tree-like format, the access plan used by DB2 for the SQL being explained. In our case we selected a high-cost SQL statement to explain and analyze (Figure 3-2). Indepth™ forDB2 also has a unique feature where you can Playback an explain. With Playback you start at the first step in the explain tree, and playback replays each step in the order of execution and displays the operation, cost, sargability of predicates, and predicates involved with each step.

10

Figure 3-2. High-cost SQL statement You can also use the Recent Activity workspace to conduct an analysis of overall database activity. Figure 3-3 shows a comparison of overall activity across a range of time units. You can use this type of information to get a good view of the type of database activity occurring, and you can then drill down through the “time unit” of interest to get the details.

11

Figure 3-3. Overall activity comparison across time units 4. Performance Warehouse Workspace The summarized hourly data from the Performance Warehouse is best suited for trending analysis conducted mainly from the Performance Warehouse workspace. The Performance Warehouse workspace is primarily used by production DBAs to investigate historical database activity. It can be used to review daily, weekly and historical application performance. There is no time limit associated with the data that is stored in the Performance Warehouse. Companies can store performance data for as long as they need it to support business requirements. A prune function is provided in the Performance Warehouse workspace for you to use to delete data of no further use from the Performance Warehouse. Refer to Figure 4-1 for an overview of the Performance Warehouse workspace.

12

Figure 4-1. Performance Warehouse Workspace -- SQL resource consumption comparison Figure 4-1 shows SQL activity at the database level. In this example, I selected the Performance tab and selected the In DB2 resource consumption comparison selection from the drop down list. Indepth for DB2 then graphed the resource consumption for the top 10 SQL statements. By using the Performance Warehouse workspace, I could identify database activity over time, down to the table level, with all associated SQL activity identified. This is an important feature because tuning changes should generally be made by reviewing performance data over time, not based on single occurrences of a problem or anomalies. It is also a tool that can be used by operations personnel and capacity planners to review database activity and resource usage over time correlated with OS level statistics. The Performance Warehouse workspace can also be used to identify and evaluate bufferpool activity and performance over time as indicated in Figure 4-2.

13

Figure 4-2. Bufferpool analysis by type of I/O over time The Performance Warehouse workspace can also be used to compare bufferpool performance and to identify the most active objects within each bufferpool. DBAs can use this information to identify objects that may be candidates for tuning or reassignment to other bufferpools. Bufferpool hit ratios and object access patterns can also be identified.

14

5. SQL Tuning In the Performance Warehouse workspace, SQL tuning can be conducted in a variety of ways. You can identify the most active bufferpools, tablespaces, tables, indexes, or SQL statements and the workspace automatically displays SQL statements associated with these objects. As can be seen in Figure 5-1, I sorted on “objects by Total in DB2 time descending”, then selected the first user table (DEPARTMENT), and then selected the SQL statement associated with the DEPARTMENT table with the highest rows written to tune.

Figure 5-1. Performance Warehouse Workspace – Analysis of high-cost SQL statement As shown in Figure 5-1, above the SQL statement, I selected the Text button to display the SQL statement text. In Figure 5-2, I selected the highlighted statement, and the workspace tree expanded to show two access plans associated with the DEPARTMENT table.

15

Figure 5-2. Access plan analysis

This indicates that the access plan had changed over time. I chose to view the first access plan and was presented with an explain on the left side of the pane or “workspace tree” as it is known in Indepth for DB2, and on the right side or workspace area, I was presented with the optimized SQL and a description of the operation involved. To view the details of any step in the access plan, you just select that step and the details appear in the workspace. In this example, I selected STEP14 of the access plan. Note that since the Nested Loop Join operation references two tables and their columns, each table and its columns are highlighted in a different color. References to the EMPLOYEE table and its columns are highlighted in blue and references to the SYSIBM.GENROW function table and its columns are highlighted in red. You can review any explain using this method. You can also review and compare the statements to determine why the access plan changed. Indepth™ for DB2 provides excellent online help that describes and provides examples for each product feature. For example, Figure 5-3 shows an example of the online help available for interpreting explains.

16

Figure 5-3. Online help example Online help is accessed from the toolbar by selecting Help and then selecting the topic of interest.

17

Figure 5-4. List of recommended indexes To have the workspace automatically recommend indexes, I selected the Recommend Indexes button on the toolbar and the workspace automatically sent the SQL to the DB2 Design Advisor as shown in Figure 5-4. A list of recommended indexes (for example, in Figure 5-4, the first index recommended by Design Advisor is IDX040129164647125) is then returned. In this example, we had three indexes recommended by Design Advisor. I was then able to view the list of recommended indexes and create a “virtual explain” (Virtual Explain is a feature of Indepth™ for DB2) as if the indexes had been created. In this particular case, the creation of the recommended indexes reduced the statement cost by 90%. I was able to tune this statement in a matter of minutes without leaving the tool, leveraging the excellent integration with Design Advisor. It should be noted that statements can be tuned in this way from within all workspaces in the product. You might be wondering how Indepth™ for DB2 tracks explain information. At install time, there is an option to setup the recurring load of the Performance Warehouse. As part of configuring that process, you indicate to Indepth™ for DB2 whether or not to explain all statements collected after the warehouse is loaded. This is the recommended approach, and does not require the use of event monitors to capture the SQL statements. Since the product does not use event monitors, there is no overhead placed on a production database to capture SQL with Indepth for DB2 (other products typically use statement event monitors to capture SQL statements, which incur additional overhead on the database). You can also select the New Explain button on the toolbar to do an immediate explain of the statement. This is useful if you have made changes to the database configuration that affects query optimization. It should be noted that Indepth™ for DB2 stores summarized data for each unique SQL statement so that duplicate SQL statements (the exact same SQL statement) are not stored. The display does record the number of executions so it is easy to see frequency of execution. History of individual SQL execution frequency and performance data can also be displayed over time from within the Performance Warehouse workspace.

18

6. Database Objects Workspace The database objects workspace is used to display object relationships and their usage patterns. You can use this workspace to identify all SQL that accessed a specific table. This is a key feature that allows a DBA to easily identify SQL associated with the most frequently accessed table or tables in the database. Figure 6-1 demonstrates how you can identify all SQL statements associated with an object. In this example, we have chosen the Employee table. You can also see how the access plan for this SQL changed over time. The initial plan was established on 12/15/2003, which changed by 1/16/2004. You can then review SQL performance by operator breakdown for the Employee table.

Figure 6-1. Database Objects Workspace -- Customer table SQL resource consumption over time A very powerful feature of the Database Statistics workspace is the capability to identify indexes that are not used. Many products have attempted to do this in the past, but Indepth™ for DB2 actually does it. Unused indexes are identified by reviewing explain information for all statements executed. Refer to Figure 6-2 for an example of this capability.

19

Figure 6-2. Identifying Unused Indexes If you request that Indepth™ for DB2 collect and explain all the statements executed in your database, you can be confident that the indexes identified are unused. As with any case that involves dropping database objects, before you drop these indexes you’ll want to check with your application developers or data modelers to make sure that future use of the indexes are not planned. 7. Database Statistics One of the cardinal rules of any type of performance and tuning activity is that tuning changes should be based on representative performance data analyzed over time. As I mentioned previously, this is to ensure that changes are not made based on a single occurrence of activity unlikely to recur or on anomalies that are extreme or rare, which would result in extreme tuning changes that could needlessly allocate and waste database and OS resources. The Indepth™ for DB2 Performance Warehouse contains the necessary data (accumulated over time) and when used in conjunction with the Performance Warehouse workspace, DBAs can expand and drill down through the database object hierarchy to identify problem areas such as:

• Most frequently accessed tablespaces and tables • Indexes used and unused • Bufferpool activity at the bufferpool, tablespace and table level (along with

associated SQL) • All of the above over a period of time

For example, you can identify access patterns that have changed, changes in access patterns for tablespaces and tables; and changes in the I/O mix (synchronous I/O, asynchronous write, or prefetch activity) at the bufferpool and tablespace level – all across the desired time interval. This is important to know as access patterns can change based on data growth, frequency of RUNSTATS and options specified, changes to the

20

optimization class or other changes to DBM and DB configuration parameters. You can use the results of your analysis to develop an updated bufferpool and tablespace strategy, and to conduct trend analysis based on historical data stored in the Performance Warehouse. The Database Statistics workspace can also be used to identify the most active tablespaces and tables in the database by various metrics. Refer to Figure 7-1 for an example of how to identify the most active table by prefetch I/0 comparison.

Figure 7-1. Database Statistics Workspace -- Identifying the most active table by prefetch I/O comparison In this case, the RECOMMEND2 table was identified as the most active table by prefetch I/O. Identifying the top tablespaces and tables in a database is one of the first steps in identifying SQL and objects that need to be tuned. Typically, these top hitters will require 80% of the tuning in a database. Tuning them will take you a long way towards having a top-performing database. The Database Statistics workspace helps to do just that.

21

8. Statement Workshop Workspace The Statement Workshop workspace provides a development environment for application developers and DBAs alike (Figure 8-1). Statements can be stored in cabinets owned by individuals or shared by a group of developers or DBAs. The statements in the cabinets can be organized into underlying folders.

Figure 8-1. Statement Workshop Workspace All the SQL analysis capabilities provided under the Recent Activity workspace are also available in the Statement Workshop workspace. “What-if” drills can be performed on various SQL statements, including explains and the Statement Workspace also interfaces with Design Advisor for index recommendations as shown in Figure 8-2.

22

Figure 8-2. Recommended indexes As with all Indepth™ workspaces, the Statement Workshop integrates seamlessly with all the other Indepth™ for DB2 workspaces. The Statement Workshop workspace can be used by Production DBAs to tune troublesome SQL or to assist application developers in debugging test or production problems. The Statement Workshop is a powerful workspace that should be integrated into all facets of the system development methodology used at your shop. It can improve the productivity of developers and DBAs, and the quality of application performance before they go into production. 9. EMC Storage Extension for EMC and IBM ESS Shark™ (SharkPoint) Agents Symmetrix and Shark agents interface with the associated storage units and pass information to the Indepth™ collector. Both agents gather similar data regarding disk, file system, and DB2 tablespace I/O activity. The combination of Indepth™ for DB2 with the Symmetrix™ and ESS Shark™ agents offers a very compelling total monitoring solution for DB2. The capability to map and relate logical and physical disk activity, tablespace activity, and file system activity offers a very complete enterprise monitoring solution for DB2 databases. The I/O mapping can help DBAs identify and eliminate disk hot spots. Since I/O throughput is critical to database performance, the integration of Symmetrix™ and IBM ESS Shark™ agents with Indepth™ for DB2 provides for a very powerful enterprise solution for DB2 databases.

23

24

Summary In summary, I found Indepth™ for DB2 to be a very powerful and flexible solution for supporting DB2 UDB for Linux, UNIX, and Windows. The product gives DBAs, IT managers, Operations personnel, and system administrator’s real time and historical data from which to identify and correct application or DB2 database design problems. The way it seamlessly interfaces with DB2 Explain and Design Advisor enables suboptimal SQL statements and associated production problems to be easily identified and solved. Furthermore, these same features can be used by application developers during the application development process to develop top performing applications. The statement workshop enables developers to write and analyze SQL and review associated access paths and costs (timerons). Developers can then explain statements, feed the results to Design Advisor, apply Design Advisor recommendations, review virtual explains, and arrive at optimized solutions – all in a test or acceptance environment, prior to moving to the production environment. We have all probably experienced failed implementations due to lack of proper SQL coding and review, or lack of application development analysis tools. Indepth™ for DB2 provides everything you need to prevent these problems from happening. This methodology enables developers to construct applications that will work the first time – instead of requiring intervention after production implementation! DBAs can use these same features to assist developers during application development, SQL reviews or when troubleshooting current or recent DB2 production problems. Gunning Technology Solutions LLC, is the world leader in DB2™ Universal Database for Linux, UNIX, and Windows consulting. Gunning Technology Solutions specializes in Performance Reviews, Tuning Engagements, Database Administration, Quick-Start to DB2 UDB course, Oracle conversions, regional and international DB2 User Group presentations, customized education, and knowledge transfer. Philip K. Gunning, Founder and principal consultant, has many years of IT and DB2 experience and recently authored the DB2 UDB V8 Handbook for Windows, UNIX, and LINUX, August 2003, IBM Press. He is also co-author of the IBM Redbook, DB2 UDB V8 ESE Performance Guide for OLTP and BI, to be published in January 2004. Phil has also written articles for DB2 Magazine and DBAzine. Phil is Co-Chair of the IDUG North American Conference Planning Committee and Chief List Admin of the IDUG worldwide DB2-L listserv where he is an active participant. Phil also provides tips on how to succeed with DB2™ at http://www.gunningts.com. Disclaimer VERITAS and Indepth are registered trademarks of VERITAS Software Corporation or its affiliates in the U.S. and other countries. DB2 is a registered trademark of IBM Corporation. UNIX is a registered trademark of The Open Group. Linux is a registered trademark of Linus Torvalds. Windows is a registered trademark of Microsoft Corporation. Solaris is a registered trademark of Sun Microsystems Corporation. HP-UX is a registered trademark of Hewlett Packard Corporation. Any other trademarks not explicitly stated but referenced in this paper are trademarks of the respective owners.