VERITAS i3 for PeopleSoft -...

VERSION INCLUDES TABLE OF CONTENTS STYLES 1

W H I T E P A P E R

VERITAS i3 for PeopleSoft

PERFORMANCE TUNING TIPS FOR PEOPLESOFT APPLICATIONS

Copyright 2002 VERITAS Software Corporation. All rights reserved. VERITAS, VERITAS Software, the VERITAS logo, and all other VERITAS product names and slogans are trademarks or registered trademarks of VERITAS Software Corporation in the US and/or other countries. Other product names and/or slogans mentioned herein may be trademarks or registered trademarks of their respective companies. Specifications and product offerings subject to change without notice. June 2002.

2

TABLE OF CONTENTS 1. Introduction ................................................................................................................................................3 2. Server Performance ....................................................................................................................................3 3. Web Server Performance ............................................................................................................................3 4. Java Performance Management ..................................................................................................................4 5. Tuxedo Performance Management ..............................................................................................................6 6. Application Performance..............................................................................................................................7 7. Database Performance................................................................................................................................7 8. Indexes ......................................................................................................................................................8 9. Performance Monitor ...................................................................................................................................9 10. Storage Performance ................................................................................................................................9 11. CPU Performance ...................................................................................................................................10 12. Queries ..................................................................................................................................................10 13. Tempdb ..................................................................................................................................................11 14. Servers and Memory Allocation................................................................................................................11 15. Conclusion..............................................................................................................................................11 16. VERITAS i3 for PeopleSoft .......................................................................................................................12


3

1. INTRODUCTION This paper presents methodologies and techniques for optimizing the performance of PeopleSoft applications. The methodologies that are discussed are intended to provide useful tips that will help to better tune your PeopleSoft applications. These tips focus on tuning several different aspects within a PeopleSoft environment ranging from servers to indexes. You will find some of these tips provide you with a significant improvement in performance while others may not apply to your environment.

2. SERVER PERFORMANCE In general, the approach to application tuning starts by examining the consumption of resources. The entire system needs to be monitored to analyze resource consumption on an individual component basis and as a whole. The key to tuning servers in a PeopleSoft environment is to implement a methodology to accurately capture as much information as possible without utilizing critical resources needed to serve the end-users. Traditional tools used to measure utilizations impact the system being measured and ultimately the end-user experience. Commands like the following provide snapshot data but not without an associated cost. These tools can consume a significant amount of resources so care should be taken when you execute them.

df size

iostat swapinfo ipcs timex netstat top ps uptime sar vmstat swapinfo also glance & gpm The goal of using these native commands is to identify, if and where, a bottleneck is in the server. Is the problem in the CPU, I/O or memory? These native tools provide indicators, but at the same time could skew the results because of the overhead associated with them. Typically, additional third party tools are needed to complete the analysis. The last hurdle companies are facing in tuning the server is making timing decisions on when to upgrade the hardware itself. To do this, much more information needs to be collected and stored in order to understand if an historical spike in resource utilization was a one-time aberration or a regular occurrence building over time. The recommendation is to look at third party vendors for solutions that can collect key performance indicators while minimizing overhead on the system. The collected data can then be put in a repository for detailed historical analysis.

3. WEB SERVER PERFORMANCE The release of PeopleSoft’s Pure Internet Architecture™ introduces new components to PeopleSoft’s architecture—the web server and application server. The application server is where most shops struggle with appropriate sizing. Web servers are used for handling the end-user requests from a web browser to eliminate the administrative costs associated with loading software (fat clients) on individual desktops. The benefit is a significant savings on software deployment costs, maintenance, and upgrades. While the shift from fat clients to thin lessens the administrative burden, it increases the need to ensure the web servers are finely tuned since they will service a large number of clients. The requirement for these web servers to achieve optimal performance is vital due to the mission critical-nature PeopleSoft plays in today’s enterprise.


4

Recommendations for ensuring good performance for your web servers: • Ensure your load balancing strategy is sound • Implement a solution to verify and highlight changes in traffic volumes • Closely monitor the response times to verify that your strategy is optimizing your web servers • Measure and review historical patterns on server resource utilization (see server section above)

When tuning web servers, it is important to build a comparison looking at the response-time each web server is providing in addition to which requests are being serviced. Simple tuning would begin with balancing workloads among servers to achieve optimal throughput.

4. JAVA PERFORMANCE MANAGEMENT Web technologies tend to focus on end-user access, however, the use of these technologies to integrate legacy systems is equally important and often considerably more complicated and costly to achieve. PeopleSoft made the decision to use Java as the middleware of choice to link PeopleSoft with non-PeopleSoft systems—providing end-users a single easy to use GUI (web browser) to navigate through several applications without having to learn multiple application GUIs. For this reason, the introduction of Pure Internet Architecture is a blessing for PeopleSoft customers—but it comes with some risk. PeopleSoft’s Pure Internet Architecture utilizes industry standard internet technologies such as XML and HTTP as a means of exchanging information. In recognition of these standards, J2EE is being used to accomplish this. Web applications employ mid-tier servers to meet stringent demands for scalability, high performance, and high availability. Web servers in the mid-tier handle user requests and provide direct response when only static content is required. When dynamic content is required, user requests are forwarded to application processing logic on the mid-tier—which also coordinates the handling of data requests from back-end data servers or other remote services. With the potential for traffic surges and high numbers of concurrent user requests, the mid-tier has become a critical component of an application’s performance and scalability strategy.


5

Java is the language of choice for development and deployment of mid-tier applications. Java running on the mid-tier is called server-side Java, to distinguish it from Java running on clients. Java 2 Enterprise Edition (J2EE) standardizes the operating environment for server-side Java applications like PeopleSoft. The introduction of Java presents a challenge for PeopleSoft administrators because it is another variable in the performance equation. It requires PeopleSoft Administrators to become proficient in Java overnight. Furthermore, it requires close oversight of overall performance and end-user response time. Using the same performance management tools for both performance testing, as well as operational performance monitoring, greatly facilitates performance analysis. Operational performance monitoring employs the same technology as load testing but must provide additional functions such as: • Monitoring service levels to ensure proper performance while minimizing investment and operating costs • Setting and triggering alarms based on J2EE performance • Responding to tuning and configuration changes made to other systems (e.g., web server, networks, DB, and

remote services) that shift and change the system bottlenecks • Understanding how application performance is affected by changes in operational databases, user behavior,

workload variations and patterns such as time of day and day of week • Establishing accurate trends and forecasts to add the fewest resources at the latest date, while ensuring

confidence in continued peak performance • Preparing for future workload growth by understanding how your application responds to yesterday’s spikes—

today’s spike is tomorrow’s workload • Monitoring remote services for service level agreement compliance Custom interfaces developed using Java will certainly streamline the business process and improve end-user service but, at what risk? While the goal is to minimize customization, individual businesses do make changes to support key business processes with the goal of providing the organization with a powerful competitive advantage. For this reason, having visibility into this critical middle tier is highly recommended.

Starting with the psp.service servlet and ending with the sendrequest servlet, there were hundreds of calls. VERITAS Indepth for J2EE provides visibility into this critical middle tier.

Start


6

5. TUXEDO PERFORMANCE MANAGEMENT Tuxedo is additional middleware PeopleSoft utilizes to manage the following Internet application server services: • Component Processor—Responsible for executing PeopleSoft Components—the core PeopleSoft application

business logic • Business Interlink Processor— Responsible for managing the interactions with third-party systems • Application Messaging Processor—Manages messages in a PeopleSoft system • User Interface Generator—Generates the user interface based on the Component or Query definition and

generates the appropriate markup language (HTML, WML, or XML) and scripting language (JavaScript, WMLScript) based on the client accessing the application

• Security Manager—Authenticates end-users and manages their system access privileges • Query Processor—Executes queries using the PeopleSoft Query tool • Application Engine—Executes PeopleSoft Application Engine processes • Process Scheduler—Executes reports and batch processes and registers the reports in the Portal’s Content

Registry • SQL Access Manager—Manages all interaction with the relational DBMS via SQL This Tuxedo middle tier is another critical and influential component of performance. Similar to the web server, what is needed is a way to see into the “black box” to further understand some of the key performance metrics. Some of the performance metrics you want to capture when analyzing tuxedo are: • Transaction volumes by domain, server, and application • Response time for each end-user request • Tuxedo service generating a poor performing SQL statement • Break down of Tuxedo time by Service time and Queue time • Identify problem origin – is it in tuxedo or the database? • Response time comparisons for multiple Tuxedo Server Our experience has shown that too often companies throw hardware at a Tuxedo performance problem when a more effective solution can be as simple as adding another domain to the existing server(s). This is due to the fact that PeopleSoft and Tuxedo lack management solutions that provide historical views of performance.

Opening the Black Box to see what processes were running during a time interval you are interested in provides key information such as which Tuxedo processes and services were utilized


7

6. APPLICATION PERFORMANCE It is an accepted fact that 80% of application and database problems reside in the application code. But, there are other technical items to consider which could influence the applications performance. Here are some specific items to focus on when evaluating your database environment: • Make sure the database is sized and configured correctly • Make sure that the hardware and O/S environments are set up correctly • Verify that patch levels are current • Fix common SQL errors • Review documentation of known problems with PeopleSoft supplied code • Be sure to check available patches from PeopleSoft that might address the problem • Review PeopleSoft’s suggested kernel parameters • Set up the right number of processes • Review the application server blocking for Long Running Queries • Make sure you don’t undersize your version 8 application server It is also recommended to continue to review these items on a periodic basis. 7. DATABASE PERFORMANCE The performance of an application depends on many factors. We will start with the overall general approach to tuning SQL statements. We will then move to such areas as indexes, performance monitoring, queries, the Tempdb (Tempdb is often referred to as plain “TEMP”), and, finally, servers and memory allocation. To understand the effect of tuning, we must compare ‘time in Oracle’ with ‘request wait time’. Request wait time is the time that a session is connected to Oracle, but not issuing SQL statements. In Oracle time shows the amount of time resolving a SQL statement once it has been submitted to Oracle for execution. If time in Oracle is not significantly smaller than the request wait time, then application tuning should be examined. Request wait time is almost always much greater than in Oracle time, especially for on line users, because of think time. One exception to this is for a batch job that connects to Oracle and submits SQL statements, then processes the returned data. A greater ratio of request wait to Oracle could indicate a loop in the application outside of Oracle. This should be identified and eliminated before continuing the performance analysis. The next step focuses on tuning the SQL statements that use the most resources. To find the most resource-consuming SQL statements, the scheduled collection approach can be used. The duration time is a commonly used criteria to locate the offensive SQL statements. Other useful criteria include the following wait states: I/O, row lock, table lock, shared pool, buffer, rollback segment, redo log buffer, internal lock, log switch and clear, background process, CPU, memory and I/O. For each offensive SQL statement, the execution plan and database statistics are analyzed. The following statistics are important: table and column selectivity, index clustering factor, and storage parameters. First, all the joins of the SQL are considered. For each join, the ordering of the tables is analyzed. It is of major importance to have the most selective filter condition for the driving table. Then, the type of the join is considered. If the join represents a Nested Loop, forcing it into a hash join can be advantageous under some conditions. The analysis stage usually results in several modification proposals, which are applied and tested in sequence. Corrective actions include database object changes and SQL changes. The typical database object changes are: index change, index rebuild and table reorganization. The typical SQL changes are: replacing subquery with a join, splitting a SQL into multiple SQLs, and inserting Oracle hints to direct the Optimizer to the right execution plan.


8

For a packaged application, object changes and hints are commonly used. To finalize object changes, impact analysis is performed. The goal of impact analysis is to identify how the performance of other SQL statements is changed with the object modifications. VERITAS Indepth for Oracle’s Impact Analysis functionality can aid in impact analysis. To validate the impact analysis results with a high level of confidence, re-running the application processes in the QA database is needed.

When you collect all of the necessary performance information for each SQL statement it makes it easy to comprehend which statements are creating bottlenecks by resource. In this example psappsrve.exe program is waiting for I/O while Pss_tuner.81.exe is spending a large amount of time waiting for CPU.

8. INDEXES Tuning indexes is another important factor in improving performance in a PeopleSoft environment. Index maintenance is crucial to maintaining good database performance. Statistics about data distribution are maintained in each index. These statistics are used by the optimizer to decide which, if any, indexes to use. The statistics must also be maintained so that the optimizer can continue to make good decisions. Thus, procedures should be setup to update the statistics as often as is practical. Keep in mind that objects that do not change, do not need to have their statistics created again. If the object has not changed, the stats will be the same. In this case, recreating the same statistics over again will waste resources. Since PeopleSoft uses a lot of temp tables that are loaded and then deleted, but not dropped, it is helpful to create the statistics when those tables are full of data. If the statistics are created when the table is empty, the stats will reflect that fact. The Optimizer will not have correct information when it chooses an access path. Periodically, indexes should be rebuilt to counter index fragmentation. An index creation script can be created via PeopleTools to drop and rebuild indexes. This procedure will eliminate index -wasted space on blocks that are created as a result of Oracle logical deletes. This is only necessary on tables that are changed often (inserts, updates or deletions).


9

Index scheme is also important to look at. The indexes in a standard PeopleSoft installation may not be the most efficient ones for all installations. Closely examine your data’s pattern, distribution, and modify the indexes accordingly. For example, the index on PS_VOUCHER (BUSINESS_UNIT, VOUCHER_ID) could be changed to (VOUCHER_ID, BUSINESS_UNIT) for an implementation with only a few business units. Use ISQLW Query Options (Show Query Plan and Show Stats I/O) to determine the effectiveness of new indexes. However, be careful to thoroughly test the new index scheme to find all of its ramifications. VERITAS ’s Simulate Change functionality is designed to assist the DBA with this analysis.

9. PERFORMANCE MONIT OR In this section we will discuss some performance monitor counters that might be useful to look at. First, look at ‘compiles per second’. If you are seeing a large number of compiles per second after the database has warmed up and all types of transactions have been issued against the database since starting up, this needs to be investigated. For example, on a 4-way box, if you are seeing 50 compiles per second, this is probably too high. Ideally, after a warm-up period, you should not see many compiles at all. If this is a problem, consider moving up to a release later than 7.65. The later releases have a feature that allows compiles of statements coming from PeopleCode functions ‘SQLExec’ and ‘*ScrollSelect*’ to be eliminated. However, this requires manually modifying all such function calls.

10. STORAGE PERFORMANCE If you are seeing some of the disks being used heavily on a consistent basis, I/O may be a bottleneck. VERITAS measures I/O wait directly. There is no need to make an assumption correlating heavy usage with poor performance. In fact, heavily used disks are often not the problem. If you can see a correlation between periods of heavy disk usage causing the CPU usage to drop drastically, I/O is the bottleneck, and this needs to be investigated. Some solutions may include striping across multiple disks and controllers, separating data files, separating indexes, separating tables that are frequently used together in a join, log files, and Tempdb. Some recommendations on disk layout to consider: • VERITAS Indepth for Database (DB2, Oracle, SQL Server identifies the files that are subject to high waits. You

may then associate to the SQL statements executing against objects in those files to determine the objects that need to be separated

• Verify tablespaces, indexes, etc. that are used together are located on separate spindles • Ensure you have sufficient cache on the disk array • Consider high performance connectivity such as fibre channel


10

VERITAS Indepth for databases provides views on I/0 to help isolate the root cause of performance degradation. In this example, VERITAS Indepth for Oracle displays the logical data files sorted by I/O wait highlighting opportunities to reduce I/O by moving objects.

11. CPU PERFORMANCE Lastly, the CPU should be addressed. If your machine is at 100% CPU utilization for extended periods of time, it would be worth adding additional processors to your machine, getting faster processors, or tuning problem queries that use excessive amounts of CPU time. Also, if you are running an application server on your database server (logical 3-tier), moving the app server to a separate box should help reduce your CPU bottlenecks. If you find your application is often waiting for CPU, it may be an indication that the server is not sized properly. The goal is to always run the application at maximum CPU utilization while meeting the desired service levels. Even if you have a database server running at 100% utilization, there may still be ways to make the SQL statements’ access path more efficient. Appropriate system configuration combined with indexes that are tailored for the SQL statements that your users are executing is key to tuning PeopleSoft applications. Managing the database’s Optimizer so that it tends to choose the most efficient access paths provides a high performance environment for the application and users to ensure SLAs are met.

12. QUERIES It is a good idea to examine queries to try and fix a problem that is affecting the application. Query analyzer can be used to see optimizer plans of slow SQL statements. Choose “Query/Display Plan” to see a graphical representation of a query plan. Alternatively, by issuing a “set showplan_text on” and running the statement you will get a textual representation of the plan, showing indexes used, the order in which the tables were used, etc. When investigating queries, worktables created per second should also be addressed. If you see a large number of work tables being created per second (i.e. hundreds per second), this means that a large amount of sorting is occurring. This may not be a serious a problem, especially if it does not correspond with a large amount of I/O. However, performance could be improved by tuning the queries and indexes involved in the sorts and, ideally, this will eliminate some sorting


11

13. TEMPDB To ensure that your application is performing at peak efficiency, it is important to look at the tempdb. The tempdb is used for sorting result sets, either because of an ‘order by’ clause in a query or to organize pre-result sets needed to execute a given query plan. If tempdb is being used extensively (evidenced by many work tables being created per second or heavy I/O to tempdb files), performance can be improved by tuning it. First, consider moving the tempdb to its own set of disks. Do this with ‘alter database’ using the ‘modify file’ option to specify a new location for tempdb’s data file and log file. It may also be worthwhile to increase the SIZE option to a larger value, such as 100MB and increase the FILEGROWTH option to around 50MB. Another option to consider is adding several data files to tempdb rather than having just one. This will help reduce contention on the tempdb. Do this by using ‘alter database’ using the ‘add file’ option. As with tempdb’s original data file, increase the SIZE option to a larger value as well as the FILEGROWTH option. 14. SERVERS AND MEMORY ALLOCATION The use of an application server is strongly recommended for all on-line connections. The application server queues incoming requests and dramatically reduces process blocking in the database. This will not help batch processes, but it will greatly increase the number of on-line users. Collecting CPU wait, memory wait, and I/O wait may show the application is having to wait on server resources. Typically, this indicates an undersized server or that other applications running on the server were hogging resources. Today, many IT organizations are looking at server consolidation to reduce the cost of ownership. Taking this approach puts you in a position to analyze performance over time as an aid to a server consolidation effort. While it is possible to share a PeopleSoft database server with other applications – database or otherwise – it is always preferable to dedicate the entire server to the PeopleSoft installation. Any process running on the server may use resources that could be better utilized by the database engine or a PeopleSoft process running on the database server. Use of the database server as a file server can seriously degrade the database response time, thus it is important to dedicate the entire server to PeopleSoft processes. The single greatest determinant of database server performance is the amount of memory allocated to it. The memory configuration parameter is expressed in 2K blocks. Thus, if you wanted to allocate 100 MB you would set the memory to 51200. Generally, the more memory allocated to a server the better it will perform. The goal is to add enough memory to increase performance, but not so much that it no longer helps. This determination can be made through the NT Performance monitor. Monitor the Cache Hit Ratio and disk usage via the Performance monitor to determine if more memory should be allocated to the database engine. For the most part, it is better to have too much memory allocated than not enough with database servers. For Application servers, additional memory usually helps but having too much could have a negative effect since more memory correlates to longer search times when there is an operation frequently looking for an object in memory.

15. CONCLUSION The presented methods are intended as tips to help better tune your PeopleSoft applications. These tips are simply suggestions, as mentioned earlier, and they need to be used with caution as each tip may not apply directly to your situation. However, if used properly, the suggested tips can tune your PeopleSoft applications to perform at an optimal level. There are many native tools available to monitor the various components that make up the PeopleSoft landscape. How effective they are in identifying the root cause of performance problems is still in question. The ultimate goal is to find a single solution to gain visibility end-to-end and in between.


12

At VERITAS, our goal is to combine best of breed products into a well-integrated suite to collect all of the data necessary from all tiers and closely correlate this information to the application and end-user experience.

16. VERITAS i3 FOR PEOPLESOFT VERITAS i3 for PeopleSoft is the leading application performance management solution for PeopleSoft environments. VERITAS i3 for PeopleSoft is an integrated suite of “best-of-breed” products that supports a time-tested methodology to significantly improve PeopleSoft performance by cross-correlating performance metrics from all components in the application infrastructure. By continuously monitoring all components of the application infrastructure, VERITAS i3 for PeopleSoft helps "detect and correct" performance degradation before it affects response times and the overall end-user experience. VERITAS i3 uniquely correlates the application flow across the multi-tiered IT infrastructure providing guided steps to improve service quality and manageability. This enables PeopleSoft Managers to make intelligent decisions on how to address immediate and long-term performance issues. VERITAS i3 unrivaled benefits allow PeopleSoft users to: • Generate an accurate, detailed picture of PeopleSoft performance • Monitor each tier supporting PeopleSoft without impeding performance • Collect performance data continuously • Gather detailed performance data for all sessions • Complete PeopleSoft trend analyses and capacity planning forecasts • Alert and report against exceptions and deviations from normal • Provide historical performance data for after the fact in-depth performance analysis


13

VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA 94043 650-527-8000 or 866-837-4827

For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at www.veritas.com.

Date post:	17-Apr-2018
Category:	Documents
Upload:	leanh
View:	216 times
Download:	1 times

VERITAS i3 for PeopleSoft -...

Documents