Date post: | 24-May-2015 |
Category: |
Technology |
Upload: | common-europe |
View: | 613 times |
Download: | 1 times |
IBM Power Systems
© 2012 IBM Corporation
IBM i Job Watcher and Wait AccountingImprove your Job Performance with Run-Wait Analysis
Dawn May - [email protected]
© 2012 IBM Corporation2
IBM Power Systems
Agenda
What is Wait Accounting? Why is Wait Accounting important? How to Use Wait Accounting
– General information– Collection Services– Job Watcher
© 2012 IBM Corporation3
IBM Power Systems
What is Wait Accounting?
Wait Accounting is the ability to determine what a job is doing when it is not running– When a job is not running, it is waiting
• But what is it waiting for?– Waits may be normal, some waits are not normal
• Job Wait Accounting helps to determine if a wait condition is a problem
IBM i has instrumented most of the wait conditions– Wait information is collected by
Collection Services and Job Watcher
i Exclusive!
© 2012 IBM Corporation4
IBM Power Systems
What is Wait Accounting? Wait Accounting is used to understand what is happening
when a job is waiting– Wait information is tracked for each job, thread and task on system
A job spends its time in one of three states– CPU
• Time spent dispatched to the processor, active/running– CPU queuing
• Ready to be processed, waiting for a processor to become available– Wait
• Waiting for something or someone, blocked or idle
http://ibmsystemsmag.blogs.com/i_can/2009/11/i-can-tell-you-why-youre-waiting.html
© 2012 IBM Corporation5
IBM Power Systems
Basics of Waiting Two basic types of waits
– Idle: waiting for a work request• Typically not indicative of a problem
E.g., waiting for the “Enter” key to be pressed on a 5250 display session• If a problem, usually external to the machine
E.g., slow arrival of work requests due to communications problem• Possible, but not typical in batch jobs
E.g., waiting for an entry to be placed on a data queue
– Blocked: waits that occur while performing a work request
Blocked waits are the ones we want to take a closer look at● “Outside of CPU usage and CPU queue time, blocked waits are the
reason jobs/threads take as long as they do to complete their work”
© 2012 IBM Corporation6
IBM Power Systems
Run/Wait Signature
CPU CPU queue WaitIdle
Elapsed time
Interactive job run/wait signature
Typical batch job run/wait signature
CPU CPU queue Wait
Elapsed time
© 2012 IBM Corporation7
IBM Power Systems
Why is Wait Accounting important? Wait time is easy to compute, but rarely scrutinized
▪Historically, little instrumentation has existed to measure and provide detail on waits
● i Wait Accounting provides the instrumentation to understand and analyze waits
For batch-type work:Waits = Elapsed time – CPU time
For interactive-type work:Waits = Elapsed time – CPU time – Idle time
© 2012 IBM Corporation8
IBM Power Systems
Basic example Batch job with total run time of 6 hours
CPU120 min
CPU queue70 min
Wait170 min
Elapsed time 6 hours (360 mins)
Run/wait signature
● Potential to run in 3 hrs 10 min if wait time could be eliminated
▪Wait analysis and reduction can be a powerful and cost-effective way to improve response time and throughput
© 2012 IBM Corporation9
IBM Power Systems
Detailing wait time
Determine the components of time spent waiting
CPU CPU queue Wait
Elapsed time
Disk reads Disk writes Recordlocks Journal
© 2012 IBM Corporation10
IBM Power Systems
Then get metrics related to these components of wait time
Detailing wait time - metrics
Disk reads
3,523
42 sec
0.012 sec
Disk writes
17,772
73 sec
0.004 sec
RecordLocks
355
45 sec
0.126 sec
Journal
5,741
44 sec
0.007 sec
Total count
Total time
Avg timeper wait
▪Even at this level, we can already see possible questions to ask:● How many of the reads are page faults? Could memory/pool changes help?● What programs are causing reads? Could they be reduced or made async?● What programs are causing writes? Could they be reduced or made async?● What DB2 files are involved with the record locks?● What files are being journaled? Are journals needed and optimally configured?
© 2012 IBM Corporation11
IBM Power Systems
Expected vs. Unexpected Waits
Some waits are “expected” and others “unexpected”– In most cases, “normal” and “abnormal” are more accurate descriptors
• Example: a record lock may be normal, but one that lasts for a very long duration is abnormal
– Regardless of the type of wait, it is always better if wait time can be minimized or eliminated
– There are a few block points on the system that are almost always considered abnormal
© 2012 IBM Corporation12
IBM Power Systems
Holders versus Waiters
IBM i keeps track of who is holding a resource, and if applicable, who is waiting to access that resource
– A Holder is the job/thread/task that is holding the serialized resource
– A Waiter is the job/thread/task that wants to access the serialized resource
IBM i also maintains call stacks for every job/thread/task
The combination of wait buckets, information regarding holders/waiters, and call stacks can provide a very powerful solution for analyzing wait conditions
Job Watcher is the tool to accomplish this– Collection services can tell you about waits, but not about holders/waiter nor call stacks
© 2012 IBM Corporation13
IBM Power Systems
Wait Accounting - “Wait Buckets”
Licensed Internal Code (LIC) has identified points where waits occur and assigned a numeric identifier to each point, also known as a “block point”
This level of detail poses challenges to efficiently evaluate the wait time in a job– Difficult to fully understand hundreds of block points– Huge amounts of storage would be needed to track counts and times for every
block point in every job
For this reason, block points are assigned to categories commonly referred to as “wait buckets”
“Mapping” refers to how block points are assigned to buckets
© 2012 IBM Corporation14
IBM Power Systems
32 Wait Buckets as Defined at 6.1 and 7.1– Time dispatched on a CPU– CPU queuing– Reserved– Other waits– Disk page faults– Disk non fault reads– Disk space usage contention– Disk op-start contention– Disk writes– Disk other– Journaling– Semaphore contention– Mutex contention– Machine level gate serialization– Seize contention– Database record lock contention
– Object lock contention– Ineligible waits– Main storage pool overcommitment– Classic Java user including locks– Classic Java JVM– Classic Java other– Socket accepts (idle)– Socket transmits– Socket receives– Socket other– IFS – PASE– Data queue receives– Idle / waiting for work– Synchronization Token contention– Abnormal contention
© 2012 IBM Corporation15
IBM Power Systems
Wait Bucket Example
Database Record Lock Contention – Bucket 16
– Several different causes for waits in this bucket• Read• Update• Weak• Transfer• Check• Conflict Exit
© 2012 IBM Corporation16
IBM Power Systems
Collection Services
As of 6.1 Collection Services collects all 32 wait buckets When Create Performance Data (CRTPFRDTA) runs, data is
written to files – QAPMJOBWT - wait bucket data file
• Information that accounts for the time a job, task, or thread spends waiting● Each job, task, or thread that consumed CPU during the interval
• Counts and clock times for each bucket• Current wait: time waiting and bucket to be used when wait is satisfied
– QAPMJOBWTD - wait bucket description file• Descriptions of the counter sets found in the above file – in unicode
– QAPMJOBWTG – wait conditions not available in the QAPMJOBWT file• One record for each job, task, or thread that did not consume CPU during the interval
Note: The Information Center also describes ‘buckets’ as ‘groups’ or ‘sets’
© 2012 IBM Corporation17
IBM Power Systems
Understanding “Time Dispatched on a CPU” Time dispatched on a CPU (Bucket 1)
– Thread or task has been assigned to a processor and is NOT waiting– Complicated by certain features
• Hardware Multi Threading (HMT) ● Allows multiple threads/tasks to be assigned to a single physical processor ● Causes bucket 1 time to be greater than actual CPU time
• Background assisting tasks● Promote their CPU usage back into the client job/thread● Causes client thread’s bucket 1 time to be smaller than measured CPU time
• LPAR shared/partial processors● Bucket 1 records time dispatched to the virtual processor● Bucket 1 time may be greater than CPU time because it may include time the
thread/task is waiting for the physical processor behind the virtual processor
Bucket 1 does NOT equal CPU Time
© 2012 IBM Corporation18
IBM Power Systems
Understanding “CPU Queuing” CPU Queuing (Wait Bucket 2)
– Thread or task has been assigned to a processor and is waiting for the CPU to become available
• Too much work on the partition causing threads to need to wait for the processors
• Spiky workloads
• Shared processors
Latency due to hypervisor sharing the physical processors among multiple partitions
© 2012 IBM Corporation19
IBM Power Systems
Waits that Applications may use
Disk waits Semaphores, Mutexes, Synchronization Tokens Journaling Database record locks Object locks Sockets
© 2012 IBM Corporation20
IBM Power Systems
Collection Services vs Job Watcher
Collection Services and Job Watcher both collect wait information– Both provide graphs that show waits– Collection Services runs by default in 6.1 and later– Job Watcher data is generally collected when additional information is
necessary to analyze a problem
Job Watcher graphs have better drill-down capabilities– Display graphs that have the data sorted by wait types– Collection Services can accomplish this via table views
Job Watcher can also collect call stacks and SQL statements – Provides additional information for detailed analysis
© 2012 IBM Corporation21
IBM Power Systems
IBM Systems Director Navigator for iJob Watcher
Job Watcher GUI is included with IBM i Performance Tools, Job Watcher Feature
From GO LICPGM:
– 5761PT1 *COMPATIBLE IBM Performance Tools for i5/OS – 5761PT1 *COMPATIBLE Performance Tools - Manager Feature – 5761PT1 *COMPATIBLE Performance Tools - Agent Feature 5761PT1 *COMPATIBLE Performance Tools - Job Watcher
© 2012 IBM Corporation22
IBM Power Systems
IBM Systems Director Navigator for i
IBM Systems Director Navigator for i is the Web console for managing IBM i
– Much of the function that exists in System i Navigator, but with a browser user interface
– Simply point your browser to http://systemname:2001
© 2012 IBM Corporation23
IBM Power Systems
Prerequisites Users need to be authorized to use the Investigate Data and
Collections performance tasks
Include users on the QPMCCDATA authorization list
Edit Authorization List Object . . . . . . . : QPMCCDATA Owner . . . . . . . : QSYS Library . . . . . : QSYS Primary group . . . : *NONE Type changes to current authorities, press Enter. Object List User Authority Mgt *PUBLIC *EXCLUDE QSYS *ALL X PDI01 *USE PDI02 *USE PDI03 *USE PDI04 *USE PDI05 *USE PDI06 *USE PDI07 *USE PDI08 *USE PDI09 *USE More...
© 2012 IBM Corporation24
IBM Power Systems
Prerequisites, continued
To see the Job Watcher definitions through the GUI, users need authority to the QAPYJWDFN file in QUSRSYS– File is shipped with *PUBLIC *EXCLUDE
© 2012 IBM Corporation25
IBM Power Systems
How do I run Job Watcher?
© 2012 IBM Corporation26
IBM Power Systems
IBM-Supplied Job Watcher Definitions Several pre-defined Job Watcher definitions are available
– The main difference is the sample intervals• Q1SEC* – 1 second intervals with call stacks, another to include SQL• Q10SEC* –10 second intervals with call stacks, another to include SQL• Q5SEC* – 5 second intervals with call stacks, another to include SQL
© 2012 IBM Corporation27
IBM Power Systems
Adding a Job Watcher Definition
© 2012 IBM Corporation28
IBM Power Systems
Starting Job Watcher
© 2012 IBM Corporation29
IBM Power Systems
How Do I Run Job Watcher with the Commands?
CL Commands also available starting with 6.1– Add Job Watcher Definition (ADDJWDFN) to define the collection
• Identifies the performance data that is to be collected– Remove Job Watcher Definition (RMVJWDFN) to remove a definition
(Note: Job Watcher Definitions can only be displayed through the GUI)
– Start Job Watcher (STRJW) to start the collection– End Job Watcher (ENDJW) to end the collection (optionally)
Prior to 6.1, the iDoctor product is required to collect Job Watcher data
© 2012 IBM Corporation30
IBM Power Systems
Job Watcher Definition Parameters Collection interval
– Interval between retrieval of job/task data– Considerations for frequency of sampling and amount of data collected
Additional data categories– Additional types of data to include in the collection
• For example, call stacks, Java, sockets and TCP data– Allows granularity on the data collected and frequency
Wait-based call stack data– A special case where call stacks can be collected for certain jobs and/or tasks
which have been in an abnormal or conflict wait Job name
– The jobs that will be included in the collection Task name
– The tasks that will be included in the collection TDE number
– The Task Dispatching Elements (TDE) that will be included in the collection Current user profile
– The current user profile of the jobs which will be included in the collection
© 2012 IBM Corporation31
IBM Power Systems
Job Watcher Definition Parameters Subsystem
– The subsystem of the jobs which will be included in the collection Current storage pool
– The current system pool of the jobs and/or tasks which will be included in the collection
Force record write– When data records are written to the database files– At the end of the interval is necessary for real time analysis
Include inactive jobs/tasks– Whether detailed data for inactive jobs and tasks will be collected during the
first interval To file ASP threshold
– Refers to the ASP where the Job Watcher database files are located– The maximum percent of this ASP that can be consumed
System ASP threshold The maximum percent of the system ASP that can be consumed
© 2012 IBM Corporation32
IBM Power Systems
Job Watcher Definition - Conditional parameters
Provides a way to limit the size of the data collection – Can specify parameters to conditionally collect data
• Average, Percentage, Rate, Direct comparisons– Can specify the type of conditional collection
• Per interval checks in every interval● Data will be collected for intervals in which the condition was satisfied
• Trigger checks until the condition is satisfied● Once the condition is met, data will unconditionally be collected
• Until Met collects data unconditionally until the condition is met● Once the condition is met, the collection ends
– Timeout option• How long the collection should run without the condition being met
– Consecutive occurrence count– History size– User exit program
• Program that will be called at the time the condition is satisfied
© 2012 IBM Corporation33
IBM Power Systems
Start Job Watcher Parameters Definition
– Name of the definition that identifies the collection parameters Collection
– The name you want the collection to be known by Library
– Library where the database files will be created Text description
– Description associated with the database file members Replace data
– Whether to overwrite existing database file members with the specified name Collection Interval
– Can override what is specified on the definition End collection
– Determines when data collection will end– Multiple end options may be specified
• Elapsed time, storage consumed, number of intervals● Whichever occurs first will end the collection
Job name, Task name, TDE number– Can override what is specified on the definition
© 2012 IBM Corporation34
IBM Power Systems
Managing Job Watcher Collections
© 2012 IBM Corporation35
IBM Power Systems
Managing Job Watcher Collections …
© 2012 IBM Corporation36
IBM Power Systems
How do I analyze Job Watcher data?
Look for trends in the data
Look for presence of waits– Drill down into wait details
Display call stacks for running or waiting jobs
© 2012 IBM Corporation37
IBM Power Systems
Job Watcher Content Package
© 2012 IBM Corporation38
IBM Power Systems
CPU Utilization and Waits Overview Both Collection Services and Job Watcher have a “CPU
Utilization and Waits Overview” graph as a general starting point for wait analysis
CPU Utilization andWaits Overview
© 2012 IBM Corporation39
IBM Power Systems
Our Mission and Profile Support the IBM Systems Agenda and accelerate the adoption of new
products and solutions Maximize performance of our clients’ existing IBM systems Deliver technical training, conferences, and other services tailored to
meet client needs Team with IBM Service Providers to optimize the deployment of IBM
solutions (GTS, GBS, SWG Lab Services and our IBM Business Partners)
Our Competitive Advantage Leverage relationships with the IBM development labs to build deep
technical skills and exploit the expertise of our developers Combined expertise of Lab Services and the Training for Systems team Skills can be deployed worldwide to assure all client needs can be met
Mainframe Systems
Power Systems
System x & Bladecenter
System Storage
IT Infrastructure Optimization
Data Center Services
Training Services
IBM Systems Lab Services and Training
Successful worldwide history: 17 years in Americas, 9 years in Europe/Middle East/Africa,
5 years in Asia Pacific
www.ibm.com/systems/services/labservices [email protected]
© 2012 IBM Corporation
IBM Power Systems
Key Offerings High Availability Services on Power Systems (including
Advanced Copy Services for PowerHA™ on IBM i) Systems Director Services PowerCare Services Performance and Scalability services (including system,
application, and database tuning) Virtualization Services for AIX® on Power Systems™
Application and database modernization consulting (SOA implementation)
Linux® on Power consulting, custom application development, implementation, and optimization services
Security on Power consulting and implementation services System consolidation and migration service High Performance Computing consulting and
implementation services SAP® on IBM i consulting Power Blades on BladeCenter (including VIOS on i and
blades running IBM i implementation) Smart Analytics services (including DB2® Web Query
implementation and consulting) Public, private, customized and self-paced virtual training Power Systems Technical University
IBM Systems Lab Services and Training Power Services
www.ibm.com/systems/services/labservices [email protected]
Americas, WW Contacts
Mark [email protected], 507-253-1313IBM i
Frank [email protected], 507-253-1354IBM i, High Availability
Karen [email protected], 972-561-6337IBM i Vouchers
Stephen [email protected], 301-803-6199PowerVouchers, Virtualization Program, AIXSWOT/SWAT, AIX
Allen [email protected], 704-340-9165PowerCare
Cindy [email protected], 507-253-6143Power Performance and Scalability Center
© 2012 IBM Corporation41
IBM Power Systems
Performance and Scalability Services
The IBM i Performance and Scalability Services Center can provide facilities and hardware IN ROCHESTER to assist you in testing hardware or software changes
“Traditional” benchmarks Release-to-release upgrades Assess application performance when migrating to a new release of IBM I Stress test your system Determine impact of application changes Proofs of Concept (e.g. HA alternatives; SSD analysis, external storage, etc.) Evaluate application scalability Capacity planning
… all with the availability of Lab Services IBM i experts and development personnel
To request any of these services, submit at:
• http://www-03.ibm.com/systems/services/labservices/psscontact.html
© 2012 IBM Corporation42
IBM Power Systems
Back Up and Reference Information
© 2012 IBM Corporation43
IBM Power Systems
iDoctor versus IBM i 6.1 Performance Tools– http://www.ibmsystemsmag.com/i5/october08/trends/21990p1.aspx
“Web Power”– http://www.ibmsystemsmag.com/i5/november08/administrator/22431p1.aspx
Introduction to Job Watcher Green Screen Commands– http://www.ibmsystemsmag.com/i5/november08/tipstechniques/22521p1.aspx
End-to-End Performance Management on IBM i – a Redbooks publication
– http://www.redbooks.ibm.com/redbooks/pdfs/sg247808.pdf
i Can … Tell You Why You Are Waiting– http://ibmsystemsmag.blogs.com/i_can/2009/11/i-can-tell-you-why-youre-waiting.html
Reference Information for Job Watcher
© 2012 IBM Corporation44
IBM Power Systems
This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area.Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied.All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions.IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice.IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment.
Revised September 26, 2006
Special notices
© 2012 IBM Corporation45
IBM Power Systems
IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 6 (logo), AS/400, Active Memory, BladeCenter, Blue Gene, CacheFlow, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, AIX 5L, Chiphopper, Chipkill, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Purpose File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER7, pureScale, System i, System p, System p5, System Storage, System z, Tivoli Enterprise, TME 10, TurboCore, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml
The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.UNIX is a registered trademark of The Open Group in the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.Intel, Itanium, Pentium are registered trademarks and Xeon is a trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.AMD Opteron is a trademark of Advanced Micro Devices, Inc.Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.AltiVec is a trademark of Freescale Semiconductor, Inc.Cell Broadband Engine is a trademark of Sony Computer Entertainment Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. Other company, product and service names may be trademarks or service marks of others.
Revised February 9, 2010
Special notices (cont.)