z/OS Performance Hot Topics - IBM WWW Page Advanced Technical Support z/OS Performance "Hot" Topics...

IBM Advanced Technical Support

z/OS Performance "Hot" TopicsSHARE - Winter ConferenceMarch, 2009EWCPSession: 2500

Kathy WalshIBM CorporationIBM Distinguished EngineerWashington Systems Center

WSC © IBM Corporation, 2009 1

Advanced Technical Support - Washington Systems Center

Trademarks and Disclaimers

HiperSocketsIBM* IBM eServerIBM logo*iSeriesLotus*OnForever*Parallel Sysplex*POWERPOWER5Predictive Failure Analysis*pSeries*

S/390ServerProven*Tivoli*TotalStorage*TotalStorage ProvenVirtualization EngineX-ArchitecturexSeries*z/OS*z/VM*zSeries*System z9*

AIX*AIX 5L*BladeCenterChipkillDB2DB2 Universal DatabaseDominoEnterprise Storage Server*e-business logo*GDPS*Geographically Dispersed Parallel Sysplex

* Registered trademarks of IBM Corporation

The following are trademarks or registered trademarks of other companies.Intel is a trademark of Intel Corporation in the Unites States, other countries or both.Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countriesLinux is a registered trademark of Linux Torvalds in the United States other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries.Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation in the United States, other countries, or both.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

* All other products may be trademarks or registered trademarks of their respective companies.

IBM System z



Agenda

New Performance Support z/OS 1.11 (Preview) and z/OS 1.10

Performance and Capacity Planning TopicsRSU InformationGetmain ChangesWLM RoutingzIIP and zAAP InformationCapacity Planning InfoCycle SteeringCompressionHiperdispatchDumping Performance

AddendumOlder APARs or Performance Information



www.ibm.com/support/techdocs



HiperDispatch improvements for zIIP processorsz/OS CIM server processing will be eligible to run on the System z zIIPs

Java based CIM client apps on z/OS can already use zAAPsNew prefetch capability to provide perf improvements for XL C/C++ applications on z10

Reduce effects of memory latency by beginning to fetch data before it is known to be needed

VSCR improvements in the base z/OS GRS STAR mode sysplex-wide query requestsXML code page supportTCP/IP sockets processingAllocation changed to significantly reduce the storage required by the Eligible Device Table (EDT) for many I/O configurationsNew SMF fields to record more information on use of virtual, real, and aux storage above 2 GB and can help with capacity planning, performance management, and accounting

DFSMS support is planned for solid state drives (also called flash memory) on DS8000 Sysplex Distributor to use new and improved WLM interfaces for more efficient workload distribution in a sysplexed multi-tiered z/OS server environment

z/OS 1.11 Performance Items



Contention Management Phase 3Promote units of work identified by exploiters for longer periods of timePromote to priority of highest-priority unit of work waiting for the held resource Info on promoted workloads recorded in RMF, SDSF DA and ENC panels

CPU Management of zIIPsWLM algorithms for adjusting dispatch priorities extended to zIIP workloads

Manage selected components in service class SYSTEMPrevent inadvertent misclassification of system address spaces regardless of the WLM Policy and protect system at high utilization

XCFAS, GRS, SMSPDSE, SMSPDSE1, CONSOLE, IEFSCHAS, IXGLOGR, SMF, and CATALOG

Support for 10 additional PB delay types and namimg by subsystem

Extract the WLM service definition in XML formatInstall and activate a WLM service definition in XML format via a CIM Server




RSU settings on z10The storage increment size on the z10 has increased from 64 to 256LPARs with RSU values should ensure the change in storage increment size does not adversely impact system performanceExample

z9 had an 8GB LPAR with an RSU value of 64 meant 4GB of the storage was considered reconfigurablez10 with the same 64 RSU value meant 16GB would be reconfigurable

Entire Central Storage allocation Impact is a lot of high impact page movementVery low capture ratios

OA27801New system health check to detect RSU over-specification conditions and issue a non-scrolling message warning of the potential impact



GETMAIN Enhancements in z/OS 1.10As of z/OS 1.10, small GETMAINs in user region private are satisfied from the beginning of the page instead of from the end of the page

Vendor and legacy code may wrongly assumed the GETMAINed area would contain zeroes which may cause overlays or other incorrect processing GETMAIN change was undocumented in z/OS 1.10

There was an undocumented DIAGxx setting 'NUCLABEL ENABLE(IGVGPVTN)' to allow GETMAIN to work as it did previously

The change was made to improve performance of DB2 open processing for a large number of datasets

OA27291 - Behavior of GETMAIN for low private storage poolsDefault behavior will remain the same as previous releasesDIAGxx keyword provided as a documented method to switch to new GETMAIN behavior

VSM USEZOSV1R9RULES(YES|NO), Provide new dispay command and IPCS support to determine the setting



z/OS 1.10 RMF Enhancements

RMF Monitor III provides new reports about spin and suspend locksSuspend Lock report will display the address spaces which hold locks and which are suspendedSpin Lock report will display how often global locks are held and who is spinning

Reporting of lock statistics is intended to help analyze lock contention in the system

RMF Overview Report Selection Menu Selection ===>

Enter selection number or command for desired report.

Basic Reports1 WFEX Workflow/Exceptions (WE)2 SYSINFO System information (SI)3 CPC CPC capacity

Detail Reports4 DELAY Delays (DLY)5 GROUP Group response time breakdown (RT)6 ENCLAVE Enclave resource consumption and delays (ENCL)7 OPD OMVS process data8 ZFSSUM zFS summary (ZFSS)9 ZFSACT zFS activity (ZFSA)10 SPACEG Storage space (SPG)11 SPACED Disk space (SPD)12 LOCKSP Spin locks (LSP)13 LOCKSU Suspend locks (LSU)



zIIPs and VIPADISTRIBUTEPrior to z/OS v1.9 capacity of specialty CPs not used by WLM when calculating weights for:

Sysplex Distributorz/OS Load Balancing Advisor

SERVERWLM - server specific weightsUses composite WLM weights and factors in capacity of specialty CPs based on the portion they are used by the specific server address spaceNo configuration change needed for SERVERWLM

BASEWLMUses composite WLM weights and factors in capacity of specialty CPs based on the portion they are used on each system (whole system rather than specific server address spaces)Uses separate WLM weights from all processor types and combines them based on CS configuration on how each processor type should impact the combined weights

Options to configure the mix of processor types needed by the application workload provided by Comm Server

VIPADISTRIBUTE ... BASEWLM PROCTYPE(CP=3, ZAAP=1,ZIIP=0)

Benefit: Better distribution of work with Specialty CP content to LPARs with the most available Specialty CP capacity

Improves overall throughput and reduce TCOIncreases utilization of zIIPs and zAAPs



WLM Routing ServicesOA27940

PI Factor used for WLM Routing service IWMSRSRS Function=SPECIFIC can have a negative impact on the routing recommendation if a very aggressive but almost unachievable goal is set

This type of goal has a PI which indicates the work is missing its goal even if the work is receiving very good serviceThe PI>1 can cause the routing service to route work away from the server

OA27063 - Doc APARFor z/OS v1.9 and v1.10 IEAOPTxx parameter WASROUTINGLEVEL has incorrect default specified in the MVS Init & Tuning, SA22-7592-xxWASROUTINGLEVEL= 0 | 1

Specify 0This is the default WLM uses most advanced routing algorithm supported by ALL systems in the sysplex

Specify 1 WLM uses the round robin algorithm This option if chosen should be the same for all systems in the sysplex



LPAR

z/VM

LPAR

z/VSE

LPAR

z/VM

LPAR

z/OS

z/VSE LinuxLinux

zAAP zAAP zAAP zIIP zIIP ICF ICFCP CP CP CP CP IFL IFL IFL IFL

LinuxLinuxLinux

Linux ProductionDev/Test and Optional Failover

z/OS and/or z/VSE Production

LPAR

z/OS

LPAR

CFCC

IFL

IBM System z10

z/VM-mode LPARCFCC CMSz/OS

LPAR

z/VM

LPAR

z/VSE

LPAR

z/VM

LPAR

z/OS

z/VSE LinuxLinux

zAAP zAAP zAAP zIIP zIIP ICF ICFCP CP CP CP CP IFL IFL IFL IFL

LinuxLinuxLinux

Linux ProductionDev/Test and Optional Failover

z/OS and/or z/VSE Production

LPAR

z/OS

LPAR

CFCC

IFL

IBM System z10

z/VM-mode LPARCFCC CMSz/OS

New LPAR type for IBM System z10: z/VM-modeAllows z/VM V5.4 users to configure all CPU types in a z10 LPAR

Offers added flexibility for hosting mainframe workloadsAdd IFLs to an existing standard-engine z/VM LPAR to host Linux workloadsAdd CPs to an existing IFL z/VM LPAR to host z/OS, z/VSE, or traditional CMS workloadsAdd zAAPs and zIIPs to host eligible z/OS specialty-engine processingTest integrated Linux and z/OS and z/VSE solutions in the same LPAR

No change to IBM software licensingSoftware continues to be licensed according to CPU type

z/VM-Mode LPAR Support for IBM System z10



In z/OS 1.10 HiperSockets has been enhanced for zIIP exploitation

Specifically, z/OS Communications Server allows the HiperSockets Multiple Write operation for outbound large messages (originating from z/OS) to be performed by a zIIP

Both sending and receiving LPAR experience fewer I/O interruptsAllows the receiving LPAR to process much more data per I/O interruptAsynchronously moves data without blocking the sending applicationMay help reduce general CPU utilization even without zIIP

This capability helps make highly secure, available, virtual HiperSockets networking more attractive

Application workloads based on XML, HTTP, SOAP, Java, etc. as well as traditional file transfer, can benefit from this function by helping to lower general purpose processor utilization of such traffic

Available on z/OS V1.10 and System z10 only

zIIP Assisted HiperSockets for Large Messages

Data DataDataDataData

Multi Write operation

Large message outboundfrom z/OS

Data DataDataDataData

Multi Write operation

Large message outboundfrom z/OS



zAAPs and zIIPS

APAR OA23479 - HIPERIn z/OS 1.9 a change resulted in too few SIGPs to be issued resulting in poor performance on zAAPs and zIIPs

APAR OA23828 - New Functionz/OS 1.8 and aboveWhen XML System Services (XML SS) are called in task mode 100% of the CPU used to parse a document are eligible for offload to a zAAPWhen XML System Services are called in enclave SRB mode:

If the calling SRB was not zIIP eligible the XML SS CPU was not eligible for offloadIf the calling SRB was zIIP eligible the XML SS CPU was offloaded at the same percentage as the calling application

Change enclave SRB processing to allow the XML SS CPU to be 100% eligible for zIIP offload



z10 System Design and Workload Characteristicsz10 has the potential for greater workload variability

z9 ECCPU

1.7 GhzSuperscalar

CachesL1 private 256k i, 256k dL2 shared 40 MBs / bookBook interconnect: ring

z10 ECCPU

4.4 GhzRedesigned pipelineSuperscalar

CachesL1 private 64k i, 128k dL1.5 private 3 MBsL2 shared 48 MBs / bookBook interconnect: star

MEMORY

L2 CACHE

CPU

L1

. . .CPU

L1

CPU

L1

MEMORY

L2 CACHE

CPU

L1.5

. . .CPU

L1.5

CPU

L1.5L1 L1 L1



Workload CharacteristicsWorkload Characteristics

CPUInstructions

MixSequenceBranch characteristics

Task dispatch profileLocked in or chatty

MemorySizeLocality of referenceMultiprogramming level

I/O rate

Current z/OS LSPR workload suite

ODE-BCB-LWASDBOLTP-TOLTP-WMIXED

CPU Use Profile I/O Memory SubsysODE-B heavy appl, light OS light lightCB-L heavy appl, light OS light lightWASDB medium appl and OS light moderateOLTP-T medium appl and OS heavy moderateOLTP-W medium appl and OS moderate stress



Customer production workloads don't look like the LSPR benchmarksInstead use mixes or blends of the different primitives

Most common workload mix is LoIO MixWe use the I/O rate per used MSUs (LoIO Mix) as an indicator for which of the mixes are a most likely match and those which are the leastI/O was an externally available metric to describe workload characteristics of under the cover behavior such as instruction mix, buffer hit rate, etc.)Low I/O was an indicator of CPU intensity, not a guarantee

Processor migrations need to be understood based on the +/- 5% zPCR projections

For example:A system with a expected ITRR of 1.02, actually needs to be positioned as 0.97 of the current capacityAssess the capacity relationship if the processor is actually 3% smaller

z10 Capacity Planning



Capacity Planning Studies

A new workload mix, DI-MIX, is introduced (Data Intensive Mix)Supported in zPCR v5.1d and zCP3000 (07/11/2008)New mix is also a Low I/O workload

Has all the characteristics of LO-IO Mix but it also has the added attribute of intense memory accesses

DI-Mix will provide the most conservative view of capacityMethod to check the outer band (lower) of performance



SMF23 Enhancements

OA22414 - New FunctionFor z/OS 1.8 and beyondSupported on any IBM System z or IBM zSeries Processor

Adds new workload characterization counters to SMF 23 recordsTCB and SRB dispatches per CPU RSM Services intensity by tracking number of getmains, first reference page faults, non first reference faults, page fixes, etcOverall I/O rates

Better workload profiling data to enhance workload-mix selection for LSPR and zPCR



WP101208 - Running IBM System z at High Utilization System z runs a diverse mix of customer applications and exploits the full capability of the underlying hardware while meeting customer SLA

WLM, IRD, PR/SM, Parallel Sysplex, etc.

Can Any Single Workload Be Run At 100% CPU Busy?The simple answer is no

Queuing theory shows under many conditions described by arrival rate and service time distribution, as a server approaches 100% the queue of work waiting to be processed grows exponentially as does response timeHomogeneous, transaction processing workloads behave like this

Is there a growth in CPU time per transaction at higher utilization?The simple answer is yes

Transactions processed on a system share the physical hardware resources (CPs, caches, memory buses) and software resources (z/OS, subsystems) on the systemOne might presume if a transaction rate of X drives a processor at 50% busy, then at 100% the processor would be able to support 2X the transaction rate, however, this will not be the case



System z: High Availability - Cycle SteeringCycle steering mode is entered whenever the Mulit-Chip Module (MCM) has a cooling issue

Often due to a Modular Refrigeration Unit (MRU) failureDone to protect the processor from failure

Processor will reduce cycle speed and notifies operating systemReduction in Speed is typically from 4-6%Higher reported SU/SEC impact are most likely due to not having MCL patch supplied in DR67L MCL211.G40965 in Bundle37

z/OS will adjust to the lower speedIssues ENF 20, Changes SU/SEC value (R72MADJ), internal reactivate the WLM policy, stop/restart intervals, issues a SMF 99 subtype 10 record to audit the speed change, issues IWM063I at each change in processor speed

The MSU rating of the processor is not changedReduction in processor speed seen as temporary error situation



Compression on System z10z9 - Compression dictionary is cached in standard L1 processor cache

Thus can interfer with all other work on the systemz10 - Each CP has a compression engine with a 16K cache

Removes compression impact from other workloads but concentrates any negative impacts to just the compression work unitsCompression work may have a lower ITRR than expected on the z10

z10 CPU Measurement FacilityNew hardware instrumentation facility available on z10 GA2 New z/OS component - Hardware Instrumentation Facility (HIS)

Requires APARs OA25755, OA25750, and OA25773CPU MF provides support built into the processor hardwarePotential Future Uses:

Better workload characterizationISV Product ImprovementsApplication Tuning

Session 2536: WSC Short Stories and Tall Tales, Thurs - 1:30 PM Room 19A



z/OS HiperDispatch ModeImproved dispatching can help reduce the effects of memory latency to improve performance and reduce CPU time

Better access to hardware cacheManage work across fewer logical CPs

Correct dispatch priorities are critical

Processors will not have equal logical share, but rather a vertical share:High: 100% shareMedium: 0-100% share, the remainder after Vertical High shareLow: 0-100% share, not needed for weight and is discretionary

IRD Vary CPU management is disabled and replacedLogical CPs provided based on weight, workload demand, and available capacity

New knowledge of processor topology when dispatching workWorks with PR/SMtm to build a strong affinity between logical and physical processors

Recommendation is to enable with the installation of the z10If HD=NO ensure sufficient capacity to handle the loss of performance

Not supported for z/OS guests under z/VM



z/OS HiperDispatch Mode - MVS BUSYRMF APAR OA24074

Definition of MVS Busy Time is changed to not include PARKED timeMVS Busy is an indication of latent demand due to LPAR weight enforcementIncreased PARKED time is now also an indication of latent demand due to LPAR weight enforcement

MVS Busy Time (Wait Time + Parked Time) Online Time - * 100=

(Online Time - Parked Time) -CPU 2097 MODEL 725 HIPERDISPATCH=YES0---CPU--- ---------------- TIME % ---------------- LOG PROC --I/O INTERRUPTS-- NUM TYPE ONLINE LPAR BUSY MVS BUSY PARKED SHARE % RATE % VIA TPI 0 CP 100.00 65.16 65.61 0.00 100.0 120.7 37.54 1 CP 100.00 74.83 75.19 0.00 100.0 3397 6.45 2 CP 100.00 56.93 57.31 0.00 100.0 1971 2.04 3 CP 100.00 63.33 64.63 0.00 100.0 2566 3.26 4 CP 100.00 70.06 74.21 0.00 50.0 218.9 20.69 5 CP 100.00 0.01 ----- 100.00 0.0 0.00 0.00 6 CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 7 CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 8 CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 9 CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 A CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 B CP 100.00 0.00 ----- 100.00 0.0 0.00 0.00 TOTAL/AVERAGE 27.53 67.39 450.0 8273 5.24



Hiperdispatch and zPCRWhen building the zPCR model from an RMF report zPCR cannot determine the number of actual logical processors unparked in an LPAR when running hiperdispatch=YES

Requires the user to manually determine the number of unparked CPs Using the number reported in the RMF report would overstate the number of logicalsCannot determine the unparked values for other LPARs

IRD users and hiperdispatch=NO environment the logical CPs are correctly determined by zPCR

IRD uses the config CPU command and so the actual environment is changedThe SMF70 record from any LPAR on a CEC can determine the actual logical CPs defined for all LPARs

Warning message is issued when Hiperdispatch=YES systems are read into zPCR



New WLM Toolshttp://www-03.ibm.com/servers/eserver/zseries/zos/wlm/tools/wlmque.html

WLMOPTDisplays the current IEAOPT settingsIdentifies the currently active OPT member, when it was installed, and the settings of all OPT parameters

WLMQUEISPF based tool to display the application environments currently being used on your z/OS systemView the number of started and active server address spaces, and the service classes being used as work queues for the application environmentsTool can be used for any kind of application environment from WebSphere, DB2 or user specified types and applications



TD104728 - z/OS Availability: Managing SVC Dumps to Mitigate Exhausting the Paging Subsystem

Planning is required to ensure proper operation of SVC DumpToo little AUX can cause a system outage or workload issues

Discusses how to set MAXSPACE parameter Shows how to review RMF Paging data to estimate how big a MAXSPACE your system can support

Discusses how big a MAXSPACE parameter is neededImpacts of Common and size of largest address spaces needing to be dumped

Importance of ensuring a well configured AUX subsystem to support a large dump of multiple address spaces, or multiple dumps taken quickly in progression



SVC Dump PlanningAs SVC Dumps grow in size it is important to properly define dump data sets

Especially as CHNGDUMP MAXSPACE parameters increase

Do not specify multi-volume dump data sets as they are not supported by z/OS Applies to both SYS1.DUMPxx and dynamically allocated data setsCan cause SWA corruption in the DUMPSRV address spaceWould only support a dump of 65535 TRKS, any dump requiring more than this would have a dynamic allocation failure

IGD17051I ALLOCATION FAILED FOR DATA SET LSE.SVC090.AMMLP1.D020321.T10 4951.S00002 , PRIMARY SPACE EXCEEDS 65535 TRKS

Specify dump data sets as SMS managed, using a Data Class with the Extended Format attribute and use the Storage Class attribute Sustained Data Rate to get stripes

3 volumes ("stripes") should be sufficient for a dump data set

Additional DocFLASH10182 - z/OS Performance: SVC Dump PerformanceDFSMS Storage Administration Reference, SC26-7402-09DOC APAR - OA22623



Addendum

Older flashes which should still be understood, or make you go Hmmmm. APARs which are still causing issues, even though they are old.




1. Provide automatic CPU promotion for canceled jobsCanceled job may be holding resources needed elsewhere, (storage, ENQ, latch), but if dispatch priority is not high enough Cancel processing cannot run

2. Provide throughput to blocked workloadsDispatch low priority workloads from time to time Helps resolve resource contention for workloads without resource management implemented

IMP 1 System IMP 3 IMP 2 IMP 5

IMP 3

IMP 3

Has resource xNeeds resource x

Getting CPU

Not Getting CPU

High Priority work is now blocked by lower priority work



Blocked Workloads

New IEAOPTxx parametersBLWLTRPCT

Percentage of the CPU capacity of the LPAR to be used for promotionIn tenths of a percent (0.1%)Range: 0 to 200 (0.1% to 20%)Default: 5

BLWLINTHDStarvation threshold in seconds. Amount of time when an address space or enclave has not received CPU service within this time and is considered blockedRange: 5 seconds to 65535 seconds (18+ hours)Default: 20 seconds

RMF Reports updated with informationCPU ActivityWorkload Activity



WSC FLASH10609 - Blocked Workload Support BLOCKED WORKLOAD ANALYSIS BLWLTRPCT (%) 0.5 PROMOTE RATE: DEFINED 14 WAITERS FOR PROMOTE: AVG 0.010 BLWLINTHD 60 USED (%) 4 PEAK 1

PROMOTE RATE: DEFINED - Number of blocked work units which may be promoted in their dispatching priority per second

PROMOTE RATE: USED (%) - The utilization of the defined promote rate during the reporting interval

It demonstrates how many trickles were actually given away (in percent of the allowed maximum) for the RMF interval

APAR OA22443New recommended defaults of BLWLINTHD=20 and BLWLTRPCT=5Applies to z/OS 1.9, 1.8 and 1.7

Recommended for the IEAOPTxx member of SYS1.PARMLIB to not code parameters specifying default values.


Date post:	09-Mar-2018
Category:	Documents
Upload:	trinhduong
View:	238 times
Download:	3 times

z/OS Performance Hot Topics - IBM WWW Page Advanced Technical Support z/OS Performance "Hot" Topics...

Documents