© 2012 IBM Corporation zCP3000 Sysplex Demo & Lab Gretchen Frye IBM ATS System z Masters Series –...

© 2012 IBM Corporation

zCP3000 Sysplex Demo & Lab

Gretchen Frye

IBM ATS System z Masters Series – Sept 10-13, 2012IBM ATS System z Masters Series – Sept 10-13, 2012


TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.

Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*BladeCenter®, DB2®, e business(logo)®, DataPower®, ESCON, eServer, FICON, IBM®, IBM (logo)®, MVS, OS/390®, POWER6®, POWER6+, POWER7®, Power Architecture®, PowerVM®, S/390®, Sysplex Timer®, System p®, System p5, System x®, System z®, System z9®, System z10®, Tivoli®, WebSphere®, X-Architecture®, zEnterprise®, z9®, z10, z/Architecture®, z/OS®, z/VM®, z/VSE®, zSeries®


Looking at Sysplex using zCP3000

• Not covering sysplex concepts

• Not covering zCP3000 methodology

Focus on• Newer topics

• Things people seem to get stuck on

• Things that could make a big difference


Model data should include …

All systems participating in all sysplexes.

• Combine production and test/dev traffic on Infiniband

• Eligibility for Sysplex Aggregation Pricing

• Sysplex level graphs (the blue logical view) enabled only when all sysplex members are present.

• Better chance of understanding “standalone” CFs


Doing a Capacity Planning Study

• Any current performance concerns?

• Which CFs matter, and which don’t?

• Should the study include overview information?

• Are any partitions currently sharing channel paths?

• Are *all* of the mainframes (z/OS and CF) co-located? Is the distance changing?

• If there is a standalone CF, what type of machine is it on, what type of engines is the CF using and how many, and is it sharing?


Sysplex Graph Recommendations

•CF Link Summary

•Service Time for Synchronous Structures

• Request Rate by System over Time

• Request Rate by Request Type and System

•CF Link Summary

•Service Time for Synchronous Structures

• Request Rate by System over Time

• Request Rate by Request Type and System

• Lock Contention % of Requests

• Synchronous Intensity

• Structures with the Highest Queue Time

• Busiest Structures with Queue Time

Probably in every study• zCP3000 main view : Topology

• Sysplex Logical window : Sysplex Topology

• For each CF : CF Summary, Health Check Analysis

Look for areas of interest• Sysplex Aggregation

• CF Structures Table

• CF Link Topology

Drill down in areas of interest• CF Logical Utilization over Time

• Advanced CF (shared ICF)

•Subchannel Contention

• Synchronous Service Time (Link)

PA Mode

PA and CP Mode


Physical View vs Sysplex View

CF23 and its connections to PLEXIO sysplex members are highlighted at left.CF23 and its connections to PLEXIO sysplex members are highlighted at left.

View->PhysicalView->Physical View->Sysplex LogicalView->Sysplex Logical


Sysplex Aggregation Pricing

Pay for software as if it was executing on a single large CPC.


Sysplex Aggregation Pricing

Not currently eligible, but what if COP2 moved to CPCE7C5?


Incomplete Configuration Information

• Red Triangles mean something is not fully understood.

• All InfiniBand CFs will come up this way the first time.

• Standalone CFs will often need further definition.


The Mysterious CF25

This is all we know about CF25:

– CFTYPE="2097-E12"– CFNCPS="01" – SR="59644"

Plus some creative guessing.

If the last 4 digits of the SR match another CEC, this might be the empty ICF partition there.


zCP3000 CF Analysis

CF Window Summary:

CF InformationCF Information

Link InformationLink Information

Structure InformationStructure Information

Error Messages Summary


Effective number of engines

CP3KEXTR Calculated as: CP3KEXTR Calculated as:

(CF Busy + CF wait)

Interval time

When EFFCP < Defined CPsWhen EFFCP < Defined CPs

Shared EnginesShared Engines

Something else is sharing that ICFOr is it a GCP?


Dynamic ICF Expansion Option

Allowed a CF partition to have *both*

shared and dedicated ICF engines.

Unsupported on z196 and above

The Lab Testcase Used

• 1 dedicated and one share ICF in production, one shared ICF in non production on z10

• 2 shared ICFs in for production, one shared ICF for non-production on z196


The effect of sharing ICF engines

z10 + ICB4 + Dyn_ICF_Expansion z196 + IFB12x + shared engines


CF Health Check

•Lock Contention

•False Lock Contention

CF Utilization < SDP

Effective Engines >= 1

Is Current Technology

Subchannel Link% ok

Subchannel Sync->Async

Subchannels online


CF Health Check - CPU

CF Utilization

should always be < SDP

Effective CPs

>1 engine, and the farther from # defined CPs, the worse it is.

Hardware Technology

Current generation Z?

* Also, System Health Check Sysplex rule 2

CF21


CF Health Check - Structures

Lock Contention

Lock Requests that had to wait

Fix in the Application.

False Lock Contention

2 locks hash to the same index

Increase the Structure size.


CF Health Check - Subchannels

Subchannel Utilization should be very low (<10%)

Sync -> Async Conversion to avoid path busy. Normally very low, this is an indicator that partitions may be sharing channel paths.

Subchannels Online

unusual; found when there is a sustained pattern of path busy.


CF Structures table

• Use to identify the busiest structures.

• Use to identify the key CF links for the busiest structures.

In your customer deliverable

• focus on critical applications

• focus on identified problems

• otherwise, use this to identify important structures and links.

CF21


Service Time

• Synchronous is more important than asynchronous

• Service Time graphs available from the CF window, the Link window, and the Structure detail window.

• CF window averages service time for all z/OS members.

• Use this more as a guide to analysis.


Structure Detail

• Request Rate and Service Time

• Synchronous and Asynchronous

• By sysplex member system

• Average, max rate, and for the study interval.

• Graphs:

•Synchronous Service Time

•Asynchronous Service Time


Coupling Links

• An important part of Sysplex performance.

• Often, there is incomplete configuration information:– Link types being used – distance from the CF to the z/OS image– Whether any channel paths are being shared– CF model and relative speed

• Estimated Service Time is both calculated and calibrated to the current service time.

• Heuristic process on z/OS decides how to launch a request (synchronously or asynchronously)


Coupling Links

Service Time is dependent on – Type of Request– Amount of data being carried*– How request will be executed (sync or async)– Path busy or anticipated path busy (queued request)– Link latency– Link speed– Link distance– CF speed and responsiveness– z/OS speed

Sysplex Overhead is dependent on– How request will be executed (sync or async)– Service Time, for synchronous requests

* (not measured)


12x InfiniBand

12x InfiniBand

HCA2-O*

HCA2-C I/O Drawer or I/0 Cage

ISC-3

ISC-3

ISC-3

ISC-3

z196, z114, z10 EC, z10 BC

IFB-MP

z196/z114

ISC-3

z196, z114, z10 EC, z10 BC, z9 EC, z9 BC

HCA2-O LR*

Up to 150 meters

Up to 150 meters

Up to 10/100 km

.... ....

HCA2-O**

.... ....

1x InfiniBand

Up to 10/100KM

z9 EC and z9 BC

Fanouts

HCA3-O HCA3-O LR

.... ....

12x InfiniBand

Up to 150 meters

1x InfiniBand

Up to 10/100KM

** Not supported on the z114 ** Carry forward only** Not supported on the z114 ** Carry forward only

zEC12/z196/z114 Coupling Links• Fanout, not I/O slot, used for InfiniBand

• ICB-4 – No longer supported

• ETR – No longer supported

• All coupling links support STP

• Sysplex Coexistence – z10 EC and BC; z9 EC and BC only with z196,z114


CF Coupling – Relative Link Speed in zCP3000

* ISC3 is supported as carryover only on zEC-12;

* Relative speed does not include latency or any other component of service time other than data transfer time.

zCP3000 Name

Latency microseconds

Relative Speed* Supported on

ISC3 ISC3* 12 (base) z800, z900, z890, z990, z9 , z10 , z114, z196, zEC-12*

IFB-1x IFB 1X 10 2 z9 , z10 , z114, z196, zEC12

IFB-12x IFB 12X – z9 8 3 z9

IFB-12x IFB 12X 8 5 z10 , z114, z196, zEC12

ICB4 ICB4 4 8 z890, z990, z9 , z10

IFB3-12xIFB3 12X – z114 3.5 13 z114

IC IC-z10 BC 1 16 z10-BC

IC IC-2084 1 18 z990

IC IC-2086 1 18 z890

IC IC-z9 BC 1 20 z9-BC

IFB3-12x IFB3 12X 3.5 25 z196, zEC12

IC IC-z9 EC 1 25 z9-EC

IC IC-z114 1 33 z114

IC IC-z10 EC 1 38 z10-EC

IC IC-z196 1 45 z196

IC IC-zEC 12 1 47 zEC-12


Coupling Links

To model a link type change:(estimate new service time)

• Must be in Capacity Planning Mode

• One CF-SYS set of links at a time

• If CEC model change also, do that first.

• Generate the Before/After link change report *before* applying.


Changing the Link Type

• Traffic for each structure will be re-calculated.

• A change in the service time can also mean a change in how the request is issued (sync vs async).

• In this example:– IFB-1X is faster– Because it’s faster, 7k requests will

be run as SYNC instead of ASYNC


CF Link Summary

• Lock structure was already mostly synchronous, and the service time will go from 48.12 to 40.05.

• DB2 cache structures will go from mostly asynchronous to mostly synchronous, so service time will go from 93.73 to 24.89 microseconds.


CF Link Consolidation

Prior to InfiniBand• One link = one chpid = one feature

• A link could be shared because the chpid could be shared, between members of the same sysplex on the same mainframe, going to the same CF

With InfiniBand• The physical link can carry multiple chpids• Chpids can be from the same sysplex or a different one• Each chpid can still be shared by partitions on the same mainframe

going to the same destination CF

zCP3000 Terminology• Link = channel path (chpid)

• Adapter = the physical path

IFB-1x : 8-16 chpids on 4 ports on HCA2-O LR

IFB-12x : 8-16 chpids across 2 ports on HCA2-O

IFB3 : max 4 chpids per port(2) on HCA3-O


CF Link Consolidation

• Minimum two HCA2-O or HCA3-O adapters

• Meaning that each machine will have 16-32 channels paths

• IFB3 is a protocol, not an adapter, requiring– HCA3-O adapter types on both ends– Maximum 4 chpids per port

• Typically multiple physical ICB4 or ISC3 links/adapters/chpids

• Often chpids are shared between 2-3 partitions

• Meaning you really have (chpids/partitions) number of ICB4 links.

Consolidate this

• One for one replacement of non-IFB chpids with IFB chpids

• Many to one replacement of non-IFB adapters with IFB adapters


Incomplete Configuration Information

Red Triangles mean something is not fully understood.

• Currently, all InfiniBand chpids come in with channel type “CIB”.

• Correct link definitions before proceeding.

• With CFCC 18, RMF will be able to differentiate various IFB link types and Detect if CIB link running "degraded“

* Not yet supported in zCP3000 *


CF Link Consolidation – Shared CHPID

•2 or more partitions on the same CEC

• talking to the same CF on another CEC

• same sysplex

• using the same chpid

• same type (ISC3, ICB4, IFB)

CHPID PATH=(CSS(0),F4),SHARED, PARTITION=((DEVJ,DEV1,DEV2),(=)), PCHID=101,TYPE=CFP


Defining Shared Channel Paths

• CP3000 does not know if a link is shared.

• Shared Links table lists the partitions that could possibly share these 6 chpids.

• CF linktype must be fully defined and must match exactly.

You have to do this yourself


Defining Shared Adapters

• 2 or more partitions on the same *pair* of CECs

• talking to any CF on either CEC

• same sysplex, or not

• using different chpids

• which run through the same IFB pipeline.



• InfiniBand only • Shared Links table lists the

sysplex members and CFs on these 2 CECs that could run on the same physical InfiniBand feature.

• *NO* performance implications.

• P2DF could be *both* logical share and physical share.



• 54 chpids running on 54 ISC3 links…

Replace with

• 54 chpids running on 7 IFB adapters.

7


CHPID Sharing

The P1DF system on CEC7B92 shares 6 chpids to the CF05 coupling facility on CEC7BA2 with 0 other partitions.

54 chpids share 14 ports on 7 InfiniBand HCA2-O adapters.

Adapter Sharing


CF Link Topology Report PLEX01:CF05:P1DF


Index to recommended Graphs

• zCP3000 main view– INV1012 Topology (7)

• zCP3000 sysplex view– PLEX1007 Sysplex Topology (7)– PLEX1005 Sysplex Aggregation by Sysid (8,9)

• CF Window– CF1014 CF Summary (12)– CF1000 Health Check Analysis (16-19)– CF1001 Logical Utilization over Time– CF1008 Subchannel Contention– CF1020 Advanced CF Utilization– CF1010 Structures Table (20)– CF1012 Service Time for Synchronous Structures (15,21)– CF1009 Request Rate by System over Time– CF1004 Request Rate by Request Type and System– CF1015 Structures with the Highest Queue Time– CF1016 Busiest Structures with Queue Time

• Structure Window

• STR1015 Lock Contention % of Requests

• STR1017 Synchronous Intensity

• CF Link Window

• CFL001 CF Link Summary (29,34,36)

•CFL005 CF Link Topology

• CFL012 Synchronous Service Time

* See it on Slide (n)


zCP3000 Online Education Other Online Courses available at:

IBM - http://w3.ibm.com/support/americas/wsc/cpsproducts.htmlBPs - http://partners.boulder.ibm.com/src/atsmastr.nsf/WebIndex/PRS1762

Contact: [email protected]


Date post:	04-Jan-2016
Category:	Documents
Upload:	kerry-perry
View:	223 times
Download:	0 times

© 2012 IBM Corporation zCP3000 Sysplex Demo & Lab Gretchen Frye IBM ATS System z Masters Series –...

Documents