Application Development Success Stories for Performance Issues on DB2 … · 2019-12-08 ·...

Application Development Success

Stories for Performance Issues

on DB2 Z/OS

Speakers: Keith Tidball & Lori Zaremba

Progressive Insurance

Session Code: E06

May 4, 2011 9:45am | Platform: DB2 Z/OS

2 Copyright 2011 Progressive Insurance. May not reproduce without permission.

I. About us

II. How we find performance opportunities

III. Performance success stories

IV. Lessons Learned

V. Q & A

AGENDA


About Us: Speaker Background

Keith Tidball

Lead Application Developer, Progressive Insurance 12 years mainframe development experience 9 years experience working with DB2

Lori Zaremba

Lead Application Developer, Progressive Insurance 15 years experience working with DB2 3 years working with SQL Server

Sorry, no Flo this year at IDUG.

But she promised to upgrade her ‘tricked out name tag’ in our

honor, though…


Claims

DMG

Billing Quoting

Payroll

Two Data Centers

Cleveland Colorado Springs

About Us: DB2 at Progressive


546 Databases

10 Way Data Sharing Env.

31,000+ Tables

Several Tables over 1 BILLION Rows

40 Terabytes of data

About Us: DB2 at Progressive


About Us: Progressive Billing System

ProBill DB2 Facts:

• Total programs = 8,023

• Total Tables = 423

• Total Views = 429

• Total Indexes = 637

• Triggers = 16

• Stored Procs = 21

• Sequence Objects = 8

• Production Data = 5 terabytes


More ProBill Facts: • 24x7 real-time payment processing and billing services for

all Progressive lines of business.

• Interacts with customers more frequently than any other of Progressive's services as many upfront systems access & display Billing data.

• Provides complete and accurate view of our insureds' accounts, coupled with rigorous financial controls.

• Supplies billing information to external customers, agents, internal service reps, and financial staff.

About Us: The Progressive Billing System


How We Find Performance Opportunities


DB2 Support Team

Team consists of: • DB2 Developers • Performance Testers • DBA

Meeting Frequency: Weekly

Meeting Objectives: • Review upcoming production SQL changes • Review/Brainstorm current issues • Update DB2 standards as necessary with new releases • Share Tips/Tricks

How we find performance opportunities


Stress testing A. Changes deemed as performance enhancements are stress

tested B. Measure and compare changes to baseline tests:

• MIPS • CPU • Run Time • Wait Time

C. All measurements are reviewed and signed off prior to

installing to production.

D. Post-production implementation performance review.



Performance Instrumentation Database

• Collect data to support the analysis and determination of the causes of performance degradation and performance hits.

• Data originates from:

i. SARQ for batch jobs and job step information

ii. DB2 DBA tables for DB2 program information

iii. Billing infrastructure tables for transaction information

• Compare application performance with historical data.

• Anticipate future performance.

• Report performance of online and unattended processes.

• SQL Server Database using SSRS for reports/charts

*** you will see examples of these reports later in the slideshow…



Post Implementation Issues (yes, they do happen sometimes!)

A. Post implementation performance issues are typically found by:

• DBA/Enterprise Application Performance Engineering groups • Post production implementation performance review • Application development team • Monthly Performance review meetings

B. Once a performance issue is identified – the application team is engaged with the DB2 support team to derive the solution.




Work With Our DBA

A. Our DBA is named Matt and he’s not shy about addressing performance related items when it comes to the ProBill database.

B. We continually work with Matt and other DBA’s to identify performance opportunities.

C. There are two reports that are used by the DBA group and ourselves to help with this process.

i. Query to show SQL statement usage times by program

ii. DB2 MIPS usage by program



SQL Statement Usage by Program

• Query against data compiled from Detector provides a handy report for programs using the most time in DB2.

SELECT DISTINCT(PROGRAM),STMT#,SQL_CALL,

SUM(INDB2_TIME) AS INDB2_TIME,

SUM(SQL_CALLS) AS SUM_SQL_CALLS,

SUM(INDB2_CPU) AS CPU_TIME,

SUM(IO_WTIME) AS IOWAIT,

SUM(LOCK_WTIME) AS LOCKTIME,

SUM(GETPAGE) AS GETPAGE

FROM XXX.PDT_STANDARD_115

WHERE INTERVAL_START BETWEEN '2011-02-10-00.00.00.000000'

AND '2011-02-11-00.00.00.000000‘)

AND PROGRAM LIKE ‘AAA%’

GROUP BY PROGRAM,STMT#,SQL_CALL

HAVING SUM(INDB2_CPU) > 1000

ORDER BY 6 DESC

This query is very

flexible. It’s easily

tailored to be as

general or specific

as you need.



PROGRAM STMT# SQL CALL INDB2 TIME

SUM SQL

CALLS CPU TIME IOWAIT LOCKTIME GETPAGEAAA 541 PREPARE 14,475 54,719,793 8,917 22 2,245 72,071,626

BBB 583 EXECUTE 39,547 10,613,835 3,651 26,078 1,616 117,097,052

CCC 586 FETCH 44,982 54,719,793 3,526 39,350 1,084 198,728,294

DDD 1860 DECLARE GLOBAL T 4,193 450,635 2,949 3 460 123,023,355

111 1551 OPEN 27,025 4,495,301 2,919 21,362 721 101,918,500

FFF 626 OPEN 7,535 5 2,795 1,779 4 14,104,260

333 2128 SET HOST VAR 7,794 15,096,233 1,793 13 890 31,393,421

HHH 672 OPEN 8,068 5 1,651 3,865 2 13,471,089

DDD 1825 UPDATE 3,106 7,548,142 1,550 9 653 105,818,246

WWW 117 SELECT 2,245 31,161,850 1,537 1 339 62,323,690

444 1882 UPDATE 3,094 7,548,128 1,521 42 784 105,817,883

JJJ 815 OPEN 7,124 8,647 1,521 3,690 22 70,130,429

SQL Statement Usage by Program

• Example of Query results ordered by CPU usage:



DB2 MIPS Usage by Program

• SMF data from DB2 Traces in DB2PM gets dumped to flat files and then moved to SAS datasets and DB2 tables.

• ITRM is used to submit SAS against the data and this eventually gets dumped to reports.

• Initially created to charge back MIPS to individual apps. Before it was one large ‘pool’ & costs were spread evenly.

• These reports can give us the following stats by programs: i. MIPS used during ‘Peak Times’ (9a – 11a & 2p – 4p)

ii. # SQL executions

iii. UOW breakdowns (wait times, elapsed times, MIPS, etc)


Top 25 MIPS Daily Consumers for Probill DB2 Packages

M-F, Peak Prime Hours (ranked by last day actuals)

0

500

1000

1500

2000

2500

12

/16

/20

10

12

/17

/20

10

12

/20

/20

10

12

/21

/20

10

12

/22

/20

10

12

/23

/20

10

12

/24

/20

10

12

/27

/20

10

12

/28

/20

10

12

/29

/20

10

12

/30

/20

10

12

/31

/20

10

01

/03

/20

11

01

/04

/20

11

01

/05

/20

11

01

/06

/20

11

01

/07

/20

11

01

/10

/20

11

01

/11

/20

11

01

/12

/20

11

01

/13

/20

11

01

/14

/20

11

01

/17

/20

11

01

/18

/20

11

01

/19

/20

11

01

/20

/20

11

01

/21

/20

11

01

/24

/20

11

01

/25

/20

11

01

/26

/20

11

01

/27

/20

11

01

/28

/20

11

Application groups based on AuthID, PackageID, Collection ID. Select "refresh" for current data.

25 - _OTHER

24 - GS2_BIL

23 - PFN_BIL

22 - QLG_BIL

21 - PPS_BIL

20 - PPA_BIL

19 - INS_BIL

18 - PNA_BIL

17 - BPT_BIL

16 – TAB_BIL

15 - BIV_BIL

14 - BED_BIL

13 - PUB_BIL

12 - PNC_BIL

11 - LRT_BIL

10 - DEL _BIL

9 - BBH_BIL

8 - OAX _BIL

7 - OAD _BIL

6 - CPY _BIL

5 - BMS_BIL

4 - PSD_BIL

3 - AAA _BIL

2 - BBP_BIL

1 - SEL_BIL


Example of MIPS Usage Report

• This report shows the top 25 MIPS consuming programs each day during peak hours for the ProBill Application.

Program Names


Performance Success Stories

OK, some were more successful than others….


EXAMPLE 1:

Success Without Using DB2


PROBLEM:

Contention issues with Critical table that houses rules for all transactions run within the Billing system. Accessed 40+ million times a day by COBOL program (PC04).

Replicated the DB2 table into TableBase and it provided speed & performance needed.

Over the years we ran into a few issues with this model:

• The timing of TableBase refresh was different than that of our programs. • Developers would forget to replicate their changes to it. • ProBill is trying to phase out TableBase usage where possible.

PROPOSED SOLUTION:

Discontinue using TableBase table and use the source table in DB2, leveraging newer features of DB2 that were missing when we first developed the system.

Executed successful stress tests using DB2 version of the table and performance

was on par with TableBase. We shared the results with DBAs and Systems Analysts and no one objected.

All this looked great and it went live in Production….

Example 1


RESULT:

PC04 was performant as expected, but with…

15,000+ seconds of CPU and close to 300 million calls to the DB2 engine daily.

Program PC04 rose to the top of the DBA’s ‘Most Wanted’ list within a week. This resulted in all our stats being reassessed and a call for a non-DB2 solution.

Example 1

Guaranteed Call from

Angry DBA!!


FINAL SOLUTION:

Used a CICS UMT to store the table data and created a more robust replication process from DB2 (via scheduled jobs).

Performance of PC04 is the same, but overall CPU usage down using UMT vs. DB2.

DB2 is still used as failover contingency within PC04 if UMT ever has an issue.

DBA’s now talking with us again (somewhat cordially, too…)

Example 1

Eliminating DB2 Calls =

DBA stopped threatening

us with bodily harm…


EXAMPLE 2:

Success by Archiving DB2 Data


PROBLEM:

Database had 5 Terabytes of data that spanned over 6 years.

Degradation of application program performance against larger tables.

Business users really liked having all the data all the time.

PROPOSED SOLUTION:

Reduce the size of our database.

Identified tables that were really transactional in nature (i.e. data is useful for a set period of time only)

Business Users helped set ‘cutoff’ for how long to keep data.

Example 2


RESULT:

We saw decreases in processing and run time across several programs. Here is one example where DB2 CPU time dropped nicely for program ECD.

Example 2

Ran Reorg w/ Discard

and dropped 65% of the

data rows.

DBA Happy Place


EXAMPLE 3:

Several Attempts Before Reaching Success


PROBLEM:

Batch Job CD77 run time steadily increasing due to called API transitioning from MF to Server.

Quoted

Policy

Data

Populate

needed

reporting info

(CD77 Job)

Quote

Data

API to gather

data from Policy

systems

Existing Policy System was based on the

MF, but its replacement was server

based. Policy volume was slowly growing

in the new system.

Quote

Balancing

Reports

Example 3


Job CD77 run time steadily increasing from 150 minutes to over 300 minutes! (about a 3 month period)

Run time increasing with more calls to server.

Example 3


Solution #1:

Remove calls to API and get data from DB2 instead via queries.

Billing

Tables

Billing

Tables

Quoted

Policy

Data

Populate

needed

reporting info

(CD77 Job)

Quote

Data

API to gather

data from Policy

systems

Quote

Balancing

Reports Billing

Tables

XX

Example 3


Solution #1 Results:

The changes performed poorly in production and had to be demoted.

(Programming Tip: Try to avoid causing HUGE upward spikes in runtime…)

Demoted changes and resumed calling the API.

Example 3


Example 3

Billing

Tables

Billing

Tables

Quoted

Policy

Data

Populate

needed

reporting info

(CD77 Job)

Quote

Data

API to gather

data from Policy

systems

Quote

Balancing

Reports Billing

Tables

X

X

Added one data element

to this table from the

Query that was taking the

majority of time to run

Removed the longest &

costliest call

X

Solution #2:

Cache the data element that was the most inefficient DB2 call and then get remaining data from existing DB2 tables within Billing vs. calling the API.


Solution #2 Results: Better than the first try, but still unacceptable from a performance

standpoint.

Example 3

(Solution #1)

(Solution #2)


Example 3

Solution #3:

Cache ALL data elements and eliminate the SQL & API calls entirely.

Billing

Tables

Billing

Tables

Quoted

Policy

Data

Populate

needed

reporting info

(CD77 Job)

Quote

Data

API to gather

data from Policy

systems

Quote

Balancing

Reports Billing

Tables

X

X

Added ALL data

elements to this table

and eliminated the

queries all together

X

XX

X


Solution #3 Results: Results were impressive. Run times dropped from hours to under 10

minutes! SQL calls and I/O were reduced to almost nil.

Example 3

(Solution #1)

(Solution #2)

(Solution #3)

Third time’s the

charm!!!


EXAMPLE 4:

MIPS for Matt


PROBLEM:

MIPS usage during ‘peak’ hours was driving a need to purchase engine upgrades, but our lease on the mainframes was expiring within 12 mos.

Example 4


PROPOSED SOLUTION:

‘MIPS for Matt’ was born as we felt our DBA was a charity worth supporting fully.

Identified programs for performance improvements by revising or eliminating SQL.

Identified processes that could run during off-peak hours.

Example 4


Example 4

Removed SQL no longer needed

Revised SQL & scheduling

Run during Off-Peak hours


Top 25 MIPS Daily Consumers for Probill DB2 Packages

M-F, Peak Prime Hours (ranked by last day actuals)

0

500

1000

1500

2000

2500

12

/16

/20

10

12

/17

/20

10

12

/20

/20

10

12

/21

/20

10

12

/22

/20

10

12

/23

/20

10

12

/24

/20

10

12

/27

/20

10

12

/28

/20

10

12

/29

/20

10

12

/30

/20

10

12

/31

/20

10

01

/03

/20

11

01

/04

/20

11

01

/05

/20

11

01

/06

/20

11

01

/07

/20

11

01

/10

/20

11

01

/11

/20

11

01

/12

/20

11

01

/13

/20

11

01

/14

/20

11

01

/17

/20

11

01

/18

/20

11

01

/19

/20

11

01

/20

/20

11

01

/21

/20

11

01

/24

/20

11

01

/25

/20

11

01

/26

/20

11

01

/27

/20

11

01

/28

/20

11

Application groups based on AuthID, PackageID, Collection ID. Select "refresh" for current data.

25 - _OTHER

24 - GS2_BIL

23 - PFN_BIL

22 - QLG_BIL

21 - PPS_BIL

20 - PPA_BIL

19 - INS_BIL

18 - PNA_BIL

17 - BPT_BIL

16 - TAB_BIL

15 - BIV_BIL

14 - BED_BIL

13 - PUB_BIL

12 - PNC_BIL

11 - LRT_BIL

10 - DEL _BIL

9 - BBH_BIL

8 - AX _BIL

7 - AD _BIL

6 - PY _BIL

5 - BMS_BIL

4 - PSD_BIL

3 - AA _BIL

2 - BBP_BIL

1 - SEL_BIL

Example 4

RESULTS:

‘MIPS for Matt’ was a success as we ‘raised’ a Peak Hours savings of 800 MIPs in a little over 2 months.

Peak MIPS for the month was barely over 2000 compared with 2900 a couple months prior


EXAMPLE 5:

Limit Trips to DB2


PROBLEM:

Batch Job CD01 had a step that routinely ran for 2+ hours, but it wasn’t doing any serious data processing (Program BL80).

The extended run time often missed the SLA we had for the reports.

Example 5

Asst. Job

Steps

Raw

Report

Data

Program

BL80

Report

Info

Policy

Info

Payment

Info

Financial

ReportAsst. Job

Steps

Asst. Job

Steps

One query was taking

90% of the run time in

the program


SOLUTION:

Program BL80 wasn’t using the most efficient index on the Payment Info table, but BL80 also lacked the data to get a better access path.

Upstream program BL24 also read this table, but used a full keyed read. We cached the data BL80 needed & passed it in BL24 output file.

Example 5

Asst. Job

Steps

Program

BL24

Payment

Info

Raw

Report

Data

Program

BL80

Report

Info

Policy

Info

Payment

Info

Financial

ReportAsst. Job

Steps

Asst. Job

Steps

Output

Data

Select the Postmark

date needed by BL80

downstream & write to

output file. X Since the input file now had

the Postmark date, BL80

didn’t have to query for it.


RESULTS:

Execution time of Job CD01 dropped by 80%. Execution of program BL80 went from 2+ hours to less than 15 minutes.

Example 5

FYI, this is kind of good…


EXAMPLE 6:

Review & Tune the SQL


PROBLEM:

New CICS program CIDE taking far too long to execute the only two SQL statements contained within it.

Example 6

IN DB2 SQL CPU

PROGRAM STMT# TIME SECS CALLS TIME SECS

--------- ----------- ------------------ ----------- ------------------

CIDE 1472 8744 61948 5981

CIDE 1418 3521 61948 2446

The program spent almost 2.5 hours daily in CPU alone!!

This caused the ‘Matt Phone’ to start ringing immediately...

(FYI, the developer responsible for this program was put into protective custody

to guarantee his safety)


SOLUTION:

Review the SQL for tuning opportunities…

Example 6

IN DB2 SQL CPU


--------- ----------- ------------------ ----------- ------------------

CIDE 1472 8744 61948 5981

CIDE 1418 3521 61948 2446

Statement 1472 really only returned one of 3 values, and these could be determined programmatically using the host variables in the WHERE clause.

Statement 1418 could drop a ‘SUBSTR’ predicate and instead use an ‘IN’ predicate against host variables populated to emulate what the SUBSTRing was looking for. This would allow a better index access path to the data.


SOLUTION:

Review SQL in statement 1472 for tuning opportunities.

Example 6

SELECT CARD_BRAND

INTO :PRGGBL-CARD-BRAND

FROM PRG_BIN_GLOBAL PBG

,PRG_BIN_LOAD_CNTL PBLC

WHERE PBLC.PART_NBR = PBG.PART_NBR

AND PBLC.TABLE_IND = 'G'

AND :PRGGBL-PAN-LEN-KEY = PAN_LEN

AND SUBSTR(:BILCCINF-PYMNT-CRD-NBR,

1,BIN_LEN) >= BIN_LOW


1,BIN_LEN) <= BIN_HIGH

AND CARD_BRAND IN ('D','M','V')

Statement 1472:

Only 3 Possible Values ever returned by query based upon 1st byte of the

credit card number used.

WHERE clause not filtering well and scanning 10K index rows =

Poor Performance!


RESULT:

The SQL in statement 1472 was replaced with program code to evaluate the 1st byte of the card number to determine the Card Brand.

Example 6

IN DB2 SQL CPU


--------- ----------- ------------------ ----------- ------------------

CIDE 1472 8744 61948 5981

CIDE 1418 3521 61948 2446

Replacing this SQL statement with program code saved 100 minutes of CPU usage per day!

Now it was time to look at the other SQL statement…


SOLUTION:

Review SQL in statement 1418 for tuning opportunities.

Example 6

Statement 1418:

BIN_LEN can be from 1 – 16 bytes based upon a table entry.

WHERE clause not filtering well and scanning ENTIRE index =

Poor Performance!

SELECT FDMS_NETWORK_ID

INTO :PRGPLD-FDMS-NETWORK-ID-KEY

FROM PRG_BIN_PLD PBP


WHERE PBLC.PART_NBR = PBP.PART_NBR

AND PBLC.TABLE_IND = 'P'


1,BIN_LEN) = BIN_NBR


SOLUTION:

Revise SQL in statement 1418 to replace SUBSTR with an ‘IN’ list.

Example 6

Revised SQL:

Since BIN_NBR could be between 1 – 16 bytes of the Card Number, we used program code to easily load up all

possible variants into host variables and use ‘IN’ List.

WHERE clause now only scans 20 index rows = WAY faster!

SELECT FDMS_NETWORK_ID

INTO :PRGPLD-FDMS-NETWORK-ID-KEY

FROM PRG_BIN_PLD PBP


WHERE PBLC.PART_NBR = PBP.PART_NBR

AND PBLC.TABLE_IND = 'P'

AND BIN_NBR IN (:WS-CRD-NBR-1,

:WS-CRD-NBR-2,:WS-CRD-NBR-3,







:WS-CRD-NBR-16)


FINAL RESULTS:

A drastic reduction in SQL execution time and CPU consumption by eliminating one statement and tuning the other!

Program now runs 99% faster* than before.

Example 6

IN DB2 SQL CPU


--------- ----------- ------------------ ----------- ------------------

CIDE 1472 8744 61948 5981

CIDE 1456 188 65189 43

STMT # changed from 1418 to 1456 after

recompiling the program.

Total time spent in DB2 for this query reduced by over 3,300 seconds!

# calls daily roughly the

same.

Total CPU used by this query reduced by 2,400 seconds!

* Results not typical. Please consult with your fellow developers before starting a new DB2 weight loss regimen…


EXAMPLE 7:

Performance Quick Hits


Sort input data in partitioning index order prior to processing it against DB2.

Utilize multi-row fetch on cursors that are ‘narrow & deep’ (few columns selected & many rows returned). *** Multi-row fetch can be

successful in other scenarios, too, it’s just that we’ve seen the biggest bang for the buck on ‘narrow & deep’ cursors.

Try loading several table partitions in parallel for larger DB2 tables.

Adjust commit count levels in long running batch DB2 programs to avoid not committing often enough or doing it way too frequently.

Remove ‘Stage 3’ processing. We’ve found programs fetching lots of data from DB2, then filtering it within the program. Use SQL to filter this data far more efficiently and reduce overall run time.

Example 7

Here are a few Performance ‘Quick Hits’ we’ve used that didn’t take a lot of effort to net nice performance gains:


Lessons Learned


A. Have a good relationship with your DBA • DBA’s really aren’t THAT scary (and they often possess very helpful knowledge). • Discuss performance tuning opportunities as a group and keep your DBA’s in the

loop on upcoming changes - when they’re happy, you’re happy .

B. Research new features of DB2 and your application software • Keep tabs on new release information and see how the newer features can benefit

your application.

C. Keep an open mind to trying solutions that don't fit in the box • i.e. Storing data outside of DB2.

D. Performance can often be an iterative process • You may fix one thing and find more opportunities along the way. • Rewrite & tune poorly performing SQL (i.e. replacing a simple Stage 2 predicate

with a more complex, yet more efficient Stage 1 predicate).

E. Consider how to monitor performance of changes while you’re developing them vs. after they install to production.

F. Network with other DB2 groups both within and outside of your organization • Set up regular meetings/forums to discuss new features, performance

enhancements, products, etc.

G. Most importantly, be careful about suggesting future IDUG topics or you may end up being asked to present that material here!

Lessons Learned


? ? ? Questions ? ? ?

Keith Tidball Progressive Insurance

[email protected]

Session - E06

Title - Application Development Success Stories for Performance Issues on DB2 Z/OS

Lori Zaremba Progressive Insurance

[email protected]

mailto:[email protected]

mailto:[email protected]

Date post:	08-Jul-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Application Development Success Stories for Performance Issues on DB2 … · 2019-12-08 ·...

Documents