Wrangling With Oracle

Ray Murray

Sirius User’s Group

June 2001

Introduction

How can OS/390 Oracle Benefit Wrangler Supplement Model 204 Complement Model 204 Replace Model 204

Tangible Benefits? Organizational Benefits? Intangible Benefits?

Complementing Model 204

How can OS/390 Oracle Complement M204 “Standard” DBMS Interfacing with “Power” User tools

MS Office MS Query Access, Excel VBA Applications

Maintain a subset of Wrangler Data Interfacing with common Web environments

Cold Fusion ...

Supplementing Model 204

Scaleable to Other Environments UNIX (Most Flavors) Linux NT

Enhance Wrangler Integration with other related applications Replicate related Oracle database on Mainframe

True Replication Export from UNIX - Import to OS/390

Replacing Model 204

Can OS/390 Oracle Host the entire Wrangler database? Does living on the mainframe have

advantages not found in the our other Oracle environments?

Database Conversion approach and effort Ability to re-use parts of Application

Software

Questions

Can Wrangler data be ported to Oracle? What are the storage usage differences? What are the performance differences? What is life with OS 390 Oracle like?

Tuning, Administration, Support In house Vendor

What other hurdles might there be?

“Try it out” Approach - II

Re-Attempt to “Port” Database to Oracle Second iteration using lessons learned Database Design Issues

More De-normalization Database Loading Retrievals

Coding Performance

Current Database

Single Major Record type (avg.. 500 bytes) Unique System Generated Primary Key (PK) 35 “OCC 1” mandatory fields 58 Mult Occ fields holding discrete values

Color, Make, Model … 26 Mult Occ fields of “range” values

Cost, Premium, Weight … 137 Optional single occ fields - descriptive info

Current Database

Substantial indexing (68 fields) Two NR fields 41 ORD CHAR

PK is ORD CHAR 11 of 58 Mult Occ discrete fields are ORD

CHAR 25 ORD NUM

19 of the Mult occ range fields are ORD NUM 2/3 Billion records - across 75 files

Oracle Test Database

Minimally “Normalized” Port 2 Tables

Base table (I) contains 152 columns 63 columns are indexed

Child table (O) contains 19 columns replaces many Mult Occ M204 fields 15 columns are indexed

No Reference or Join tables defined Didn’t define columns not used in M204 file sample

Sirius FUNLOAD FSTATS report very useful here First try 7-table port too inefficient

Database Porting Issues

Dates/Time pairs Combined in Oracle

Numeric values Varchar or Number?

Text Concatenated a multiply occ M204 field

Invisible fields Oracle indexing is very flexible, but ..

Database Preparation

Model 204 test database Extracted 3 million production DB records

Funload UAI job ran in 3 minutes Created 3234 Cyl M204 file in 10.1 minutes Sirius LAI job loaded M204 file in 46

minutes Loaded data and built all indexes

Oracle Database Preparation Extraction of data from M204 database

Prepared Sequential files to load “I” and “O” M204 User Language extract job took 67 minutes

FUEL would have been much faster Prepared SQL*Loader Positional scripts M204-Centric approach to preparing data for

Oracle is most straightforward due to power of User Language and Funload/FUEL tools

Added 3234 Cyl dataset to Oracle database VSAM job took 28.5 minutes to format it

Loading Oracle Test Database Loaded “I” table in 27 minutes (direct=true)

3 Million rows Loaded “O” table in 23 minutes(direct=true)

5,449,864 rows Created 63 “I” table indexes in 540 minutes Created 15 “O” table indexes in 66 minutes Total 656 minutes to build this heavily

indexed Oracle test database ‘Best’ numbers from several runs

Space Usage

M204 file Asize=4, Bhighpg=230769, Csize=2300,

Dpgsused=56102 Total: 1788.4 M space used

Oracle “I” table 1228.8 M “O” table 358.4 M “I” indexes 1867.1 M “O” indexes 685.7 M Total: 4140 M space used (231% x M204)

Query Comparisons

Oracle Queries Run via SQL*Plus running under TSO Statistics reported from SET TIMING ON Oracle Expert product guidelines used to tune Oracle

M204 Queries Run in an M204 online test region - large buffer pool Statistics reported from TIME REQUEST Both M204 and Oracle running on same machine

using same type of disk drives Identical query results required

Timing Comparisons

Run time Oracle Elapsed time of 0.02.15.55 means 0 hours,

2 minutes, 15 seconds, 550 milliseconds M204 CNCT (connect time) is reported in

seconds. Above Oracle time would be the same as CNCT=136

CPU time Oracle CPU time is reported in seconds. CPU of

20.25 is twenty and 1/4 second. M204 CPU is reported in milli-seconds. 20.25

Oracle seconds is the same as CPU=20250

Query Scenario 1

How many claims were filed having collision damage (CD) between 9000 and

9150 dollars? liability damage (LD) between 3000 and

4000 dollars? injury amount (INJ) between 200 and 300

dollars?

Query 1 - M204

User Language:

BEGINCLAIMS.FD: IN CLAIMS FPC

RECTYPE = CLAIMCD IS IN RANGE FROM 9000 TO 9150LD IS IN RANGE FROM 3000 TO 4000INJ IS IN RANGE FROM 200 TO 300

END

Answer: 9155CNCT=4, CPU=922, DKRD=381, RQTM=1681

Query 1 - Oracle

SQL*Plus (O table only, no join)Select Count(*) from O where CD between 9000 and 9150 and LD between 3000 and 4000 and INJ between 200 and 300;

Answer: 11004Elapsed=0:00:03:58, CPU=2.14 remember: Oracle reported CPU=2.14 is

like M204 reported CPU of 2140

Query 1 - completed

Join added to match M204 answer Select count(distinct I.PK) from I, O where

CD between 9000 and 9150 andLD between 3000 and 4000 andINJ between 200 and 300 and

I.PK = O.PK; Answer: 9155 (matches M204)

Elapsed=0.00:42:02 and CPU=5.65

Query 1 - narrowed ranges

LD between 9100 and 9110CD between 3500 and 3600INJ between 200 and 300

Answer: 38 from both systems Oracle Elapsed=0:00:00.53 CPU=.25

secs M204 CNCT=1, RQTM=416, CPU=340

ms

Query 1 - Widened Ranges

LD between 5000 and 12000CD between 2000 and 5000INJ between 200 and 300

Answer: 215959 “I” rows Oracle

Elapsed 0:07:17.06 Cpu 146.20 M204

Cnct=30, Cpu=16446, Dkrd=4754, Rqtm=25252

Query 2 - Simple Count

Oracle - Answer: 2,999,995 Select count(*) from I; Elapsed 0.02:15.55 CPU 20.25

M204 Begin

FPCEnd

CNCT=2, DKRD=127, RQTM=411, DKPR=251

Query 3 - Range Query

Oracle - Answer: 2,410,583 Select count(*) where mydate

between 970705 and 970725; Elapsed: 0.00.26.02 CPU 8.82

M204 Begin

FPC mydate is in range from 970705 to 970725end

M204 - cnct=1, cpu=25, dkpr=816, rqtm=543

Query 4 - Multiple criteria

Oracle Select count(*) from I where mydate between ‘07-JUL-97’ and ‘25-JUL-97’ and make = ‘AQ’ and model = ‘BC’ and ratinga between -600 and 1500 and ratingb between -10000 and 4000;

Answer: 861Elapsed 0:01:10.87 CPU 22.28

Query 4 - M204 version

M204BeginFPC mydate is in range from 970705 to 970725 make = ‘AQ’ and model = ‘BC’ ratinga is in range from -600 to +1500 ratingb is in range from -10000 to 4000End Answer: 861cnct=3, cpu=303, dkrd=1171, rqtm=3391

Query 5 - Scanning rows

Oracle - Answer: 685Select count(*) from I where make = ‘AQ’ and mydate between ‘07-JUL-97’ and ‘25-JUL-97’ andratingc like ‘235%’ Elapsed: 0:02:33.48 Cpu 26.51 Scanned through 445904 rows

M204 equivalent (ratingc like ‘235*’) CNCT=10, CPU=2637, RQTM=3094,

DIRRCD=445904, DKPR=127506 (many pages in buffer pool) CNCT=100, CPU=8072, DKRD=62201, RQTM=100528,

DKPR=127506 (no pages in buffer pool)

Query 6 - ‘Is Present’ Scan

Select same 445904 rows/records to scan Look for presence of a column/field

Answer: 235997 Oracle - mynamefield is not null

Elapsed 0.02.39.31 CPU 25.47 M204 - mynamefield is present

CPU=8785, CNCT=10 (yes - it’s true), RQTM=9997, DKPR=563314 (reused buffers)

CPU=14306, CNCT=129, DKRD=62201, RQTM=129608, DKPR=563314 (fresh pages)

Query 7 - Extracting data

Output 14 fields from 445904 rows in a columnar format

Both extracts output 1100 tracks of data Identical DCB for output files

Oracle (Using SQL*Plus) Elapsed: 0:06:20.81 Cpu: 118.54

M204 User LanguageCNCT=222, CPU=45228, DKRD=62196, RQTM=201628

Sirius FUEL via $FUNLOAD Total Time: 57.557 CPU=9.875

Query 8 - Count of each value Oracle

Select make,count(make) from Igroup by make;

Elapsed 0:02:39.76 cpu=118.54 M204

Make.frv: frv makeprint value in make.frv tab ...FPC make = value in make.frv

cpu=112, dkrd=485, finds=11, rqtm=996, cnct=1

Changes since Wrangling I

Simplified Oracle Database design Handles 98% of “I” type Wrangler data Need to handle other 2% with another

design “O” table design less generic

An intermediate table between I and O was eliminated

Changes to reduce Joins

Three multiply occurring M204 fields rigidified in Oracle to reduce joins Eliminated PK repetition in 3 tables Field A became A1, A2, A3, A4, A5, A6 Field M became M1, M2, M3, M4 Field R became R1, R2, R3, R4, R5, R6,

R7, R8 (Field “R” is used in every Wrangler Find)

Changes - continued

Index tuning Reduced segments used for indices Several major “O” table indices created

with the BITMAP option Matched M204 column indexing Refined column attributes Combined three date/time pairs

Observations: Space Mgmt

Oracle - More pools of Space to work with Flexibility Can become overdone

Additional Job Security for tuners

M204 Need to tune indexes ‘together’

Share same pools (C, D)

Observations: Programming

Simple Ad-hocs (1 table) Much easier with SQL Our end users don’t do such requests Would have to limit using Views

Complex queries SQL appears delusionally simple Short code can be very complex Pgmr has to learn about paths through database M204 Procedural cross referencing approach is

easier to follow.

Living With Oracle vs M204

Retrofitting Performance De-normalization Non-Standard SQL (Hints) Learning how optimizer works Moving Indices around / Eliminating Indices

Enqueueing Less enqueueing issues in Oracle

Read-consistent views

Backup/Recovery

Oracle Recovery Rollback Segment oriented can set up App specific RBS Each database’s recovery is independent Closer to TBO than Chkp/Rf

Backup/Exports Nothing like Fast Dump or UAI/LAI Dumps DDL and data as Inserts ...

Losing Oracle Control table is disastrous

System Administration

Oracle - Everything is in a bunch of tables Tools use those tables

Experts can be helpful but ... M204 Information in system stats,

journal .. Special commands and tools

Monitor Sirmon

DBMS Data Integrity Enforcement Oracle has “Standard” DBMS features

Triggers Referential Integrity Field constraints Major ongoing effort to setup, tune, maintain Programs still have to handle exceptions

“Decoupled” approach used by Wrangler Updates limited to controlled processes that

validate data

Should Wrangler be Ported?

Porting the Wrangler Database to Oracle is not economically feasible Substantially more design, prototyping, and

analysis is necessary Special Techniques necessary

The effort to develop and support an Oracle database grows exponentially as size and scope of database increases

Complementing M204

The OS 390 Version of Oracle works well and can effectively support hot backup of a Wrangler adjunct Oracle database that is currently under development. No need to setup and administer another Unix or

NT hardware/software environment for this. System Administration, Storage management,

Security processes already in place

Supplementing M204

Oracle makes it possible to “port” subsets of Wrangler data to Unix and NT machines that run Oracle for special purpose applications. Ported Wrangler subsets should be de-indexed Port can be rehearsed effectively on mainframe

Oracle integrates effectively with the rest of the World OS Environments Java (COBOL of the 00’s) Standards

Wrangler - Currently with Model 204 Current M204 Offerings (CCA/Sirius)

Leverage M204’s unique strengths Leverage knowledge of User Language Leverage Mainframe instruction set and

other features Wrangler performance stays ahead

Sirtune Evaluated MP/204

Clever product truly helps - easy to misunderstand

Wrangler - relies on M204

Wrangler survives in the M204 world because of “Current” features Reorganized entire database for structural

changes in one week. Sirius UAI/LAI

Dump operational M204 files while transactions are in-flight.

Sirius Fast Dump Simple monitoring and problem chasing

Sirscan Sirmon

Connecting Wrangler

Wrangler survives in M204 because it is becoming more and more interoperable Janus Web Server

Supports majority of retrieval user base now Software for selected updating users being tested

Janus Sockets Proof of Concept Wrangler Retrieval API

API for talking to Java, Perl, VB, C/C++ ... Tested with Perl on Unix

Connect * Evaluated too

Tangible Benefits ?

$$$ Performance

Not obvious Database Organization

Major effort to define Database and Views that would make “Standard” simple SQL programming work with Wrangler data.

Programmability - need to use proprietary Oracle features

Organizational Benefits?

Less database systems to support? Major effort to convert Extra effort necessary to port Wrangler

applications to Oracle (or DB2) will more than offset cost of continuing to support M204 on mainframe

Inherent RDBMS complexity Oracle “Knowledge” is much more

common - bookstore & libraries.

Intangible Benefits?

Career Fear Issue M204 - Should we call it M X’CC’ instead?

“USER” part of language name Mainframe?

Remaining “Current” call PAI the “self-describing data” statement?

Ongoing Supportability What is “Current” now anyway There are antique Oracle databases around too

OS/390 Oracle and Wrangler

Serve as a replicated site for a smaller (150M) Wrangler related database.

Use Janus Open Client to extract from Oracle database

Extract subsets of Wrangler data, then load them into Oracle form to be exported to offsite Oracle databases.

“Mainframe” Oracle

Observations about OS 390 Oracle 8i SQL*Plus works well with TSO and Batch SQL*Loader works well in Batch JCL env Works with NT based administration tools Truly indistinguishable from Unix and NT versions

via Net 8 connections Very stable Leverages Mainframe behind the scenes

VSAM, Media Manager ...

Oracle for Ad-hoc support?

Use of generalized “Power User” tools to query Wrangler data not feasible. Just too much data User Language oriented Selection best Specialized $FUNLOAD for high speed

extracts - limited purpose Ongoing improvements to UL Extract

speed

M204 - Alive and well

Evolutionary approach benefited Wrangler Incremental system upgrades Incremental database upgrades Incremental app performance upgrades

Our team is effective at combining knowledge of legacy system processes with ongoing incremental M204 refinements

M204 - Alive and Well

Modernized User Interface Ever-expanding Janus Web Server Application

suite Providing Wrangler services to external

apps Specialized FUEL extracts

Store and Forward Specialized API

Janus Sockets based Wrangler data server

Porting to Oracle?

“Porting” a big M204 database to Oracle will require iteratively trying many different design ideas.

In the first port attempt, 1 million Wrangler M204 records became 14 million Oracle rows.

In the second port attempt, 3 million Wrangler M204 records became 8.5 million Oracle rows. This design is less general.

Technical Observations

Only one hiccup (OC4) was encountered with OS/390 Oracle during this exercise.

Oracle Bitmap indexes used 30-50% less space than Normal indexes. 3 “O” table indexes were created as Bitmapped

Once formatted, space assigned to Oracle is easily re-used.

Living with Oracle - 1

World of M204 is less subject to “Press 1 for …” support.

Most Oracle support requests require entering request via Web and waiting for a formal reply.

As is the case with M204 the DBA for a large Oracle database must be familiar with the application.

Oracle documentation is readily available. Useful for OS 390 Oracle OS 390 based examples not readily available

Living with Oracle - 2

Ongoing effort to maintain performance of Oracle database is more involved than with M204. Oracle recommends rebuilds of indexes

periodically Programmers have to learn how

optimizer reacts to their database. Not feasible for end users

Living with Oracle - 3

World of Oracle encourages “Technique-ing”

M204 puzzling to those inclined to “Technique-ing”

Development of Wall Poster database designs

Many parts to Oracle What do you choose?

Pro C, SQLJ, JSQL, ???

Living with Oracle - 4

Upgrading an Oracle database can be cumbersome Ex: Changing length of a field that is used

in referential integrity constraints. Knowledge Base

Lots of basic knowledge duplicated Enough to run in circles forever

Unexpected findings

CCA and Sirius are not burdened with keeping M204 features working on multiple platforms.

Oracle helps give new respect to the M204 Coded field attribute.

What’s Next ?

Can SQL access be provided to existing Wrangler M204 data? Is it really needed? Janus Open Server/ Janus Specialty data

store? Connect *

Brief initial testing with WebGate suggests better performance than with Oracle