TECHNICAL TRACK: Getting the Most From the Progress V9 Database

Copyright 2002 - BravePoint

The Progress V9 Database

John Harlow

BravePoint, Inc.

Email: [email protected]


AgendaSetting the StageStorage AreasPerformanceUtilities (new and improved)SQL Engine (new)Schema (Metaschema and VST Schema)Size LimitsMiscellaneous


Setting the Stage• John Harlow - Instructor, Consultant• Progress User since 1984 (V2.1)• Presenter at USA Progress Users

Conferences


Setting the Stage: MeScott M. DuleckiPresenter at Explore! and regional user groupsBoard Member, Midwest MFG/PRO Users GroupPresident, Michigan Progress Users GroupPEG member 1998061901 Author of:

Safe Haven: Archiving in MFG/PRO


Introduction• DBA Tools

• Progress Performance Tuning Guide (July 2002)• Progress Database Admin Guide (July 2002)• Virtual System Tables (July 2002)• V9 Database Administration (July 2002)• ProMonitor - database monitoring tool• WebProMonitor – Web Frontend for ProMonitor• Pro D&L - Accelerated Dump/Load Utility• DBA Resource Kit• STGEN - V8 to V9 Conversion Tool• All Books are Now Available Online


BravePoint, Inc.Formerly United Systems, Inc (Feb 2001)Progress Services Provider

Training (Progress, QAD, Linux, and more) Performance Tuning & DB AdministrationInternet, Java, & WebSpeed InterfacesCustom Programming

Progress Experience125+ Employees3 members have used Progress since 1984Our most junior instructor has 8 years experience


Who Are You?

Are you currently using the V9 Database in Production?V9.1A, V9.1B, V9.1C, V9.1D

If not when?


What’s New in the V9 Database

Storage AreasPerformanceUtilities (new and improved)SQL Engine (new)Schema (Metaschema and VST Schema)Size LimitsMiscellaneous


Storage Areas - What are TheySimilar concept to a Version 8 Multi-Volume

database but there can be one or more Areas in a database or a "family" of V8 Databases under one "head"

Also similar to Oracle/DB2 table spacesLogical group of database Objects (i.e. tables and

indexes)Physical group of storage units (one or more

extents on disk)


Storage Areas - BenefitsRecords per Block (1 to 256) set by Area means more

efficient use of space in the Database (more details later)Multi-database transactions may be subject to logical data

corruption if 2 Phase Commit is not enabled; Storage Areas make it easier to keep all data in one database (no more split schema which is an obsolete technique)

It may be possible to improve performance by using Areas to distribute tables with heavy I/O volume over multiple disks


Storage Areas - Benefits

• Areas make it possible to keep static (infrequently changed) and dynamic (heavy create and delete activity and subject to scattering) data separated

• Doing a dump/load on a particular table (or tables) now has real value


Storage Area - DisadvantagesAreas make DB administration more complicated

There are NO Single Volume DatabasesEven the empty DBs are Multi-Volumeprobkup, procopy are Area ‘Sensitive’

Multiple High Water Marks (HWM) to Monitor (since there can be a variable extent for each Area) - there are 36 Areas in the MFG/PRO eB Database Structure File


Storage Area - Disadvantages

A V9 DB will probably be larger than V8 because of the additional space allocated for each Area

Still can't add new Extents or Areas On-Linepromon does not have Area HWM information


Storage Areas - Physical View

Area 6 Area 7 Area 8

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Ext

ent

Disk Storage - Files or Raw Partitions

Ta

ble

A

Ta

ble

B

Ind

ex

A-1

Ind

ex

B-1

Ta

ble

C

Ta

ble

D

Ind

ex

C-1

Ind

ex

D-1

Ind

ex

E-1

Ta

ble

E

Area 51

Ext

ent

Ind

ex

E


Storage Areas

Control Area (.db File) In V8 (and before) contained Extent path names In V9 contains _Area, _AreaExtent TablesThese tables can be queried with the 4GL

Primary Recovery Area (V8 BI Extents)2GB Limit is Gone! (it’s just much higher)

After-Image Areas (V8 AI Extents)User Defined Data Areas (V8 RM & IX Blocks)Schema Area (No V8 comparison exists)


Storage AreasArea 6 is the Schema Area. It contains:

The Database Master blockObject to Area mapping informationSequence value blockSchema tables, indexes, and SQL views

It is the “default” Area May contain user data but not recommendedIf no other Area is created, Area 6 will contain all data (as

is the case after conversion from V8 with proutil conv89)V9.1D - mvsch option on proutil


Storage AreasThe Physical Area layout is described in the Structure (.st)

fileAreas can have many extentsAn Extent is a separate file just like V8Only the last extent of an area may be variable sizeEach Area grows independently

Some options use the .st file to create and update DBs (prostrct, prorest, procopy)

Area contents (what tables goes to what Area) are defined in the Dictionary and/or Data Definitions (.df) - example on next page


Storage Areas

ADD TABLE "Order" AREA "od1" DESCRIPTION "Order header Info" VALEXP "1 = 1" DUMP-NAME "order"

ADD INDEX "Order-Num" ON "Order" AREA "odi1" UNIQUE PRIMARY INDEX-FIELD "Order-num" ASCENDING


Storage AreasA Table or Index Object can be placed in any data

AreaData Areas can contain indexes and/or tablesAn Area can be dedicated to one objectObjects cannot span AreasSimpler is better - do not go to extremesMy recommendation: 19 data Areas maximum;

Why?


C o n t r o l A r e a S c h e m a A r e a R e c o v e r y A r e a

s p o r t s . d b s p o r t s . d 1 s p o r t s . b 1

A r e a 7C u s t o m e r + I n d e x e s

A r e a 8O r d e r O r d e r - l i n e s

A r e a 9O r d e r O r d e r - l i n e s I n d e x e s

O n l y

s p o r t s _ 7 . d1

s p o r t s _ 7 . d2

s p o r t s _ 8 . d1

s p o r t s _ 8 . d2

s p o r t s _ 8 . d3

s p o r t s _ 9 . d1

A r e a 1 0O t h e r T a b l e s & I n d e x e s

s p o r t s _ 1 0 . d 1 s p o r t s _ 1 0 . d 2

A f t e r I m a g i n g A r e a sE a c h E x t e n t i s 1 A r e a

s p o r t s . a 1

s p o r t s . a 2

s p o r t s a 3


Records per BlockRecords per block may be different in each Area:

1, 2, 4, 8, 16, 32, 64, 128, or 256The DBKey/Recid/Rowid format varies but is transparent to

applications

Benefits Improved storage utilizationAllows use of large block size (8k) for better performance

DisadvantageHigher Rec/Blk = Lower Maximum Area Size


Preparing for Storage AreasV8.3 Table and Index I/O Monitoring

-tablebase & -tablelimit -indexbase & -indexlimit_tablestat & _indexstat VSTs

Identify Static vs Dynamic vs Historical DataIdentify Large TablesIdentify tables that are frequently and easily scatteredIdentify tables with large record sizesIdentify unused indexes


Options to Convert to V9proutil conv89

Usually takes less than 5 minutesAll Data & Indexes are stored in the Schema AreaWill NOT take advantage of Storage AreasMoving Tables and Indexes still leaves a Large Schema

Area (V9.1D proutil/mvsch)

Dump/LoadWill take ……longer than conv89Can take full advantage of Areas through easy

modifications to the .df file


V9 Structure (.st) Files# Schema Area# Records/Block:64# Located in the /db directoryd “schema area”,64 /db

# Data Area# Area Name: data1 Area Number: 10# Records/Block:128d “data1”:10,128 /db1/data_10.d1 f 100000d “data1”:10,128 /db2/data_10.d2 f 100000d “data1”:10,128 /db3/data_10.d3

# Index Area#Area Name:index1Area Number: 20#Records/Block:Not given because intended for index blocksd “index1”:20 /db4/ f 100000d “index1”:20 /db4/

# Variable size BI extentb /bi/prod.b1

# After Image Extentsa /ai/data.a1 f 100000a /ai/data.a2 f 100000a /ai/data.a3 f 100000


V9 Utilities - Table Moveproutil <db> -C tablemove <table> <area> Rebuilds the Indexes during the move since Recids

are no longer unique to a DB (but are unique to an Area)

Off-line as well as on-lineOn-line: Will Exclusive Lock the entire Table for the

duration of the moveOne Move = One Trx = Bigger BI/AI FilesMonitor the process with the _UserStatus VST; the

Table Lock can be monitored with promon or the _Lock VST


V9 Utilities - Index Moveproutil <db> -C indexmove <Index def> <area>Index def: <owner>.<table>.<index>Run off-line or on-lineOn-line:

Only read operations will be allowed The table will be SHARE locked

Benefits: flexibility and availabilityMonitor the process with the _UserStatus VST


V9 Utilities - Index Compactproutil <db> -C indexcompact

<owner>.<tablename>.<indexname>Combines partially filled adjacent index blocks Runs off-line or on-lineIndex Fix + Index Compact = Index RebuildBenefits:

Performance - fewer disk i/o’sReduces index sizeAvailability - index not locked

DB Corrupting Bug thru V9.1D (no patches)


Enhancements to Utilities

Most utilities are now ‘Area Aware’Example: proutil idxbuild will only scan

the blocks in the Area where the table exists

proutil dbanalys/tabanalys/ixanalys can now report individual Areas (this is undocumented)proutil <db> -C dbanalys <area name>



procopy & probkupTarget Area Numbers must MatchArea Names don’t need to MatchTarget Records per Block must Matchprorest - the # of Areas must matchUses .st File (if one exists) in Target Directory



proutil truncate areaIf no Area is specified, all Areas that have no

Storage Objects have the HWM moved to ‘ground zero’

If an Area name is specified, the data is in the Area is removed (after confirmation)

In each case there is no impact on the physical extents


V9 Performance ImprovementsDeferred Transaction Start

“transaction-begin” BI note write is deferredNo “transaction-begin” or “transaction-end” notes are written

for “read only” transactionsThis optimization is invisible to applications (you can’t forget

the 4gl transaction scoping rules)

Benefits:BI clusters may be reused sooner; Smaller BI log Improved performance for “empty” transactionsODBC Connections & many 4gl programs won’t cause an

open transaction


V9 Read BuffersProcesses such as large reports reports can evict and

replace many shared buffersRead buffers might be able prevent this Read buffers are taken from the shared poolRead buffers remain accessible to all users When read buffers are enabled:

Read operations will use the read buffersWrite operations will use the public -B pool


V9 Read BuffersNew client startup parameter -Bp n enables read

buffers per user# of buffers can be changed at runtime:

do for _myconnection transaction :find first _MyConnection._Myconn-NumSeqBuffers = 10.

end.run bad-report.p.do for _myconnection transaction :

find first _MyConnection._Myconn-NumSeqBuffers = 0.

end.


V9 Read BuffersPer Client maximum is 64 (until V9.1C)

if more are requested, only 64 are granted

Total read buffers of all users cannot exceed 25% of -Bwhen this limit is reached further requests are denied and the

public buffers will be used

V9.1C -Bpmax Broker parameterProblems:

How to Allocate? VERY difficult questionNo way to monitor how many buffers in use and how they are

being used


New Virtual System Tables

VSTs are Automatically Enabled in all V9 Databases

New VSTs_AreaStatus HWM Info_MyConnection Info about my connection_StatBase Reset table/index I/O

limits_UserStatus Monitor some online

utilities


V9 Size LimitsDatabase size: BIG!Primary Recovery Area (BI) size: 16-32TB Maximum Number of Areas: 1,000Data Area size: Varies

16TB (1 row per block, 8k block size)8GB (256 rows per block, 1k block size)

Data Area block size: 8,192 bytesExtent size: Still 2 gigabytes until V9.1C

proutil <db> -C EnableLargeFiles


V9 Size LimitsRecord size: 32,000 bytes (unchanged for now)Records per Area/Table: 2,147,483,647

Depends on Block Size & Record/Block ValueNumber of Records per block: 256Number of Indexes: 32,000Number Sequences: 2,000 (with 8k blocks)

Increased from 100Number of concurrent users: 10,000Number of concurrent transactions: 10,000


What V9 is NOTMulti-Threaded BI AccessBuffer Cache (-B) per Area2GB File Size Limit Removed

Temp FilesDump (.d) filesDB, BI, and AI Extents (until V9.1C)

‘Pure’ 24 x 7 OperationSchema ChangesAdding new ExtentsReactivating indexes after proutil idxfix


For More Information

BravePoint’s V9 Database Administration Jumpstart Class is designed for V6-V8 DBAs that want find out everything they need to know for moving to V9

Dan Foreman’s V9 Database Administration Jumpstart publication

Dan [email protected]


Date post:	02-Jul-2015
Category:	Documents
Upload:	databaseguys
View:	93 times
Download:	2 times

TECHNICAL TRACK: Getting the Most From the Progress V9 Database

Documents