Lessons Learned With Oracle9i on Linux for S/390

• Denny Dutcavich

• dutch@us.ibm.com

• Introduction– Objectives

– Overview

• Lessons learned– CPU

– Memory

– Paging and Swap Space

– VM and Linux Guest Setup

– Horizontal and Vertical Scaling

– Async I/O

– Storage (disk) considerations

– Backup Options

– Measurement Tools

– Oracle

• Summary of Key Factors

• New Workshop

Agenda

Introduction

• The objective of this presentation is to highlight our experiences with monitoring and performance tuning gained from:– The development team at Oracle in preparing for GA – Early adopter customers– Testing done at Oracle and IBM Boeblingen

• These topics are covered in the following Redbooks– Experiences with Oracle9i for Linux for S/390 SG24-6552 – Linux on IBM eServer zSeries Performance Measurement

and Tuning SG24-6926

CPU

• Plan for sufficient CPU resource– How many Linux guests can run on an IFL?

• Options– Linux guest under z/VM

– Linux native LPAR, IFL or general CPUs • CPU Allocation

– Virtual CPUs per constrained guests should equal Real CPUs

– Never have Virtual CPUs per guest greater than Real CPUs

– Use QUICKDISP for constrained quests

• Speed Matters – Similar clock speeds will exhibit similar performance

• Insure proper expectations when consolidating servers– Spectrum of performance runs from CPU intensive to I/O

intensive

Operational Choices

z/VM 4.3

Linux guests

MVS01MVS01 MVS03 MVS04

z/VM 4.3

MVS02 Linux guests

LPARs

IFLs

Memory

• Plan size of the Linux Guest– Linux uses all memory available to it

• Allow only for Oracle (or application) and Linux

– Size the Oracle System Global Area (memory footprint)

• 31-bit Oracle has maximum 1.5GB SGA for both 31bit and 64bit Linux on zSeries

– Need genksms from 9.2.0.2

• Don’t expect to move a very large SGA into a 256MB Virtual Machine

– Linux may appear hung if it detects memory shortage

• KILLMEM process shuts down processes to free memory

– Paging will occur if memory is over-committed • not a good thing

Paging and Swap Space

• VM paging space– Configure expanded storage for VM to use as first

paging device• Configure 25% of physical memory for

expanded storage• Alter as performance needs are better

understood• Separate paging space from guest data

– Monitor VM paging• VM likes contiguous space on paging DASD

Keep paging space at apprx < 50%

Paging and Swap Space

• Linux Swap– Linux swaps pages when it needs

memory• Could be double paging

– Configure VDISKS• Swap space on VDISKs not utilized until

Linux pages

• Linux completes P.O.– Moves page to

swap device

• Page B paged out by z/VM– Available for P.I.– Linux clueless

• Page B paged in by z/VM– Linux needs to page– Page fault accessing Page

B

Double Paging

DCBA

DCBA

.

.

.

z/VM Expanded Storage

Linux Pagingz/VM Main Memory

Linux Guest

B

B

1 2

3

1

2

3

VM and Linux Guest Setup

• Decisions - z/VM or standalone in an LPAR– Value proposition is to consolidate many servers using the

virtualization capabilities of z/VM

– Linux can be run standalone in an LPAR • Nothing is free - z/VM has its overhead

– VM overhead 0 - 12% on our tests– Minidisk I/O uses more VM processing

Minidisk caching may improve throughput on data re-

use • Think through all the options

VM and Linux Guest Setup

• DASD - sample setup201 root 200-600 CYL

202 swap VDISK (2GB)

203 swap VDISK (2GB)

204 home

205 tmp 200 – 1000 CYL

206 Usr 2700-3000 CYL

Linux Guest Priorities

• Idle Linux guest machine takes resources– Logoff VM after shutting down Linux– Timer patch

• Popup blocker for Linux• Stops Linux from waking up to check for work

– Remove unneeded cron tasks– Stop unnecessary daemons

• LPAR Weights determine CPU for guests in that LPAR

• Great tuning tips at:– http://www.vm.ibm.com/perf/tips/#Tune– http://www.vm.ibm.com/perf/tips/linuxper.html

Scheduling

Dormant ListDormant List

Logoff Logon

Work To DoIdle

Eligible ListDispatch List

Time Slice Expires

Resource Availableor E0

Resource Limit Exceeded

VM Machine ready

VM Machine waiting

VM DefinitionBlock E1

VM DefinitionBlock En

VM DefinitionBlock D1

VM DefinitionBlock Dn

Scaling

• Horizontal scaling– Add new instances of Linux – a Penguin

colony– Have sufficient resource

• Vertical scaling– Increase LPAR weight – Increase number of real CPUs or IFLs– Don’t forget to retune Oracle

Async I/O

• Oracle 9i 31bit for Linux for S/390 was built without support for Asynchronous I/O

• You must put DBWR_IO_SLAVES=4 in init.ora– Number could be higher than 4 for very

intensive I/O workloads

• Should not be an issue in future releases of Oracle

Storage (Disk) Considerations

– ESS 800 should not be treated as a black box

• Layout of files for database can have huge effect on performance

• Our tests showed 3x to 4x improvement in throughput

Storage Configuration

• Best performance will be achieved using:– multiple host adapters, – both caches, – more channels, – more CHPIDs and more paths

• Spread data across multiple arrays (ranks)• FICON Express is better than ESCON

Creating LVMs

• Use Logical Volume Manager to create file systems for best performance

• Striping critical to get more than one channel used for the data transfers – Performance gated by number of CHIPIDs

• File systems in TBytes can be created– Change extent size– Default value is 256M

• RedHat 7.2 does not have LVM – use software raid

Backup

• Backup SW used to manage backup sets created with Oracle RMANFDR UpstreamTivioliCA

• PPRC can be used for backup (ESS remote mirroring) or Oracle Advanced Replication feature

Monitoring Performance

• VM has the only accurate view of what resources each guest is consuming, so any measurements need to start there.

• Goal of VM is to provide each guest with the illusion that it's got an entire machine all to itself.

• A result is that no guest can possibly tell what real resources it is using, unless it asks VM for the information.

Performance Monitoring - Tools

• Used VM measurement tools such as– CP Indicate command (IND)

• Ind user vmlinux2 exp– CP Monitor facility– VM Real Time Monitor (RTM)

• Used Linux facilities such as– VMSTAT– NETSTAT– IOSTAT and SAR (not on SuSE)– TOP

• Total Storage Expert for ESS• CA Unicenter• Velocity Software (ESALPS)

Performance Monitoring – CP Cmds

• INDICATE - Broad overview of how system resources are being used

• LOCK - Lock in real storage selected pages

• SET SHARE - Control percentage of system resources a guest receives

• SET QUICKDSP - Designate guests that don't wait in the eligible list

• SET RESERVED - Set number of pages resident in real storage

• DEDICATE - Allocate a processor to a guest

Oracle is Oracle is Oracle

• Most Oracle performance problems are application issues– Remember to use StatsPack

• Oracle looks the same on Linux for S/390 as it does on other platforms

• Check init.ora parameter and SGA size

Key Success Factors

• Memory for Linux guest – less is better or Linux spends it’s time managing memory

• Swap space and expanded storage are mandatory

• I/O bottlenecks can be avoided byUsing multiple arrays in ESSLVM with striping

Oracle Database 10g Workshop

A new off ering - Dennis Wunder f rom the Poughkeepsie eServer Briefi ng Center has scheduled 2 newworkshops in Poughkeepsie, New York. This workshop is intended to help customers gain an understandingof how to install and manage an Oracle 10g database on Linux on zSeries and z/ VM, by providing the basicskills necessary to launch a Linux guest with Oracle on zSeries.

The workshop is designed with minimal lecture and maximum hands-on, practicing a learning by doingmethodology. Experimentation is encouraged.

Attendees will learn: Overview of VM and Linux Overview of Oracle on Linux on zSeries I nstalling a Linux guest I nstalling Oracle 10g Binaries

****

Measuring the Oracle workloadEtc

The Workshop will be held at the eServer Executive Briefi ng Center located in Poughkeepsie, N.Y.. The dates selected for the Workshop are as f ollows:- J une 22 -24-August 10 - 12

Acknowledgements

• Thanks to the following contributors– Oracle – Barry Perkins, Andres Lubomirsky, Thomas

Niewel– IBM BOE- Martin Kammerer, Juergen Doelle– IBM Storage – Bill Worthington, Pat Blaney, Sandy Abu– IBM/Oracle team- Bruce Frank, Tom Russell, Kathryn

Arrell, Denny Dutcavich