Date post: | 28-Nov-2014 |
Category: |
Documents |
Upload: | ussama-aoun-muhammad |
View: | 588 times |
Download: | 11 times |
IBM Oracle Solutions Advanced Technical Support - Americas
Oracle on AIX Workshop © 2006 IBM Corporation
Oracle on AIX – Configuration & Tuning
R. Ballough, IBM Advanced Technical Support
2
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Legal information
The information in this presentation is provided by IBM on an "AS IS" basis without any warranty, guarantee or assurance of any kind. IBM also does not provide any warranty, guarantee or assurance that the information in this paper is free from errors or omissions. Information is believed to be accurate as of the date of publication. You should check with the appropriate vendor to obtain current product information.
Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use.
IBM, ̂ , and pSeries are trademarks or registered trademarks of the International Business Machines Corporation.
Oracle and Oracle9i are trademarks or registered trademarks of Oracle Corporation.
All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.
3
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Basic AIX Configuration/Tuning for Oracle– Memory
– CPU
– I/O
– Network
– Miscellaneous
RAC ConfigurationRAC Tuning
Agenda
4
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
The suggestions presented here are considered to be basic configuration “starting points” for general Oracle workloads
Customer workloads will vary
Ongoing performance monitoring and tuning is recommended to ensure that the configuration is optimal for the particular workload characteristics
AIX Configuration for Oracle “starting points”
5
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Oracle Server Architecture – Memory Structures
System Global Area (SGA)Program Global Area (PGA)
CHKP
PMON SMON
DBWR
LGWR
ARC0
D000
Redo Log Buffer
Online Redo Logs
ArchiveLogs
DatabaseFiles
Control FilesUser
Private SQL Area
DatabaseBuffer Cache
SharedPool
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation6 © 2005 IBM Corporation
The AIX “vmo” command provides for the display and/or update of several parameters which influence the way AIX manages physical memory
– The “-a” option displays current parameter settingsvmo –a
– The “-o” option is used to change parameter values vmo –o minfree=1440
– The “-p” option is used to make changes persist across a reboot
vmo –p –o minfree=1440
Memory Tools - Virtual Memory Manager (VMM)
A number of the default “vmo” settings are not optimized for database workloads and should be modified for Oracle environments
7
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
General Memory Tuning
Memory Use testsys 8/2/2005
0102030405060708090
100
8:03
8:04
8:05
8:06
8:07
8:08
8:09
8:10
8:11
8:12
8:13
8:14
8:15
8:16
8:17
8:18
8:19
8:20
8:21
8:22
8:23
8:24
8:25
8:26
8:27
8:28
8:29
8:30
8:31
8:32
8:33
8:34
8:35
%comp %file
8
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
VMM Tuning Pre AIX 5.2 ML4
MINPERM% – minimum % real memory for fs buffer cache
15-20%: JFS or JFS2 filesystems without DIO or CIO5%: RAW logical volumes
JFS or JFS2 with DIO or CIOGPFS
MAXPERM%, MAXCLIENT% - max % real memory for fs buffer cache
40-60%: JFS or JFS2 filesystems without DIO or CIO<= 20%: Raw logical volumes
JFS or JFS2 with DIO or CIOGPFS
Never more than 20 GB prior to AIX 5.3To start, set to vmtune "numperm" valueReduce until vmstat freed (fr) to scanned (sr) ratio is 4:1
9
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
VMM Tuning – AIX 5.2ML4+
MINPERM% =5%
MAXPERM%, MAXCLIENT%=80% or highermake this a threshold which is > (1-computational memory)
LRU_FILE_REPAGE=0LRU_POLL_INTERVAL=10ms
LRU_FILE_REPAGE=0 is a “hint” to lrud to ignore repage rates when determining what to page out – effectively favoring paging out file pages (filesystem buffer cache) rather than computational pages
LRU_POLL_INTERVAL indicates the time period after which LRUD pauses and interrupts can be serviced. Default value of “0” means no preemption.
STRICT_MAXPERM=0 (default) STRICT_MAXCLIENT=1 (default)
10
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Understanding Memory Pools
Memory pools are configured at boot time based on physical hardware configurationNumber can be seen with ‘vmstat –v |grep pools’Size can only be seen using KDB
LRUD operates per memory pool
SGA is allocated equally from memory pools
If free list has been depleted in a memory pool, LRUD will scan/reclaim memory in that pool
Consider implementing one of the following: Setting memory_affinity=0 ignores physical hardware configuration and
allocates evenly sized memory poolsAIX 5.3 ML3 with APAR IY69237 modifies VMM to consider all free frames in
system regardless of memory pool before reclaiming memory
11
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
VMM Page Stealing ThresholdsThe following define thresholds for the VMM page stealing process (lrud):
minfreeSet minfree = 120 x # logical CPUs /#mem poolsConsider increasing if vmstat “fre” column frequently approaches zero or if
“vmstat –s” shows significant “free frame waits”
maxfreeSet maxfree = minfree + (MAX(maxpgahead, j2_maxPageReadAhead) * #
logical CPUs)/ # mem pools
Example:For a 6-way LPAR with SMT enabled, maxpgahead=8 and j2_maxPageReadAhead=8:minfree = 1440 = 120 x 6 x 2maxfree = 1536 = 1440 + (max(8,8) x 6 x 2)
vmo –o minfree=1440 –o maxfree=1536 -p
12
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
General AIX memory tuningNo requirements for allocating SGA
SGA may be pinned on AIX, but IS NOT RECOMMENDED – we have better ways of keeping sga resident in memory
Do not over commit real memory!!!!Server should be configured with enough physical memory to satisfy memory
requirements
Paging spaceWith AIX demand paging, paging space does not have to be large½ memory + 4GB
Monitor paging activity:vmstat -ssar -rnmon
Resolve paging issues:Reduce Filesystem cache size (MAXPERM, MAXCLIENT)Reduce Oracle SGA or PGA (9i or later) sizeAdd physical memory
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation13 © 2005 IBM Corporation
Changing SGA size
MemoryOracle 9i or 10g– SGA can be dynamically resized, but has an upper bound by the parameter
SGA_MAX_SIZE. • SQL> alter system set db_cache_size=2048m scope=both;• (
SGA_TARGET (10g)• DB_CACHE_SIZE, SHARED_POOL_SIZE., etc.
– PGA_AGGREGATE_TARGET can be dynamically resized
– SGA_TARGET and PGA_AGGREGATE_TARGET are not hard limits
Something to keep in mind for use of DLPAR with memory…..
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation14 © 2005 IBM Corporation
Determining SGA size
SGA Memory Summary for DB: test01 Instance: test01 Snaps: 1046 -1047
SGA regions Size in Bytes
------------------------------ ----------------
Database Buffers 16,928,210,944
Fixed Size 768,448
Redo Buffers 2,371,584
Variable Size 1,241,513,984
----------------
sum 18,172,864,960
SQLPLUS:SQL> show sga
SQL> show parameters
Statspack:
15
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Basic AIX Configuration/Tuning for Oracle– Memory
– CPU
– I/O
– Network
– Miscellaneous
Agenda
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation16 © 2005 IBM Corporation
Oracle parameters based on # CPUs
– DB_WRITER_PROCESSES
– Degree of Parallelism
– - table level
– - query level
– - PARALLEL_MAX_SERVERS or AUTOMATIC_PARALLEL_TUNING (CPU_COUNT * PARALLEL_THREADS_PER_CPU)
– CPU_COUNT
– FAST_START_PARALLEL_ROLLBACK – should be using UNDO instead
– CBO – execution plan may be affected; check explain plan
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation17 © 2005 IBM Corporation
CPU ConsiderationsUse SMT with AIX 5.3/Power5 environments
Micropartitioning considerations- Virtual cpus)<= physical processors in shared pool- CAPPED
- Virtual CPUs should be the nearest integer >= capping limit- UNCAPPED- Virtual CPUS should be set to the max peak demand requirement
DLPAR considerationsOracle 9i– Oracle CPU count does not recognize change in # cpus– AIX scheduler can still use the added CPUsOracle 10g– Oracle CPU count recognizes change in # cpus
18
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Basic AIX Configuration/Tuning for Oracle– Memory
– CPU
– I/O
– Network
– Miscellaneous
Agenda
19
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Oracle Server Architecture - Files
System Global Area (SGA)Program Global Area (PGA)
CHKP
PMON SMON
DBWR
LGWR
ARC0
D000
Redo Log Buffer
Online Redo Logs
ArchiveLogs
DatabaseFiles
Control FilesUser
Private SQL Area
DatabaseBuffer Cache
SharedPool
20
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Flashback database design
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation21 © 2005 IBM Corporation
Options for storing Oracle data files
Filesystems– Single-instance:
– JFS, JFS2, Veritas VxFS
– Clustered:– GPFS, Veritas CFS
Raw
Automatic Storage Management (ASM) – new in 10g
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation22 © 2005 IBM Corporation
Data Layout for Optimal I/O Performance
Stripe and mirror everything (SAME) approach:
Goal is to balance I/O activity across all disks, loops, adapters, etc...
Avoid/Eliminate I/O hotspots
Manual file-by-file data placement is time consuming, resource intensive and iterative
Use RAID-5 or RAID-10 to create striped LUNs (hdisks)
Create AIX Volume Group(s) (VG) w/ LUNs from multiple arrays, striping on the front end as well for maximum distribution
Physical Partition Spreading (mklv –e x) –or-
Large Grained LVM striping (>= 1MB stripe size)
http://www-1.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100319
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation23 © 2005 IBM Corporation
Data Layout cont’d…Stripe using Logical Volume (LV) or Physical Partition (PP) striping
LV Striping– Oracle recommends stripe width of a multiple of
Db_block_size * db_file_multiblock_read_countUsually around 1 MB
– Valid LV Strip sizes:AIX 5.2: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1 MBAIX 5.3: AIX 5.2 Strip sizes + 2M, 4M, 16 MB, 32M, 64M, 128M
– Use AIX Logical Volume 0 offset (9i Release 2 or later) • Use Scalable Volume Groups (VGs), or use “mklv –T O” with Big VGs• Requires AIX APAR IY36656 and Oracle patch (bug 2620053)
PP Striping– Use minimum Physical Partition (PP) size (mklv -t, -s parms)
Spread AIX Logical Volume (LV) PPs across multiple hdisksin VG
• (mklv –e x)
24
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Single Instance Environments - Filesystems
Filesystems
JFS – no longer being enhancedBetter for lots of small file creates & deletes
JFS2 – generally the preferred single-instance filesystemBetter for large files/filesystems
Mount options:Buffer Caching (default)– stage data in fs buffer cacheDirect I/O (DIO)– no caching on readsConcurrent I/O (CIO) – DIO + no write lock (JFS2 only)Release Behind Read (RBR)– During sequential reads, memory pages released after
pages copied to internal buffersRelease Behind Write (RBW) – During sequential writes, memory pages released after
pages written to disk
In 9i, DIO and CIO must be specified at the filesystem level
In 10g, Oracle issues o_cio and o_dio calls as appropriate
25
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Single-instance environments Cached vs. non-Cached (Direct) I/O
File System caching tends to benefit heavily sequential workloads with low write content. To enable caching for JFS/JFS2:
Use default filesystem mount options
Set Oracle filesystemio_options=ASYNCH
DIO tends to benefit heavily random access workloads and CIO tends to benefit heavy update workloads. To disable JFS, JFS2 caching:
In 9i, set filesystemio_options=SETALL and use dio or ciomount option
In 10g, set filesystemio_options=SETALL
When using DIO/CIO, fs buffer cache isn’t used. Consider the following db changes:
Increase db_cache_size
Increase db_file_multiblock_read_count
Read Metalink Note #272520.1
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation26 © 2005 IBM Corporation
Single-instance environments - Oracle Database File AccessData Base Files (DBF)
I/O size is db_block_size or db_block_size * db_file_multiblock_read_countUse CIO or no mount options for extremely sequential I/OIf block size is >=4096, use a filesystem block size of 4096, else use 2048
Redo Log/Control FilesI/O size is always a multiple of 512 bytesUse CIO or DIO and set filesystem block size to 512
Archive Log FilesDo not use CIO or DIO‘rbrw’ mount option can be advantageous
Oracle BinariesDo not use CIO or DIO
27
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Single-instance environments - I/O Tuning (ioo)READ-AHEAD (Only applicable to JFS/JFS2 with caching enabled)
MINPGAHEAD (JFS) or j2_minPageReadAhead (JFS2)Default: 2Starting value: MAX(2,DB_BLOCK_SIZE / 4096)
MAXPGAHEAD (JFS) or j2_maxPageReadAhead (JFS2)Default: 8 (JFS), 128 (JFS2)Set equal to (or multiple of) size of largest Oracle I/O request
DB_BLOCK_SIZE * DB_FILE_MULTI_BLOCK_READ_COUNT
Number of buffer structures per filesystem:
NUMFSBUFS (JFS2): Default: 196, Starting Value: 568Monitor with “vmstat –v”, increase if value of “filesystem I/Os blocked with no
fsbuf” is increasingj2_dynamicBufferPreallocationDefault: 16Monitor with “vmstat –v”, increase if value of “external pager filesystem I/Os
blocked with no fsbuf “is increasing
28
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Asynchronous I/O for filesystem environmentsAIX parameters
minservers = 10 * # cpusmaxservers = (10 * # disks) / # cpusmaxreqs = a multiple of 4096 > 4 * #disks * queue_depth“enable” at system restartTypical settings: minservers=100, maxservers=200, maxreqs=16384
Oracle parametersdisk_asynch_io = TRUEfilesystemio_options = {ASYNCH | SETALL}db_writer_processes = default
Monitor usage:Watch alert.log for errors:
Warning “lio_listo returned EAGAIN”Monitor from AIX
“pstat –a | grep aios”Use “-A” option for NMONIostat –Aq (new in AIX 5.3)
29
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
ASM configurations
AIX parameters– Async I/O needs to be enabled, but default values may be used
ASM instance parameters– ASM_POWER_LIMIT=1
Makes ASM rebalancing a low-priority operation. May be changed dynamically. It is common to set this value to 0, then increase to a higher value during maintenance windows
– PROCESSES=25+ 15n, where n=# of instances using ASM
DB instance parameters– disk_asynch_io=TRUE– filesystemio_options=ASYNCH– Increase Processes by 16– Increase Large_Pool by 600k– Increase Shared_Pool by [(1M per 100GB of usable space) + 2M]
30
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Basic AIX Configuration/Tuning for Oracle– Memory
– CPU
– I/O
– Network
– MiscellaneousRAC ConfigurationRAC Tuning
Agenda
31
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Network Parameters – all environments
Set sb_max >= 1 MB (1048576) (generally ok by default)
Set tcp_sendspace = 262144
Set tcp_recvspace = 262144
Set rfc1323=1
Also confirm these are set properly at network interface level
32
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Miscellaneous parameters
/etc/security/limitsSet to “-1” for everything except core for Oracle user
Sys0 attribute maxuproc >= 4096
Environment variables:AIXTHREAD_SCOPE=SNUM_SPAREVP=1 (AIX 5.1 only)
Use a 64-bit kernel
33
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Basic AIX Configuration/Tuning for Oracle– Memory
– CPU
– I/O
– Network
– Miscellaneous
RAC Configuration/Tuning
Agenda
34
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Full Cache FusionCache-to-cache data shipping Shared cache eliminates
slow I/OEnhanced IPC
Allows flexibleand transparentdeployment
Users
Shared Shared CacheCache
Drive and Exploit Industry Advances in Clustering
ClusteredDatabase Servers
Storage Area NetworkShared Disk Subsystem
Low Latency InterconnectVIA or Proprietary
High Speed Switch or Interconnect
Hub or Switch Fabric
Centralized Management Console
Cache FusionCache Fusion
RAC Architecture Summary
35
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Logical Shared Disk Architecture
Each RAC node requires access to all shared disk:Physical Shared disk
typically Fiber Channel or SAN attached
Best Practices: 1) Use 2 or more HBAs using multipathing software for load balancing and path failover2) plan for fabric redundancy
N o d e 1
N e t w o r k I n t e r c o n n e c t
S h a r e dD is k
N o d e 2 N o d e 3
36
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
GPFS (http://publib.boulder.ibm.com/clresctr/library/gpfs_aix_faq.html)See metalink note 302806.1 for details
10gR2: GPFS 2.3.0.3+10gR1: GPFS 2.1, 2.2, 2.3.01+9i: GPFS 2.1, 2.2, 2.3.0.1+
Link above includes references to supported storage platforms
HACMP Raw Logical Volumes10g: 5.1, 5.2 – NOTE: HACMP 5.3 not certified with 10g9i: 4.4.x, 4.5, 5.1, 5.2, 5.3
Automatic Storage Management (10g only)Check with Storage Vendor for support/recommended configurationCheck with Internal operations groups for management implications
eg, OS level backups, scripts to manage disk space, etc.Required for Standard Edition RAC
VERITAS SFRAC (http://support.veritas.com)
Other Notes:ORACLE_HOME directories must be local unless using GPFS or SFRAC.10g REQUIRES separate LUNs for Voting Disk and OCR partitions unless a clustered filesystem or HACMP is usedUse separate directories for CRS, ASM, ORACLE_HOME
Information subject to change, and additional requirements may apply.
Storage Software Options for Oracle RAC:
37
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Oracle Options for Data Storage
Voting Disk
OCR
Oracle Binaries
Archive Log Files
Control Files
Redo Log Files
Data Base Files
ASMGPFSRAW
38
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Clustering Software Requirements for RAC on pSeries:
Oracle 9i RAC:HACMP - 5.3 recommendedVeritas SFRAC 4.0
Other:PSSP 3.5 is required for use of SP switch and can replace HACMP as
clusterware
Oracle 10g RAC:Oracle CRS provides basic clusterware capabilityHACMP required for raw logical volume environments only - 5.2 currently
most recent certified versionVeritas SFRAC may be used if desired
Other:PSSP 3.5 is required for use of SP switch, but cannot function as
clusterware
Latest Requirements can be found at http://metalink.oracle.comCheck note # 282036.1 for detail
39
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Oracle 10g R2: Clusterware
Provides same base functions as 10g R1 CRS
Extended to provide protection for 3rd Party Application components
– Employs Application Virtual IPs for transparent network connectivity
– May require Clustered File System support for Application Configuration Files
– Applications do not necessarily have to reside on DB node
40
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Public Interface Requirements for RAC on pSeries:
Public interfaceAt least one network interface for client trafficMay use virtual ethernet if sufficient bandwidth – watch for 10.2.0.1 bug10g requires one virtual IP address for client traffic for each node
Validate that the application uses OCI calls and can take advantage of TAFJDBC Thick does not support use of OCI
TAF Supports failure/reconnect of failed connectionsOptional select statement failover
ORHardware load balancer/sprayerSoftware load balancerDNS CNAMES
41
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
RAC Interconnect Configuration
Gigabit Ethernet satisfactory for most applicationsIBM HPS offers lower latency/higher bandwidthIP over InfiniBand 10.2.0.1+UDP network tuning required for optimum performanceSwitched Networks REQUIRED – crossover cables not supportedVirtual Ethernet not yet supported
Oracle9i: Oracle “Fault Tolerant IP” (FTP-IP) feature provides Interconnect Network Fault Tolerance
Oracle10g: Etherchannel or 802.3ad Link Aggregation AIX options replace Oracle FTP-IP codeMay also be used with 9i CLUSTER_INTERCONNECTS parameter
N ode 1 N ode 2 N ode 3
P ublic N etw ork
P rivate N etw ork (B ackup)P rivate N etw ork (P rim ary) RAC Interconnect
42
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Private Interconnect Requirements for RAC on pSeries:
Private interconnect
Oracle 9i RAC:HACMP/RAW solutions:
cluster interconnect: minimum of 1, recommend 2 gigabit EthernetHACMP/GPFS solutions:
cluster interconnect: same as aboveadditional: private subnet required for GPFS traffic; can be 10/100 ethernet
VERITAS SFRAC:cluster interconnect: recommended 3 Ethernet interfaces, minimum 2
Oracle 10g RAC:HACMP/RAW solutions
cluster interconnect: minimum of 1, recommend 2 gigabit Ethernet
GPFS solutions:cluster interconnect: same as aboveadditional: GPFS traffic can share with the cluster interconnect!ASM:
cluster interconnect: same as above
43
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
RAC Interconnect: 9i R2 (Default Configuration)
Private Networks (normally 2) defined to HACMP
At instance startup, Oracle searches HACMP Network Topology to identify up to 3 eligible networks for Interconnect use
– Normally, the 2 private networks, plus 1 add’l public one
Networks are operated in High Availability “failover”(primary/backup/backup) mode via Oracle provided “Fault Tolerant IP” code
Oracle “Fault Tolerant IP” code registers DB connections with HACMP Event Manager (haemd) to assist with network failover
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation44 © 2006 IBM Corporation
RAC Interconnect: 9i R2 (Default Configuration)
en0en1en2en3
en0en1en2en3
Switch1Switch2
Public
Private
Node 1 Node 2
Oracle
AIX/HACMP
FT-IP
• Fault Tolerant
• Primary/Backup
• No Bandwidth Aggregation
Oracle
AIX/HACMP
FT-IP
10.0.0.1
10.0.0.2 10.0.0.4
10.0.0.3
45
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
10g RAC Interconnect Configuration
Best practice: Use 2 Gigabit Ethernet interfaces for interconnect
Oracle9i “Fault Tolerant IP” feature not available in 10gEtherchannel and 802.3ad Link Aggregation recommended
Network interface names must be the same on all nodes – eg, if en0 is a cluster interconnect, it must be a cluster interconnect on all nodes.
N ode 1 N ode 2 N ode 3
P ublic N etw ork
P rivate N etw ork (B ackup)P rivate N etw ork (P rim ary) RAC Interconnect
46
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
RAC Interconnect Configuration: 10g
HACMP Network Topology no longer used
Oracle “Fault Tolerant IP” code has been removed
Public and Private (RAC Interconnect) network names are specified at cluster configuration time
– Public interface names (e.g. en0) MUST be the same on all nodes
– Private interface names (e.g. en6) SHOULD be the same on all nodes
– Private networks should be non-routable
Fault Tolerance provided by either:– Logical Network with Primary/Backup adapters (9iR2 Alternative 2)
– EtherChannel or 802.3ad Link Aggregation
Requires protocol capable switches
Supports bandwidth aggregation and load balancing
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation47 © 2006 IBM Corporation
RAC Interconnect: 10g (Bandwidth Aggregation/Backup)
Oracle
en0en1en2en3
en0en1en2en3
Switch1
Public
Private
Node 1 Node 2
AIX/TCP-IP Fault Tolerant
Load Balanced or Active/Standby
en6
Oracle
AIX/TCP-IPen6
• Fault Tolerant
• Load Balanced
• Bandwidth Aggregation
Switch2
en4
en410.0.0.5 10.0.0.6
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation48 © 2006 IBM Corporation
Additional Network Parameters for RAC:Set udp_sendspace = db_block_size * db_file_multiblock_read_count +4k
• (not less than 65536)
Set udp_recvspace = 10 * udp_sendspace– Must be < sb_max
– Increase if buffer overflows occur
Ipqmaxlen=512 for GPFS environments
Use Jumbo Frames if supported at the switch layer
Time synchronization – use the “-x” flag with xntpd
Examples:no -a |grep udp_sendspaceno –o -p udp_sendspace=65536netstat -s |grep "socket buffer overflows"
49
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
GPFS tunablesSee metalink note 302806.1
Async I/O:Oracle parameter filesystemio_options is ignoredSet Oracle parameter disk_asynch_io=TRUEWorker1threads = GPFS asynch I/OPrefetchthreads= exactly what the name saysWorker1threads +prefetchThreads<=550Usually set prefetchthreads=default (64) and worker1threads=550-prefetchthreadsSet aio maxrequests=(worker1threads/#cpus) + 10
Other settings:GPFS block size is configurable; most will use 512KB-1MBPagepool – GPFS fs buffer cache, not used for RAC but may be for binaries. Default=64M
mmchconfig pagepool=100MMaxMBpS = maximum I/O that GPFS can submit per second. Default=150MBpS, should be set to approximate capacity of I/O subsystem
Ipqmaxlen=512No –r –o ipqmaxlen=512
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation50 © 2006 IBM Corporation
GPFS & Oracle – best practices
Have < 10 filesystems
Because of different usage patterns, use separate filesystems for binaries than Oracle data files
Use local filesystems for CRS HOME
When I/O to GPFS filesystems outside of Oracle instance I/O is expected, it can be desirable to increase the pagerpool parameter
OCR, vote placement is recommended on raw luns, rather than as GPFS files
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation51 © 2006 IBM Corporation
Scalability: OLTP Environments Scale-out tends to be good for OLTP environments when:
– There is low to moderate update activity– The workload is relatively uniform and predictable– The application is well designed and there are minimal lock/latch or serialization
related contention issues– A functional partitioning strategy is used to direct users to a limited number of
nodes–However, functional partitioning may reduce load balancing effectiveness
Industry benchmark proof points are limited (as of 11/1/05):– 1 out of 178 TPC-C benchmarks used RAC (16-node HP Integrity rx5670)
Relatively poor scale-out vs. non-RAC result on same hardwareNo demonstrable $/tpmC advantage vs. IBM p595 non-RAC results
– Limited number of SAP, Oracle E-Business Suite, PeopleSoft, other results
Most customer RAC environments are 2 or 3 nodes
A deployment involving a small number (2 or 3) of large nodes carries significantly less business risk than one involving many (4+) small nodes
IBM Advanced Technical Support - Americas
© 2003 IBM Corporation52 © 2006 IBM Corporation
Scalability: DSS Environments Scale-out tends to be good for Decision Support environments when:
– There is minimal data update or DDL operations during peak shift– Data is predominantly in large, partitioned tables which have even key/data
distribution– Data has a low “locality of reference” (low buffer cache hit%)– Are used predominantly for large query with Parallel Query Option (PQO) and
PARALLEL_AUTOMATIC_TUNING=TRUE– Query response time is not critical
Oracle focus on RAC based TPC-H benchmarks (as of 11/1/05):– 8 of 17 clustered results (all DB) were with Oracle RAC– 8 of 23 Oracle DB results were with Oracle RAC
Small number of known Oracle accounts > 1TB and/or involving more than 6 nodes
If you want to scale-out beyond 4-nodes, plan to do comprehensive stress testing before production deployment
53
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Oracle DocumentationOracle Reference Manuals:
http://otn.oracle.com/documentation/index.html
Oracle Whitepapers:http://otn.oracle.com/products/database/clustering/RACWhitepapers.html
Oracle (Metalink) Certification Info:http://otn.oracle.com/support/metalink/index.html
Oracle Database 10g Release 2 Automatic Storage Management Overview and Technical Best Practices
http://www.oracle.com/technology/products/database/asm/pdf/asm_10gr2_bptwp_sept05.pdf
Oracle Metalink:http://metalink.oracle.com– 282036.1
IBM Redbooks: (www.ibm.com/redbooks)Techdocs – Technical Sales Library:
http://www.ibm.com/support/techdocs
54
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
GPFS Documentation
GPFS for AIX home page:www.ibm.com/servers/eserver/pseries/software/sp/gpfs.html
GPFS for AIX Library:www.ibm.com/servers/eserver/pseries/library/gpfs.html– Concepts, Planning and Installation Guide" (GA22-7453)– Administration and Programming Reference (SA22-7452)– Problem Determination Guide (GA22-7434)
Clusters Library:http://www.ibm.com/servers/eserver/clusters/library/– GPFS for AIX Frequently Asked Questions
Oracle Metalink:http://metalink.oracle.com– 302806.1: IBM General Parallel File System (GPFS) and Oracle RAC on AIX
5L and IBM eServer pSeries
55
IBM Oracle Solutions - Advanced Technical Support - Americas
Oracle for AIX Workshop © 2006 IBM Corporation
Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S