Oracle RAC 11gR2
RAC Joins the Cloud
RAC Grows Up RAC 11gR2 (aka 12gR1) adds a number of
features that transforms RAC clusters into database clouds
Many features are geared towards enterprises but there are new features that any organization can take advantage of
With today’s high power x86 and 11gR2 RAC it is entirely possible to replace many silo clusters with fewer 11gR2 “clouds”
Many 11gR1 and older RAC commands are depreciated
11gR2 RAC New Features Brand New Installer Clusterware is now Grid Infrastructure ASM is now part of the Grid Infrastructure
installation Voting and OCR disk are ASM diskgroup
Raw devices are only supported for upgraded clusters Future Linux release may drop support for raw devices
Plug and Play For Clusters Cluster is now a DNS service DHCP for server VIPs
New tools are introduced old tools retired
RAC 11gR2 Install
Non Linear Installer
Grid Plug and Play
SSH Keys
OCR/Voting Disks
OCR/Voting Disks OCR/Voting disks can be ASM Diskgroup
Raw devices are only supported for upgraded clusters
Other options are NFS and certified clustered filesystems
3 disks for normal redundancy, 5 disks for high
IPMI
IPMI (cont) Intelligent Platform Management Interface Defines a cross vendor standard to monitor
and manage hardware Dell, HP, IBM, Intel, AMD and dozens more IPMI in relation to RAC allows other nodes to
manage the server regardless of the OS state
ASM Roles
Fix Common Problems Installer can create a script that can fix some
problems Won’t install missing packages
Finish Install After GUI finished is will ask to run the
command oraInrootinst.sh and root.sh
Cluster Startup Sequence Init starts Oracle High Availability Service Deamon
(OHASD) OHASD starts
cssdagent – starts cssd orarootagent – manages all root owned ohasd
processes oraagent – manages all oracle owned ohsad resources cssdmonintor – monitors cssd and node health
OHASD rootagent starts crsd – main deamon for managing cluster resources ctssd – time synchronization Diskmon ACFS – ASM cluster file system drivers
Cluster Startup Sequence Oraagent starts – level 2
mdnsd – used for DNS lookups gipcd – inter-process and inter-node communication gpnpd – Grid Plug & Play deamon EVMD – event monitor ASM – resource for monitoring ASM instance
CRSD – level 3 orarootagent oraagent
CRSD rootagent – level 4 SCAN VIP(s) - Single Client Access Name Node VIPs – one per node ACFS Registry – For mounting ASM Cluster File System GNS VIP - optional
SCAN SIMPLE CLIENT ACCESS NAME
A single DNS entry that represents the cluster has a whole
Clients no longer need to know the actual servers that are in a cluster
THIN connections or connections not using a VIP are better protected against a node failure
If using GNS SCAN must be in the domain managed by GNS
DNS needs to be able to delegate resolution to GNS
If not using GNS DNS should be configured to round robin up to 3 IPs for the SCAN name. Recommend use 11.2 client
If using GNS 3 IPs will be requested from DHCP
SCAN Listener
Listener that is dependent on a SCAN VIP If the SCAN VIP moves to a different node in the cluster
the listener will move with the VIP Depending on the number of nodes in a cluster a single
node may have more than one SCAN VIP and listener remote_listener parameter should be set to a SCAN VIP
for 11gR2 databases Simplify RAC Dataguard connections for clients
Two new sqlnet parameters CONNECT_TIME = timeout in seconds RETRY_COUNT = number of tries In a single TNS entry enter both primary and standby servers SCAN
address. If the connection is made to standby it will failover to primary.
SCAN Load Balancing Client receives IP from DNS or GNS that
resolves to SCAN VIP SCAN Listener bound to SCAN VIP redirects
client to local listener that has the least load Client connects to the local listener on the
node VIP Client needs to be able to connect to SCAN
VIP and to the local node VIP and be able to resolve all names
SCAN SCAN Listeners Overview
Grid - Plug And Play Greatly simplifies the adding/removing and
connecting to a cluster Basically run cluster verify script and then
addNode.sh Configures cluster listener and ASM
No longer use static IPs for VIPs
GNS Grid Naming Service
Handles name resolution for the cluster Requires DHCP for VIP addresses! DNS needs to delegate resolution for the cluster to GNS Use the GNS VIP not server or scan VIP
Bind Example prodcluster.db.oracle.com NS gns.prodcluster.db.oracle.com gns.prodcluster.db.oracle.com. 10.90.51.20
When a client connects the cluster will provide the IP address
As RAC servers are added DHCP assigns a VIP address, GNS is updated with the new server’s VIP address and now can route connects to the new server. No client connection strings are never changed Remove a node is basically reversing the process
Client Connections With GNS
ASM Changes ASM part of grid infrastructure – one less HOME
to manage New GUI management tool – asmca
Replaces DBCA for ASM management Create diskgroups, volumes, ACFS mounts Enhanced controls to manage ASM attributes
SPFILE is stored in ASM diskgroup Cluster reads the diskgroup before ASM starts ASMCMD is greatly enhanced and mostly
replaces sqlplus ASM roles can play a large role in managing ASM Diskgroups can be cluster resources
ASM Changes (cont) New O/S aware cluster file system (ACFS)
ASM disks can be used to create filesystem mounted by O/S
Only supported on Redhat/Oracle Linux, other O/S support coming in the future
If a file can natively support ASM then the file is not supported on ACFS! No data, control or redo files
DirectIO is not available Take advantage of ASM performance over other filesystems ACFS is able to do snapshots
Still able to access ASM diskgroups with FTP/HTTP via XMLDB
Copy command can copy out and in of ASM (11.1.0.7)
ASM Changes (cont) Intelligent placement of data
ASM is able to put hot datafiles on the outer edge of the disk
Can provide a 50% increase in I/O performance Compatible parameters for both ASM & RDBMS
11.2 Work best with the geometry of the disks are know
JOBD – uses disks sectors for data placment LUN – logical sectoring number which may not have any
relation to the physical layout of the lun on the disk Diskgroup should be 50% full for full benefit
alter diskgroup data add template datafile_hot attributes (hot mirrorhot);
ascmd lstmpl –l –G data
ASM – Access Control Lists Set permissions at the ASM file level Set permissions for users and groups User is the oracle database software owner Only available for Linux and Unix Not so much for security as for separation of
duties Files created in ASM are owned by DBUSER Create a separate OSDBA group for each
database using a separate ORACLE_HOME. Need different groups for OSASM and OSDBA for ASM
Compatible 11.2 diskgroup both for ASM and RDBMS
ASM – Access Control Lists (cont) Disk attributes
ACCESS_CONTROL.ENABLED = TRUE ACCESS_CONTROL.MASK has to be set
Mask values 6 removes all 2 removes write 0 removes nothing
Mask of 026 sets values of 640 read-write for owner, read for group and nothing for
everyone else
asmcmd> setattr –d data access_control_enabled true
sql> alter diskgroup data set attribute ‘access_control.enabled’ = true;
ASM Permissions ALTER DISKGROUP ADD USERGROUP … WITH MEMBER ALTER DISKGROUP DROP USERGROUP ALTER DISKGROUP MODIFY USERGROUP ADD MEMBER ALTER DISKGROUP MODIFY USERGROUP DROP
MEMBER ALTER DISKGROUP ADD USER ALTER DISKGROUP DROP USER ALTER DISKGROUP SET PERMISSION
ALTER DISKGROUP SET OWNERSHIP SELECT * FROM V$ASM_USER SELECT * FROM V$ASM_USERGROUP ALTER DISKGROUP ADD USERGROUP myfiles WITH
MEMBER bill;
ASM Roles OS groups control rights in ASM
OSDBA – can access ASM files and set ACLs ASMOPER – can start/stop ASM OSASM – full ASM control
Allow people to administer ASM without having sysdba rights to the databases
ASM Commandline Tool
ASM Commandline Tool (cont) Create diskgroups from ascmd
Same syntax as sqlplus Create diskgroup mydata external redundancy DISK ‘ORCL:DISK1’ NAME mydata1, ‘ORCL:DISK2’ NAME
mydata2;
Create diskgroup from XML via asmcmd asmcmd> chkdg
ASM Commandline Tool (cont) lsdg – list diskgroup information
Very similar to select * from v$asm_diskgroup lsdsk – list disk information
Same information found in select * from v$asm_disk
Cluster Registry & Voting Disks Raw devices only supported for upgraded
clusters Can move OCR and Voting disks to ASM
diskgroup after upgrade ocrconfig add <diskgroup name> ocrcheck ocrconfig delete <raw device> crsctl query css votedisk crsctl add css votedisk <diskgroup name> crsctl delete css votedisk <raw device>
Cluster Management crsctl in the grid infrastructure home manages
almost every cluster command crs_* scripts are depreicated
Depreciated Commands crs_stat crs_register crs_unregister crs_start crs_stop crs_getperm crs_profile crs_relocate crs_setperm crsctl check crsd crsctl check cssd crsctl check evmd crsctl debug log crsctl set css votedisk crsctl start resources crsctl stop resources
Server Pools Create “mini” clusters inside of larger clusters
for policy based databases Two default pools Free/Generic
Free pool contains servers not assigned to another pool Generic pool is for running pre 11.2 databases and non
policy managed databases Specify min/max nodes and priority
Cluster manages the members of the pool based on load, availability, priority and min/max requirements
Cluster will move servers from free and lower priority pools to meet the needs of the server pool
Can create ACL on server pools for role based management
Vote/OCR Disk Management Raw is supported for upgraded clusters but it
is possible to migrate to ASM disks after 11.2 cluster upgrade
Move vote disks to ASM crsctl query css votedisk crsctl replace votedisk +asm_disk_group
Move OCR to ASM ocrconfig -add +new_disk_group ocrconfig -delete old_storage_location
Troubleshooting Cluster Problems Golden Rule – Ensure time is synced across cluster
Help with comparing log files Ensures nodes aren’t evicted due to time skew If NTP is not used Oracle will use Cluste Time
Synchronization service Recommend to have cluster down if changing the time
Logs are stored under GRID_HOME/log/hostname Main alert log Separate directory for each cluster process
Diagnostic Script GRID_HOME/bin/diagcollection.pl
Collects cluster and system logs for debugging
Troubleshooting (cont) Modify time before node is evicted
Default is 30 seconds crsctl set css miscount 45
On busy systems logs may not flush before node reboots Crsctl set css diagwait 13 – force
Not set by default Requires cluster outage to set
Adding A Node Much easier in 11gR2 Setup ssh keys for new host Run cluster verify tool
cluvfy stage –pre crsint –n <new host> <-fixup> Fixes any problems
cluvfy stage –pre nodeadd –n <new host> $ORACLE_HOME/oui/bin/addNode.sh –silent
“CLUSTER_NEW_NODES=<new host>” “CLUSTER_NEW_VIRTUAL_HOSTNAMES=<new host vip>”
Instance Caging Puts a cap on the number of CPUs a instance
will use Set the cpu_count parameter Enable resource manager with CPU polices
Oracle Joins The Cloud With 11gR2 it is now possible to create few
larger clusters while being able to maintain performance, security and reassure customers concerns Increased utilization rates Lower administration costs Happy Management