Post on 06-Mar-2018
transcript
Oracle 11 g R1/R2 Real
Application Clusters Essentials
Design, implement, and support complex Oracle 11 g
RAC environments for real-world deployments
Ben Prusinski
Syed Jaffer Hussain
PUBLISHING
BIRMINGHAM - MUMBAI
] so88
Table of Contents
Preface 1
Chapter 1: High Availability.
J_
High availability concepts 7
Planned versus unplanned downtime 8
Service Level Agreements for high availability 8
High availability interpretations 9
Recovery time and high availability 10
System design for high availability 11
Business Continuity and high availability 11
Disaster Recovery 12
Business Continuity and Disaster Recovery guidelines 13
Fault-tolerant systems and high availability 14
Requirements for implementing fault tolerance 15
Fault tolerance and replication 16
High availability solutions for Oracle 17
Oracle Data Guard 17
Oracle Streams 17
Oracle Application Server Clustering 18
High availability: Oracle 11g R1 Real Application Clusters (RAC) 19
High availability: Oracle 11g R2 Real Application Clusters (RAC) 19
Summary 20
Chapter 2: Oracle 11g RAC Architecture 21
Oracle 11g RAC architecture 22
Certification matrix for Oracle 11g RAC architecture 23
Hardware architecture for Oracle 11 g RAC 25
Server configurations for Oracle 11 g RAC 26
CPU processors 26
Choosing between 32-bit and 64-bit CPU architectures 27
Dual core and multicore processors 29
Table of Contents
Network architecture for Oracle 11 g RAC 30
The private network and the Oracle 11 g RAC interconnect 30Choices for private interconnect and 11 g RAC 31
Redundancy for Ethernet interconnects with 11g RAC 33
Network bonding (NIC teaming) 39
Storage architecture for Oracle 11 g RAC 39
RAID configurations for Oracle 11g RAC 40
RAID 0 (striping) 42
RAID 1 (mirroring) 42
RAID 5 (striped with parity) 42
RAID 10 (striped mirrors) 43
Third-party RAID implementations 44IBM AIX LPAR disk volume management for RAID 44
Linux volume management for RAID configuration 45
Storage protocols for RAC 45
SCSI 46
Fibre Channel 47
Point-to-Point (FC-P2P) 47
Fibre Channel Arbitrated Loop (FC-AL) 47
Fibre Channel Switched Fabric (FC-SW) 48
Which Fibre Channel topology is best? 48
iSCSI 50
Which storage protocol is best for RAC? 50Asynchronous I/O versus Direct I/O for Oracle 11 g RAC 51
Oracle 11g RAC components 53
Voting Disk 53
Oracle Cluster Registry 53
Oracle 11g R1 RAC background processes 54ACMS Atomic Controlfile to Memory Service 54
GTXO-j Global Transaction Process 54LMON Global Enqueue Service Monitor 54LMD Global Enqueue Service Daemon 54
LMS Global Cache Service Process 54
LCK0 Instance Enqueue Process 55
RMSn Oracle RAC Management Processes 55RSMN Remote Slave Monitor 55
Oracle 11g R2 RAC background processes 56Grid Plug and Play 56
Grid Interprocess Communication 56
Multicast Domain Name Service 56
Oracle Grid Naming Service 56
How RAC differs from Oracle 11 g single-instance implementations 57
New ASM features and RAC 58
New SYSASM privilege for Oracle 11 g R1 ASM 58
Oracle 11 g R2 ASM features 59
Table ofContents
OCR and Voting Disk stored in ASM 59Oracle Automatic Storage Management Cluster Filesystem (Oracle ACFS) 59
New Oracle 11 g ASM Disk Group compatibility features 60
Summary 61
Chapter 3: Clusterware Installation 63
Preparing for a cluster installation 63
Server (node) requirements 64Network requirements 64Kernel parameters 65
Operating system packages 66OS groups and users 68OS user settings 68
Configuring Secure Shell (SSH) 69
Verifying prerequisites with the CLUVFY utility 71
Oracle 11 g R1 Clusterware installation 72
Initiating Oracle Universal Installer for Oracle 11 g R1 Clusterware 73What happens when orainstRoot.sh and root.sh is run? 82
Oracle 11 g R1 Clusterware post-installation checks 86
Installing Oracle 11g R1 RAC software 88
Initiating Oracle Universal Installer for Oracle 11 g R1 RAC software 88
Post-installation tasks 90Oracle 11 g R2 Clusterware installation 91
Initiating Oracle Universal Installer for 11 g R2 Clusterware 92What happens when the root.sh is run? 99Oracle 11 g R2 Clusterware post-installation checks 100
Installing Oracle 11g R2 RAC software 102
Initiating Oracle Universal Installer for Oracle 11 g R2 RAC software 103
Post-installation tasks 108
Oracle 11 g R2 Clusterware new features highlights 109
Removing/Reconfiguring a Grid Infrastructure configuration 110
Removing a successful Grid Infrastructure configuration 110
Reconfiguring a failed Grid Infrastructure configuration 111
Summary 112
Chapter 4: Automatic Storage Management 113
Overview of Automatic Storage Management (ASM) 114
Filesystem versus ASM storage architecture 115ASM disk 116ASM disk group 117
ASM instance configuration and management 118ASM initialization parameters 119
Creating an ASM instance 120Initializing DBCA 120
ASM background processes 122
Tabic of Contents
ASM dynamic views 122
V$ASM_DISK 123
V$ASM_DISKGROUP 124
V$ASM_OPERATION 124
V$ASM_DISK_STAT 124
V$ASM_DISKGROUP_STAT 124
V$ASM_CLIENT 125
ASM instance startup/shutdown 125
ASM disk group administration 126
Creating a disk group 127
Altering a disk group 128
Dropping a disk group 129
Overview of ASMCMD 130
ASMCMD in action 131
ASM 11 g R1 new features 133
ASM fast mirror resync 133
ASM preferred mirror read 134
ASM fast rebalance 135
ASM disk group compatibility attributes 135
ASM performance enhancements 136
New SYSASM role 136
ASM 11 g R2 new features 137
Automatic Storage Management Configuration Assistant (ASMCA) 137
Initiating ASMCA 137
Configuring a new ASM instance 138
Managing an ASM instance 140
Managing an ASM disk group 142
Creating an ASM disk group 143
Creating an ASM disk group in silent mode 144
Automatic Storage Management Dynamic Volume Manager (ADVM) 145
ADVM volume trivia 145
Creating ASM volumes 146
Automatic Storage Management Cluster File System 152
(ACFS) 152
Using ACFS as Oracle database home 153
Oracle ACFS drivers 153
Prerequisites for creating ACFS 154
ACFS creation methods 154
ACFS mount registry 161
Managing ACFS 161
ASM new background processes to support ACFS 163
Querying V$ASM views to obtain information about ACFS 164
ACFS snapshots 164
Creating a snapshot 166
Removing a snapshot 168
[iv]
Table of Contents
ASM Intelligent Data Placement (IDP) 169
Managing IDP settings 169
Finding IDP setting information 169
ASMCMD enhancements 170
ASM backup strategies 171
mdjoackup and md_restore commands 171
Summary 173
Chapter 5: Managing and Troubleshooting Oracle 11 g
Clusterware 175
Oracle 11 g RAC Clusterware administration 175
About Oracle Clusterware 176
Oracle 11 g Clusterware concepts 176
Oracle Cluster Registry 177
Voting disk 177
Initialization and shutdown scripts for Clusterware 179
Oracle 11 g Clusterware background processes 180
Cluster Ready Services Daemon 180
Additional background processes for Oracle 11g Clusterware 181
Fatal Clusterware processes and Oracle 11g RAC 184
Managing Oracle 11g Clusterware utilities 185
CRSCTL 185
CRS_STAT 188
OCRCHECK 190
OCRCONFIG 192
CLSCFG 193
CLUVFY 196
Troubleshooting Oracle 11g Clusterware 197
Failed, missing, or offline 11 g Clusterware resources 198
Offline Clusterware resources for Oracle 11 g RAC 199
Problems with the Voting disk and OCR 200
Vote disk issues with 11 g Clusterware resources 200
Failed or corrupted Vote Disks 201
Failed or corrupted OCR 202
How to recover the OCR from backup 202
Steps to perform recovery of lost and/or corrupted OCR 203
Check status 11 g RAC Clusterware 204
Root cause analysis 11 g RAC 205
Oracle 11 g RAC node reboot issues 206
Oracle 11 g RAC Clusterware processes—node reboot issues 207
Root cause analysis for solving node reboots with 11 g RAC 208
OCSSD Reboots and 11 g RAC 210
OPROCD failure and node reboots 211
OCLSOMON-RAC node reboot 211
[v]
Table of Contents
Hardware, storage, and network problems with RAC 212
Hardware, storage, and network resolutions 215
New features in Oracle 11 g R2 Clusterware 215
Oracle Real Application Clusters one node (Oracle RAC one node) 216
Improved Oracle Clusterware resource modeling 216
Policy-based cluster and capacity management 217
Cluster time synchronization service 217
Oracle Cluster Registry and voting disks within Oracle ASM 217
New features for upgrading to Oracle 11 g Clusterware 218
Oracle 11 g R2 Cluster Verification Utility new features 218
Zero downtime patching for Oracle Clusterware 218
Summary 219
Chapter 6: RAC Database Administration and Workload
Management 221
RAC database configuration and creation 222
Creating a database using DBCA 222
Choosing database storage options 231
We have created a database using DBCA—now what? 241
What's new in Oracle 11g R1 and R2 databases? 243
Automatic Memory Management 244
New AMM dynamic performance V$ views 245
Tuning AMM 246
Database Smart Flash Cache 248
Configuring Smart Flash Cache 248
Instance caging 249
New background processes in Oracle 11 g 249
Finding the alert.log file location in Oracle 11g 250
Automatic Diagnostic Repository 251
V$DIAG_INFO view 252
RAC database administration 253
Using the Server Control Utility 253
Automatic Workload Management 255
Overview of services 255
Creating and managing services 256
What's new in Oracle 11 g services' behavior? 262
Scalability (Load Balancing) 263Client Side Connect Time Load Balance 263
Server Side Listener Connection Load Balance 264
Transparent Application Failover 265
Configuring Transparent Application Failover 266
Fast Connection Failover 267
Configuring Fast Connection Failover 267
Summary 268
Table of Contents
Chapter 7: Backup and Recovery 269
An overview of backup and recovery 270
An overview of Recovery Manager (RMAN) 270
RMAN architecture 272
RMAN performance tuning tips 273
Backup types and methods 274
Logical backup 274
Physical backup 275
ONLINE RMAN backups 275
OFFLINE RMAN backups 275
RMAN new features in 11g R1 and 11g R2 275
Database Recovery Advisor 276
Multisection backups for very large datafiles 276
Undo tablespace backup optimization 277
Faster backup compression 277
Active database duplication 278
Archivelog deletion policy enhancements 280
Automatic Block Recovery (ABR) 280
Tablespace point-in-time recovery enhancements 281
RMAN best practices for RAC 281
Configuring the Flash Recovery Area for a RAC database 282
Instance recovery versus Crash recovery in RAC 283
Parallelism for backup and recovery in RAC 287
Backing up a RAC database with RMAN 287
Configuring multiple channels 289
OCR and Voting disk backup and recovery strategies 290
Adding a Mirror location for the OCR and Voting disk 291
OCR automatic backups 291
Performing OCR manual backups 293
Voting disk manual backups 294
Restoring OCR 294
Restoring the Voting disk 297
Summary 298
Chapter 8: Performance Tuning 299
Tuning differences: single instance versus RAC 300
Oracle 11 g single instance database 300
Oracle RAC 11g database 300
New Oracle 11g performance tuning features 300
Database Replay 301
SQL Performance Analyzer 302
Database Health Monitor 302
Table of Cantenh
PL/SQL Native CompilationServer Result Cache
Client Side Result Cache
SQL Tuning Advisor
New performance features in Oracle 11gR2In-Memory Parallel Execution
Analyzing the Cache Fusion impact on RAC performance
Cache Fusion
Latency statistics
RAC wait events
Monitoring RAC cluster interconnect performanceOracle cluster interconnects
Monitoring RAC wait events
Summary
Chapter 9: Oracle 11g Clusterware UpgradeOverview of an upgrade 323
Upgrade sequence 324
Upgrading Oracle 10g R2 Clusterware to Oracle 11g R1 324
Kernel parameter values 325
Packages required on Linux 5 326
Oracle 11 g R1 Clusterware upgrade steps 326
Performing preinstallation checks with cluvfy 327
Executing runlnstaller sh script 327
Post-upgrade steps for 11 g R1 Clusterware 332
Upgrading to Oracle 11 g R2 Clusterware 333
Overview of our environment 333Upgrading nodes 334
11g R2 upgrade changes and restrictions 334
Kernel parameter values 335
Packages required on Linux 5 335
Performing preinstallation checks with cluvfy 336
Oracle 11g R2 Clusterware upgrade steps 336Executing the runlnstaller.sh script 337Post-upgrade checks for 11 g R2 Clusterware 349
Post-upgrade steps for 11g R2 Clusterware 351
Downgrading Oracle Clusterware after an upgrade 352
Summary 353
Chapter 10: Real-world Scenarios 355
Adding a new node to an existing cluster 356
Performing prechecks with the cluvfy utility 357addNode sh 360
Adding a node in silent mode in Oracle 11 g R2 363
303
304
305
306
307
307
307
308
309
310
315
315
316
321
323
Table of Contents
Post-installation status checks for Clusterware 364
OCR file manual backup syntax 368
Voting Disk backup syntax 368
Installing ASM and RDBMS software using addNode.sh script 368
Cloning ASM software using addNode.sh script on Oracle 11 g R1 369
Post-node addition steps 370
Removing a node from the cluster 370
Adding an RAC database instance 373
Adding a new instance using DBCA 374
We have added an instance—what next? 379
Verifying new instance status 379
Using DBCA in silent mode to add an instance 380
Post-add instance steps 380
Deleting an RAC database instance 380
Using DBCA in silent mode to delete an instance 381
Converting a single-instance database to an RAC database 382
Overview of RCONFIG command-line tool 383
What you need to accomplish the conversion task 384
Sample of a modified ConvertToRAC.xml input file 384
How to test a conversion without actually performing the conversion 386
Converting a single-instance database to an RAC database 387
How to resume a failed rconfig operation 389
Checking log files 389
How to optimize rconfig to run faster 389
Post-conversion steps 390
Relocating an RAC database and instances across nodes 390
Relocating the instance 391
Adding the instance example 392
Workaround when a database and instance are configured on the
same node 392Adding the database example 393
Post-relocation steps 394
Summary 394
Chapter 11: Enabling RAC for EBS 395
EBS architecture 396
Oracle 11 g RAC suitability 399
Installing EBS 12.1.1 400
EBS implementation on Oracle 11g RAC 407
RAC-enabling EBS 12.1.1 410
Configuration prerequisites 410
ASM and RAC-enabling the EBS database with the rconfig utility 412
Running AutoConfig 418
Table of Contents
Copying AutoConfig to the new RDBMS ORACLE_HOME for
Oracle 11g R1 11.1.0.7 419
Generating your database context file 422
Preparing for AutoConfig by completing the following AutoConfig steps 422
Generating and applying AutoConfig configuration files 423
Executing AutoConfig on all database nodes in the cluster 424
Performing Init file, tnsnames, and listener file activities 424
Establishing applications environment for Oracle RAC 426
Setting up load balancing 427
Configuring Parallel Concurrent Processing 428
Prerequisites for setting up Parallel Concurrent Processing 428
Cloning EBS concepts in brief 429
Preparing the source system 430
Copying the source system 430
Configuring the target system 431
Adding a new node to an existing EBS system 431
Setting up Parallel Concurrent Processing 433
Setting up Transaction Managers 434
Setting up load balancing on concurrent processing nodes 435
Summary 435
Chapter 12: Maximum Availability 437
Oracle 11 g Streams for RAC 438
Oracle 11g Streams architecture for RAC 438
Capture 438
Staging 439
Propagation 439
Consumption 439
Default apply 440
User-defined function apply 440
Explicit de-queue 440
Understanding Oracle Streams rules 440
Transformations and Streams 440
Capture and Apply processes in an RAC instance 441
Streams in the RAC environment 441
New features in Oracle 11g Streams 442
Synchronous Capture 442
Splitting and merging of a Stream Destination 442
Tracking LCRs through a Stream 443
Streams Topology and Performance Advisor 443
Combined Capture and Apply 443
Best practices for Streams in an RAC environment 444
Additional configuration of RAC environments for a Source Database 444
Queue ownership 445
Propagation restart 446
Table of Contents
Changing the GLOBAL_NAME of the source database 447
Additional configuration for RAC environments for the Apply Database 448
Changing the GLOBAL_NAME of the Target Database 448
New features for Streams in Oracle 11 g R2 448
XStream 449
Statement DML Handlers 449
Ability to record table changes 449
SQL generation 449
Support for compressed tables 450
Support for SecureFile LOBs 450
Automatic splitting and merging 450
New Apply process parameter 450
Monitoring jobs 451
New 11 g R2 Oracle Streams view 451
Oracle 11 g Data Guard and RAC 451
New features for Oracle 11g Data Guard 451
Active Data Guard 452
Snapshot Standby 452
Configuring Data Guard Physical Standby for 11g RAC 452
Configuring Oracle RAC primary database to send redo data 453
Design considerations in an Oracle RAC environment 453
Format for archived redo log filenames 453
Troubleshooting Oracle 11 g Data Guard and RAC 454
Switchover fails in an Oracle 11 g RAC configuration 455
How to recover from corrupt datafile on standby 455
How to recover from a corrupt block on standby 457
Automatic repairing of corrupt data blocks 457
New features for Data Guard in Oracle 11 g R2 458
New Oracle Data Guard 11g R2 features for Redo Apply 458
New Oracle 11 g R2 Data Guard features for SQL Apply 459
Summary 460
Appendix: Additional Resources and Tools for the Oracle
RAC Professional 461
Sample configurations 461
Reviewing and resolving manual configuration changes 464
adchkcfg utility 464
Oracle RAC commands and tips 467
Cluster deconfig tool for Oracle RAC 468
Using the cluster deconfig tool 469
Limitations of the cluster deconfig tool 470
Problems and limitations of the cluster deconfig tool 470
Starting the cluster deconfig tool 471
Silent mode operations using cluster deconfig 471
Table ofContents
Manual cleanup for RAC 474
Repairing the RAC environment without reinstalling 476
Reinitializing OCR and Voting Disks without reinstalling RAC 476
Using ROOTDELETE.SH in debug mode 478
Using rootdeinstall.sh 480
Reinstalling CRS on the same cluster in another CRS_HOME 480
Stopping CRS processes 480
Reinstalling CRS on same cluster in another CRSJHOME 481
Oracle 11g R2 cluster removal tools for RAC 481
Tracing RAC issues with Oradebug 482Using Oradebug to trace Oracle 11g Clusterware 485
Server Control Utility 486
Oracle 11 g R2 SRVCTL commands 486
Managing Oracle Clusterware with the CRSCTL utility 487Differences between 11 g R1 and 11 g R2 syntax for CRSCTL 487
Operating system-level commands for tuning and diagnosis 492
Strace 492
Truss 495
GDB 496
Additional references and tips 499
Clusterware startup sequence for Oracle 11g R2 500
Index 503
[xii]