Maximum Availability Architecture - ITtoolboxhosteddocs.ittoolbox.com/db_us_en_wp_maxyexa.pdf ·...

Deploying Oracle Maximum Availability Architecture with Exadata Database Machine Oracle Maximum Availability Architecture White Paper

June 2011

Maximum Availability Architecture Oracle Best Practices For High Availability

Oracle White Paper—Deploying Oracle Maximum Availability Architecture with Sun Oracle Database Machine and Oracle Exadata

Executive Overview ........................................................................... 2

Exadata MAA Architecture................................................................. 3

Initial Deployment .............................................................................. 5

Hardware Components .................................................................. 6

Software Components: .................................................................. 7

High Performance .......................................................................... 8

Post Deployment – Exadata MAA Configuration ................................ 8

Database Archivelog Mode and Enable Database Force Logging . 9

Fast Recovery Area ....................................................................... 9

Oracle Flashback Technologies ..................................................... 9

Backup, Restore, and Recovery .................................................. 11

Oracle Data Guard and Oracle Active Data Guard ...................... 11

Comprehensive Data Corruption Protection ................................. 12

Application Clients - Failover Best Practices ................................ 13

Operational Best Practices for Exadata MAA ................................... 14

Importance of a Test Environment ............................................... 16

Conclusion ...................................................................................... 17

Appendix 1: Exadata MAA Outage and Solution Matrix ................... 18

Unplanned Outages ..................................................................... 18

Planned Maintenance .................................................................. 20

Appendix 2: Exadata MAA Performance .......................................... 22

References ...................................................................................... 24


2

Executive Overview

The integration of Oracle Maximum Availability Architecture (MAA) operational and configuration best practices with Oracle Exadata Database Machine (Exadata MAA) provides the most comprehensive high availability solution available for the Oracle Database.

This paper provides Exadata MAA insight for rapid deployment and efficient operation of Exadata Database Machine. It is divided into four main areas:

1. Exadata MAA Architecture

2. Initial Deployment

3. Post Deployment: Exadata MAA Configuration

4. Operational Best Practices for Exadata MAA


3

Exadata MAA Architecture

Exadata MAA architecture shown in Figure 1 is designed to tolerate unplanned outages of all types, providing both high availability (HA) and data protection. Exadata MAA also minimizes planned downtime by performing maintenance in a rolling fashion. The range of potential outages and planned maintenance addressed by Exadata MAA are described more fully in Appendix 1 of this paper: Exadata MAA Outage and Solution Matrix.

Figure 1. Basic Oracle Exadata Database Machine Configuration

The Exadata MAA architecture consists of the following major building blocks:

1. A production Exadata system (primary). The production system may consist of one or more interconnected Exadata Database Machines as needed to address performance and scale-out requirements for data warehouse, OLTP, or consolidated application environments.

2. A standby Exadata system that is a replica of the primary. Oracle Data Guard is used to maintain synchronized standby databases that are exact, physical replicas of production databases hosted on the primary system. This provides optimal data protection and high availability if an unplanned outage makes the primary system unavailable. A standby Exadata system is most often located in a different data center or geography to provide disaster recovery (DR) by isolating the standby from primary site failures. Configuring the standby system with identical capacity as the primary also guarantees that performance service-level agreements can be met after a switchover or failover operation. Note that Data Guard is able to support up to 30 standby databases in a single configuration. An increasing number of customers use this flexibility to deploy both a local Data Guard standby for HA and a remote Data Guard standby for DR. A local Data Guard standby database complements the internal HA features of Exadata Database Machine by providing an additional layer of HA should unexpected events or human error make the


4

production database unavailable even though the primary site is still operational. Low network latency enables synchronous redo transport to a local standby resulting in zero data loss if a failover is required. A local standby database is also useful for offloading backups from the primary database, for use as a test system, or for implementing planned maintenance in rolling fashion (e.g. database rolling upgrades). The close proximity of the local standby to the application tier also enables fast redirection of application clients to the new primary database at failover time. Following a failover or switchover to a local standby database, the remote standby database in such a configuration will recognize that a role transition has occurred and automatically begin receiving redo from the new primary database - maintaining disaster protection at all times. While the term ‘standby’ is used to describe a database where Data Guard maintains synchronization with a primary database, standby databases are not idle while they are in standby role. High return on investment is achieved by utilizing the standby database for purposes in addition to high availability, data protection, and disaster recovery. These include:

o Active Data Guard enables users to move read-only queries, reporting, and fast incremental backups from the primary database, and run them on a physical standby database instead. This improves performance for all workloads by bringing the standby online as a production system in its own right. Active Data Guard also improves availability by performing automatic repair should a corrupt data block be detected at either the primary or standby database, transparent to the user.

o Data Guard Snapshot Standby enables standby databases on the secondary system to be used for final pre-production testing while they also provide disaster protection. Oracle Real Application Testing can be used in conjunction with Snapshot Standby to capture actual production workload on the primary and replay on the standby database. This creates the ideal test scenario, a replica of the production system that uses real production workload – enabling thorough testing at production scale.

o Oracle Patch Assurance using Data Guard standby-first patching (My Oracle Support Note 1265700.1) or Data Guard Database Rolling Upgrades are two methods of reducing downtime and risk during periods of planned maintenance. This is a key element of Exadata MAA Operational Best Practices discussed later in this paper.

3. A development/test Exadata system that is independent of the primary and standby Exadata systems. This system will host a number of development/test databases used to support production applications. The test system may even have its own standby system to create a test configuration that is a complete mirror of production. Ideally the test system is configured similar to the production system to enable:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1265700.1�


5

o Use of a workload framework (e.g. Real Application Testing) that can mimic the production workload.

o Validation of changes in the test environment - including evaluating the impact of the change and the fallback procedure - before introducing any change to the production environment.

o Validation of operational and recovery best practices.

Some users will try to reduce cost by consolidating these activities on their standby Exadata Database Machine. This is a business decision that represents a trade-off between cost and operational simplicity/flexibility. In the case where the standby Exadata Database Machine is also used to host other development and test databases, additional measures may be required at failover time to conserve system resources for production needs. For example, non-critical test and development activities may have to be deferred until failed system is repaired and back in production.

The Exadata MAA architecture provides the foundation needed to achieve high availability. Equally important are configuration and operational practices described in the sections that follow.

Initial Deployment

Oracle Exadata Database Machine is a pre-optimized, pre-configured, integrated system of software, servers, and storage that comes ready-built to implement Exadata MAA. This section focuses on what is pre-configured and the HA best practices that apply to initial deployment of the Exadata Database Machine.

Oracle recommends that the following configurable deployment options be specified when completing the Exadata Database Machine Configurator spreadsheet prior to delivery of the system:

• Choose HA bonded networks for client access.

• Choose Oracle Automatic Storage Management (ASM) high redundancy for DATA and RECO disk groups for best tolerance from data and storage failures. ASM high redundancy is the ideal recommendation for business critical databases. Note that Exadata Database Machine Half Rack is the minimum system required to place OCR and voting devices in a high redundancy ASM disk group. ASM high redundancy will maintain availability in the event of: double disk failures, disk failure while another Exadata storage cell is down for maintenance (such as during a rolling upgrade), and disk failure followed by a read error (sector failure) on the redundant disk. Optionally, if space pressure makes high redundancy impractical for


6

both DATA and RECO disk groups then an acceptable compromise its to configure high redundancy for DATA and normal redundancy for RECO disk groups.

The Exadata Database Machine Configurator spreadsheet validates and generates configuration files that will be used as part of the automated deployment. After the Exadata Database Machine is delivered, an Oracle systems engineer will validate the hardware and run a final check for any components that may have been damaged while in-transit. Next, Oracle Advanced Support Services or Consulting Services will execute the Oracle OneCommand utility using the supplied configuration files.

Within a few days after hardware arrival the system and database will be fully functional with:

● Validated hardware, InfiniBand network, and Exadata storage cells that are available and performing according to specifications.

● Recommended operating system, firmware and Oracle software and patches as described in My Oracle Support Note 888828.1

● An Exadata Storage Grid consisting of DATA, RECO and DBFS ASM disk groups. DATA and RECO diskgroups are configured using the ASM redundancy option(s) previously selected in the Exadata Database Machine Configurator spreadsheet.

● Network infrastructure that includes InfiniBand networks for all communication between database and Exadata storage servers. The network is configured for both client and management access.

● Oracle Real Application Clusters (Oracle RAC) database and Grid Infrastructure preconfigured using Exadata MAA best practices and performance settings

The HA characteristics inherent in a preconfigured Exadata Database Machine are described in the following sections.

Hardware Components

The following hardware and component redundancy is common to all models of Exadata Database Machine X2-2 and X2-8:

Redundant database servers

The Exadata Database Machine comes preconfigured with multiple, industry-standard Oracle Database servers running Oracle Database 11g Release 2 (11.2).

Redundant storage



7

Exadata Storage components - local drives, Exadata disks, and Oracle Exadata Storage Server (Exadata cell)- are all redundant. Exadata cells are network-accessible storage devices with Oracle Exadata Storage Server Software pre-installed. Database data blocks and metadata are mirrored across cells to ensure that the failure of an Exadata disk or Exadata cell does not result in loss of data or availability.

Redundant connectivity

Redundant InfiniBand network connectivity using dual-ported Quad Data Rate (QDR) Host Channel Adapters (HCAs) and redundant switches are pre-configured. Configuring the same redundancy for client access is recommended and can be done at deployment time.

Redundant power supply

Each Exadata Database Machine has redundant power distribution units (PDUs) for high availability. The PDUs accept separate power sources and provide a redundant power supply to:

● Oracle Database nodes

● Exadata Storage Cells

● InfiniBand switches

● Cisco network switch

Power supplies for Oracle Database nodes, Exadata Storage Cells, InfiniBand and Cisco switches are all hot swappable.

Software Components:

The following are standard Oracle software components explicitly optimized and validated for Exadata Database Machine.

Firmware and Operating System

All database and Exadata storage servers are packaged with validated firmware and operating system software preinstalled.

Database Server Tier

Grid Infrastructure (Oracle Clusterware and ASM) and Oracle RAC software is installed and patched to recommended software version at deployment, enabling applications to tolerate and react to instance and node failures automatically with zero to near zero application brown out. As described in Appendix 1, all Grid Infrastructure patches and most database patches can be applied in rolling fashion.

Storage Tier


8

The storage tier with ASM and Exadata software supports high availability by:

● Tolerating disk and cell failures

● Applying many software changes in a rolling manner

Exadata storage cells include Oracle Hardware Assisted Resilient Data (HARD) to provide a unique level of validation for Oracle block data structures such as data block address, checksum and magic numbers prior to allowing a write to physical disks. HARD validation with Exadata is automatic (setting DB_BLOCK_CHECKSUM is required to enable checksum validation). The HARD checks transparently handle all cases including ASM disk rebalance operations and disk failures.

High Performance

Oracle Development teams who focus on high performance for OLTP and Data Warehouse applications have optimized the configuration defaults that are pre-set on Exadata Database Machine. In some cases, there will be different default settings for Exadata Database Machine X2-2 and X2-8. These settings are the result of extensive performance testing with various workloads, both in Oracle labs and in real-world Exadata Database Machine deployments.

Validated configuration defaults have also undergone fault injection testing in a full MAA environment that includes Oracle RAC, ASM, RMAN, Flashback, and Data Guard. Oracle strongly recommends that you DO NOT use initialization settings from previous Oracle configurations. Begin with the default configuration delivered with Exadata Database Machine and then tune if required. Most customers will find minimal need for tuning database parameters in the post deployment phase.

Post Deployment – Exadata MAA Configuration

Additional configuration using HA best practices in each of the following areas is required for complete implementation of Exadata MAA:

● Database Archivelog Mode and Enable Force Logging

● Fast Recovery Area

● Oracle Flashback Technologies

● Backup, Restore, and Recovery

● Data Guard and Active Data Guard

● Comprehensive Data Corruption Protection


9

● Automatic Failover of Application Clients

Database Archivelog Mode and Enable Database Force Logging

Archivelog Mode and Database Force Logging are prerequisites to insure all changes are captured in the redo and applied during database recovery operation. Enable database archive logging and database force logging to prevent any nologging operations.

SQL> ALTER DATABASE ARCHIVELOG;

SQL> ALTER DATABASE FORCE LOGGING;

Optionally you may set logging attributes at the tablespace level if it is possible to isolate data that does not require recovery into a separate tablespace. For more information refer to “Reduce Overhead and Redo Volume During ETL Operations” in the technical white paper, Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine [7].

Fast Recovery Area

The Fast Recovery Area (FRA) is Oracle managed disk space that provides a centralized disk location for backup and recovery files.

The FRA is defined by setting two database initialization parameters.

● The DB_RECOVERY_FILE_DEST parameter specifies the location for the Fast Recovery Area. Use the RECO disk group.

● The DB_RECOVERY_FILE_DEST_SIZE parameter specifies (in bytes) a hard limit on the total space to be used by database recovery files created in the recovery area location.

Oracle Flashback Technologies

Oracle Flashback technologies provide fast point-in-time recovery to repair logical database corruptions, most often caused by human error. There is a full suite of Oracle Flashback technologies that enable repair at an optimal level of granularity to achieve the fastest possible repair with the minimum amount of data loss. These technologies include:

Flashback Database

Flashback Database uses flashback logs to ‘rewind’ an entire Oracle Database to a previous point in time. Flashback Database is used for fast point-in-time recovery from logical corruptions, most often caused by human error, that cause widespread damage to a production database. It is


10

also used to quickly reinstate a failed primary database as a new standby database after a Data Guard failover, avoiding the costly and time-consuming effort of restoring from a backup - see section 13.2 of Data Guard Concepts and Administration for more information on fast reinstatement using Flashback Database [8].

● Issue the following command on both primary and standby databases to enable Flashback Database. ALTER DATABASE FLASHBACK ON;

● Oracle Database 11.2.0.2 includes substantial performance enhancements that reduce the impact of flashback logging on the primary database and in particular, its impact on load operations and initial flashback allocations. This enables Flashback Database to be used with OLTP and data warehouse applications with minimal performance impact. Oracle MAA tests have achieved a load rate of 3 TB/hour using Oracle Database Release 11.2.0.2 with Flashback Database enabled. This rate is possible due to the large I/O bandwidth of Exadata Database Machine, 11.2.0.2 optimizations and MAA configuration best practices:

▪ Flashback Database best practices documented in My Oracle Support Note 565535.1.

▪ Use of local extent managed tablespaces.

▪ Re-creation of objects instead of truncating tables prior to direct load.

Additional Oracle Flashback Technologies – For More Granular Repair

More granular levels of repair are enabled by other Oracle Flashback technologies that include: Flashback Query, Flashback Version Query, Flashback Transaction, Flashback Transaction Query, Flashback Table and Flashback Drop.

Flashback Drop requires configuration of a Recycle Bin. All other features use automatic undo management and require that you allocate sufficient disk space to achieve your desired undo retention guarantee – or how far back in the past you want to be able to recover.

● Configure undo retention guarantee

Undo retention period (in seconds) is specified by setting the UNDO_RETENTION initialization parameter to a value that is at least double (2x) the desired detection period to ensure Oracle Flashback operations can operate.

See Section 4.2.7 of Oracle Database High Availability Best Practices [3] for detailed best practices on the full complement of Oracle Flashback Technologies



11

Backup, Restore, and Recovery

Oracle Database backup technologies, Oracle Recovery Manager and Oracle Secure Backup, perform especially well on Exadata Database Machine with its high bandwidth InfiniBand network and Exadata Storage Server Grid. Examples of performance in Oracle Database release 11.2.0.2 or higher include:

● Disk-based backups taking full-image copies can be done at over 17 TB/hour.

● Effective backup rates of up to 50 TB/hour using incremental backups.

● Database restore rates of over 17 TB/hour into normal redundancy disk group and 14 TB/hour into a high redundancy disk group.

● Database redo apply rate of up to 2.1 TB/hour (637MB/sec), or 1.7 TB/hour (496MB/sec) with Flashback Database enabled (the recommended best practice) for load operations.

● Tape-based backups of up to 7 TB/hour for a full backup, and an effective backup rate of up to 50 TB/hour using incremental backup. Higher full backup rates are possible by configuring additional media servers and tape drives.

● These above rates are achieved with a database CPU utilization of less than 5% on the targeted database servers, leaving plenty of CPU bandwidth for concurrent user workloads.

For more information, see the MAA technical whitepaper “Backup and Recovery Best Practices and Performance for Exadata Database Machine” appropriate for your release, either Oracle Database 11.2.0.2 [4] or Oracle Database 11.2.0.1 and prior releases [5].

Oracle Data Guard and Oracle Active Data Guard

Oracle Data Guard is the disaster recovery solution prescribed by the Maximum Availability Architecture (MAA) to protect mission critical databases residing on Exadata Database Machine. Data Guard is also used to maintain availability should any outage unexpectedly impact the production database and to minimize downtime during planned maintenance. Oracle Data Guard is included in the Enterprise Edition license for Oracle Database.

Active Data Guard provides the following extensions to basic Data Guard functionality (Active Data Guard requires an additional license to access its advanced features).

● A physical standby database may be open read-only while it applies updates received from the primary. This enables offload of read-only workload to a synchronized standby database, thereby increasing capacity and improving the performance of the primary database. It is significant to note that read-only queries on an Active Data Guard standby database have the same guarantee of read-consistency as queries executing on the primary database – no physical replication solution supplied by any other DBMS vendor can provide this level of read-consistency.

http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-11202-183503.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-11202-183503.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-final-129256.pdf�


12

● RMAN block-change tracking can be enabled on the physical standby database – enabling fast incremental backups to be offloaded from the primary database.

● Automatic block repair is enabled for the Active Data Guard configuration. Should Oracle detect a corrupt block on either primary or standby, it will automatically repair the corruption using the good copy from the other database – transparent to the application and user.

Data Guard is the disaster recovery solution used by Exadata MAA because:

● It provides data protection superior to storage remote-mirroring [11].

● It is a highly efficient physical replication process that supports all Oracle data types and Oracle Database features, providing a level of simplicity and performance that is superior to logical replication.

● It also addresses planned maintenance, including support for database rolling upgrades when upgrading to new Oracle Patch-Sets (e.g. 11.1.0.7 to 11.2.0.2) or future Oracle Database Releases using Transient Logical Standby [6,7].

● It provides a zero-risk, simple method for installing Oracle patches, including, Oracle Exadata Database Machine bundled patches, Oracle Exadata Storage Server Software patches, Patch Set Updates (PSU), Critical Patch Updates (CPU), or Patch Set Exceptions (PSE) using Standby-First Patch Apply. Standby-First Patch Apply is supported for certified software patches for Oracle Database Enterprise Edition Release 2 (11.2) release 11.2.0.1 and later. Refer to My Oracle Support Note 1265700.1 for more information and the README for each patch to determine if a target patch is certified as being a Standby-First Patch.

● Active Data Guard provides high return on investment on standby systems while they are in standby role by enabling read-only workload to be offloaded from primary systems.

For the complete set of best practices for optimal data protection and availability when configuring Data Guard, see the technical white paper, Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine [6] and the documentation for “Configuring Oracle Database 11g with Oracle Data Guard” in Oracle Database High Availability Best Practices [3].

Comprehensive Data Corruption Protection

Oracle Database includes comprehensive database-aware capabilities to either prevent, or automatically detect and repair data corruption that could otherwise lead to substantial downtime and data loss. Data corruption occurs when a data block is not in a valid Oracle Database format, or whose contents are not internally consistent. Corruption can result from either hardware or software defects.

Listed below are the Exadata MAA recommended initialization settings for optimal data corruption protection. The recommended settings place a greater emphasis on protection


http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/configbp006.htm#i1007026�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/configbp006.htm#i1007026�


13

compared to the default settings that favor minimizing performance impact. Where different from the default value, the recommended initialization settings will need to be configured manually. Note that a Data Guard standby database is necessary for complete protection.

Oracle strongly recommends that you read My Oracle Support Note 1302539.1 for additional background and for important insight into the potential performance tradeoff. This tradeoff requires that you conduct performance testing for your application workload prior to production go-live.

On the Primary Database

● DB_BLOCK_CHECKSUM=FULL (default TYPICAL on Exadata)

● DB_BLOCK_CHECKING=FULL (default OFF on Exadata)

● DB_LOST_WRITE_PROTECT=TYPICAL (default TYPICAL on Exadata)

● Use ASM HIGH REDUNDANCY for optimal HA and corruption repair

● Enable Flashback Technologies for fast point-in-time recovery from logical corruptions most often caused by human error and for fast reinstatement of a primary database following failover.

On the Data Guard Physical Standby Database

● DB_BLOCK_CHECKSUM=FULL

● DB_BLOCK_CHECKING=OFF

● DB_LOST_WRITE_PROTECT=TYPICAL

● Use ASM HIGH REDUNDANCY for optimal HA and corruption repair

● Enable Flashback Technologies for fast point-in-time recovery from logical corruptions most often caused by human error and for fast reinstatement of a primary database following failover.

● Use Active Data Guard to enable Automatic Block Repair (Data Guard 11.2 onward).

Application Clients - Failover Best Practices

High application availability is only achieved when an application reacts and fails over transparently if an unplanned outage or planned maintenance activity impacts the availability of the production database. Automate the detection and failover process by following the best practices described in the technical white paper, Client Failover Best Practices for Data Guard 11g Release 2 [9].



14

In addition there are Exadata specific best practices for Oracle packaged applications. Please refer to the Oracle Technology Network for MAA Best Practices for Exadata Database Machine [10] for additional recommendations for Oracle e-Business Suite, Siebel, PeopleSoft, and other packaged applications.

Operational Best Practices for Exadata MAA

The following operational best practices are required for a successful Exadata Database Machine implementation:

1. Document your high availability and performance service-level agreements (SLAs) and create an outage/solution matrix that maps to your SLAs.

Understanding the impact to the business and the resulting cost of downtime and data loss is fundamental to establishing Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). RTO measures your tolerance for downtime while RPO measures your tolerance for data loss. It is also likely that RTO and RPO will be different for different classes of outages. For example, server and disk failures usually have RTO/RPO of zero. A complete site failure may have larger RTO/RPO as well as less stringent performance SLAs. This is due to managing the trade-off between the potential frequency of an outage occurring and the cost or complexity of implementing HA/DR.

The range of outages that must be planned for in an Exadata MAA environment are described in Appendix 1, “Exadata MAA Outage and Solution Matrix”. It is important to document your procedures for detection and repair for each type of outage. In some cases you will take advantage of Oracle software infrastructure to configure an automatic response. In other cases, Oracle software infrastructure will provide automatic notification but resolution will occur via human intervention. In all cases is it required to validate that each detection and repair procedure is able to meet SLAs. A similar process is followed to implement performance monitoring and rapid response to insure that performance SLAs, such as throughput and response time requirements, are also achieved.

2. Validate HA and Performance SLAs. Perform fault-injection testing to validate the expected HA response and all automatic, automated, or manual repair solutions. Ensure that the application RTO and RPO requirements are met. Ensure that application performance is acceptable under different scenarios of component failures. For example, does the application continue to meet performance SLAs after node failure, Exadata Storage cell failure, and Data Guard role transition?

3. Test and upgrade to software recommended in My Oracle Support Note 888828.1.

Exadata Database Machine will be delivered and deployed with the then current recommended HA software and system components. Once deployed it is necessary to periodically check My Oracle Support Note 888828.1 for new software updates and




15

recommendations. It is best practice to apply the recommended patches and upgrades quarterly or bi-annually. Of course any change is first validated on test systems prior to applying them to a production system. The standby database can also be used for final pre-production testing, and to implement the change in rolling fashion.

4. Follow the Exadata testing and patching practices described in My Oracle Support Note 1262380.1.

Pre-production validation and testing of software patches is one of the most important ways to maintain stability. The high-level steps are:

a. Review the patch and upgrade documentation. Evaluate any rolling upgrade opportunities in order to minimize or eliminate planned downtime. Evaluate whether the patch qualifies for Standby-First Patching, described in My Oracle Support Note 1265700.1.

b. Validate the application in a test environment and ensure the change meets or exceeds your functionality, performance, and availability requirements. Automate the procedure and be sure to also document and test a fallback procedure.

c. If applicable, perform final pre-production validation of all changes on a Data Guard standby database before applying them to production.

d. Apply the change in your production environment.

5. Execute the Exadata MAA health check (exachk), as described in My Oracle Support Note 1070954.1.

Before and after each software patch, or minimally every two months, download the latest release of exachk and run it in your test and production environments to detect any environment and configuration issues. Checks include verifying the software and hardware and warning if any existing or new MAA, Oracle RAC or Exadata hardware configuration best practices need to be implemented.

6. Execute Data Guard role transitions.

Periodically execute Application and Data Guard switchovers to fully validate all role transition procedures. We recommend conducting role transition testing a minimum of once per quarter. In addition to Data Guard documentation, refer to My Oracle Support Notes for switchover best practices for Data Guard Physical Standby (11.2.0.2); either Note 1304939.1 if using SQL*Plus, or Note 1305019.1 if using the Data Guard Broker/Enterprise Manager.

7. Configure Exadata Database Machine monitoring and Automatic Service Request.

Incorporate monitoring best practices for the most comprehensive monitoring of the Exadata Database Machine as described in My Oracle Support Note 1110675.1. See http://www.oracle.com/asr and refer to the Oracle Exadata Database Machine Owner’s







http://www.oracle.com/asr�


16

Guide (delivered with Exadata Database Machine) for additional information on Automatic Service Request.

8. Check the Exadata and MAA repositories for the latest Exadata best practices.

See the following repositories of information:

• Exadata best practices at My Oracle Support Note 757552.1

• MAA OTN Portal [1]

• Oracle 11g Release 2 Oracle Database High Availability Best Practices [3] documentation for general MAA configuration and operational practices, including security best practices, change control procedures and test plans.

Importance of a Test Environment

Investment in sufficient test system infrastructure is essential to Exadata MAA. Unfortunately, test systems are frequently given lowest priority, with scarce capital dollars being reserved for production environments. The table below describes the benefits and trade-offs of various strategies for deploying test systems.

TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS

TEST ENVIRONMENT BENEFITS AND TRADEOFFS

Full Replica of Production Database

Machine

Validate all patches and software changes. Validate all functional tests.

Full performance validation at production scale

Full HA validation especially if the replica includes the standby system.

Standby Database Machine Validate most patches and software changes. Validate all functional tests.

Full performance validation if using Data Guard Snapshot Standby but this can

extend recovery time if a failover is required.

Role transition validation.

Resource management and scheduling is required.

Shared Exadata Database Machine Validate most patches and software changes. Validate all functional tests.

This environment may be suitable for performance testing if enough system

resources can be allocated to mimic production.

Typically, however, a subset of production system resources, compromising

performance testing/validation.

Resource scheduling is required.

Smaller Exadata Database Machine

Common example is production using

Validate all patches and software changes. Validate all functional tests.

No performance testing at production scale.

Limited full-scale high availability evaluations.



17

TABLE 1. TRADEOFFS FOR DIFFERENT TEST AND QA ENVIRONMENTS

TEST ENVIRONMENT BENEFITS AND TRADEOFFS

High Performance (HP) disks and standby

using High Capacity (HC) disks or standby

being a smaller rack Exadata Database

Machine.

Older Exadata Database Machine Validate most patches and software changes. Limited firmware patching test.

Validate all functional tests unless limited by some new hardware feature

Limited production scale performance tests.

Limited full-scale high availability evaluations.

Non-Exadata Database Machine Validate database and grid infrastructure software and patches only.

Validate database generic functional tests.

Limited testing of Exadata specific software features (e.g., EHCC, IORM,

Storage Index, etc.)

Very limited production scale performance tests

Limited high availability evaluations.

Conclusion

The integration of Oracle Maximum Availability Architecture (MAA) operational and configuration best practices with Oracle Exadata Database Machine (Exadata MAA) provides the most comprehensive high availability solution available for the Oracle Database. This technical whitepaper has highlighted the HA capabilities that are delivered pre-configured, and post-delivery configuration and operational best practices used to fully deploy Exadata MAA [11].


18

Appendix 1: Exadata MAA Outage and Solution Matrix

Unplanned Outages

Use the following outage and solution matrix in Table 2 as a baseline when performing high availability testing. The MAA recommended solution is provided for each type of outage along with the expected application recovery time (RTO). Oracle recommends that you perform tests by injecting these faults while running a real-world workload on an Exadata MAA test system. Most outages should incur zero database downtime and a minimal application brownout for any affected connections.

Whether you are deploying manual or automatic failover, evaluate end-to-end application failover time or brownout in addition to understanding the impact that individual components have on database availability. The table includes links to detailed descriptions in Section 4.2, "Recovering from Unscheduled Outages" in Oracle Database High Availability Best Practices [3].

TABLE 2. OUTAGE/SOLUTION MATRIX

OUTAGE SCOPE FAULT INJECTION PROCESS EXADATA MAA

site failure Seconds to 5 minutesFootref 1

Database Failover using Data Guard

Complete Site Failover

Application Failover

clusterwide failure

or production

Exadata Database

Machine failure

Seconds to 5 minutes

Database Failover using Data Guard


Application Failoverr

computer failure

(node)

1. Unplug or power off or reboot a database node

2. Wait 30 seconds or more 3. Restore power and power up database

node, if needed 4. Wait for database node to be fully up

Small application downtime for affected

connections, no database downtimeFootref 3

Managed automatically by Oracle RAC

Recovery for Unscheduled Outages

computer failure

(instance)

Shutdown abort the target instance Small application downtime for affected

connections, No database downtimeFootref 3

Managed automatically by Oracle RAC

Recovery for Unscheduled Outages

Exadata Storage

failure or reboot

Cell Failure: 1. Unplug or power off or reboot cell 2. Wait longer than ASM disk repair timer Cell Reboot: 1. Unplug or power off or reboot cell 2. Wait at least 30 seconds but not longer

than ASM disk repair timer 3. Restore power and power up cell if

Small application brownout for outstanding

IOs until ASM and Exadata detect cell failure

and offline the corresponding Exadata disks,

no database downtime

Exadata and Oracle ASM tolerate storage

failures

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#CACBFBHF�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#CACBFBHF�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/glossary.htm#CHDCHHCJ�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#i1006521�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#BABHCAIA�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#CACFJFEG�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/glossary.htm#CHDHIGBH�




http://st-doc.us.oracle.com/11/111/server.111/b28282/outage.htm#CACFJFEG�

http://st-doc.us.oracle.com/11/111/server.111/b28282/outage.htm#CACFJFEG�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/glossary.htm#CHDJJCII�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/outage.htm#BABIDGFJ�


http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/glossary.htm#CHDJJCII�




19

TABLE 2. OUTAGE/SOLUTION MATRIX

OUTAGE SCOPE FAULT INJECTION PROCESS EXADATA MAA

needed 4. Wait for cell to be fully up

Exadata cellsrv

process dies

1. Kill -11 cellsrv process on cell (simulate SIGSEGV)

Small application brownout, no database

downtime

Cellsrv restarts automatically

Exadata disk

failure or disk pull

1. Pull out spinning disk drive 2. Wait 5 seconds or more 3. Plug the same disk drive back in the

same slot

Small application brownout for outstanding

IOs until ASM and Exadata offlines or

proactively drops the disk, no database

downtime

Exadata and Oracle ASM tolerate storage

failures

InfiniBand spine

switch failure or

port failure

1. Pull IB cable on leaf switch connecting to spine switch (pull done on leaf due to spine switch inaccessibility)

2. Plug IB cable back into leaf switch

No downtime

InfiniBand leaf

switch failure or

port failure

1. Pull IB cable on leaf switch connecting to cell or database node

2. Wait 5 seconds or more 3. Plug IB cable back into leaf switch

Small application brownout, no database

downtime

Cisco switch

failure

1. Remove access to Cisco switch port 2. Wait 5 seconds or more 3. Restore access to Cisco switch port

No application downtime

Client access

network failure

with HA bonded

network

1. Pull Ethernet cable from eth1 interface on database node

2. Wait 10 seconds or more 3. Plug Ethernet cable back into eth1

interface on database node

Small application brownout

human error < 30 minutesFootref 5

Recovering from Human Error

hangs or slow

down

See Oracle Database High Availability Overview documentation for solutions for

unplanned downtime


Footnote 1 Recovery time indicated applies to database and existing connection failover. Network connection changes and other site-specific failover activities may lengthen overall recovery time.

Footnote 2 Recovery time consists largely of the time it takes to restart the failed system.

Footnote 3 Database is still available, but portion of application connected to failed system is temporarily affected.

http://st-doc.us.oracle.com/11/111/server.111/b28282/glossary.htm#CHDJADJJ�


http://st-doc.us.oracle.com/11/111/server.111/b28282/glossary.htm#CHDDBJDA�

http://st-doc.us.oracle.com/11/111/server.111/b28282/glossary.htm#CHDDBJDA�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28281/toc.htm�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28281/toc.htm�



20

Footnote 4 Storage failures are prevented by using ASM with mirroring and its automatic rebalance capability.

Footnote 5 Recovery times from human errors depend primarily on detection time. If it takes seconds to detect a malicious DML or DLL transaction, then it typically only requires seconds to flash back the appropriate transactions, if properly rehearsed. Referential or integrity constraints must be considered.

Planned Maintenance

Table 3 shows the preferred approaches for performing scheduled maintenance on the Exadata Database Machine after testing and patching best practices have been followed. The table includes links to detailed descriptions in Section 5.2, "Eliminating or Reducing Downtime for Scheduled Outages" in Oracle Database High Availability Best Practices [3]. Also see My Oracle Support Note 888828.1 for additional information.

TABLE 3. SOLUTIONS FOR SCHEDULED OUTAGES ON THE PRIMARY SITE

PLANNED MAINTENANCE PREFERRED ORACLE SOLUTION ESTIMATED

DOWNTIME

Site maintenance Site, Hardware, and Software Maintenance Using

Database Switchover



< 5 minutes

Exadata Database Server

Firmware, Operating System

Apply at the standby and evaluate.

Oracle RAC rolling upgrade and service relocation (see

System Maintenance)

No downtime

Hardware maintenance or system software

maintenance (node impact)


POracle RAC rolling upgrade and service relocation

(see System Maintenance)

No downtime

Oracle Grid Infrastructure (GI) Upgrades Apply at the standby and evaluate.

Oracle Grid Infrastructure rolling patch upgrade (see

your platform-specific Oracle Clusterware Installation

Guide for complete details)

No downtime

Oracle patch upgrade for the database,

Exadata Bundle Patch which includes Critical

Patch Updates (CPUs), Patch Set Updates

(PSEs) and database fixes

Apply using Standby-First patching and evaluate.

Oracle RAC rolling patch upgrade using opatch (see

Oracle RAC Database Patches)

No downtime

Oracle diagnostic "one-off" patches Apply Standby First and evaluate.

Online Patching

No downtime

ASM upgrades Apply at the standby and evaluate.

Online ASM upgrade (see the section "Using ASM

Rolling Upgrades" in the Oracle Database Storage

No downtime

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#CACCFHHC�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#CACCFHHC�


http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#i1006524�




http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#BABCEJGA�

http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#BABCEJGA�


http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#BABCAGCJ�

http://st-doc.us.oracle.com/11/111/server.111/b31107/asminst.htm#OSTMG02400�


21

TABLE 3. SOLUTIONS FOR SCHEDULED OUTAGES ON THE PRIMARY SITE

PLANNED MAINTENANCE PREFERRED ORACLE SOLUTION ESTIMATED

DOWNTIME

Administrator's Guide)

Exadata storage upgrade or patch

(firmware, operating system, Exadata

software)


Exadata rolling upgrade

or Switchover

No downtime

< 5 minutes

Switches

(InfiniBand, Ethernet)


Apply and restart switch

Small

application

brownout

Oracle patch set or software upgrade for the

database

Oracle Database rolling upgrade with Data Guard

(transient logical standby)

If Data Guard is not applicable or if less downtime is

required by using active-active replication, consider

Oracle GoldenGate.

< 5 minutes

Power distribution unit (PDU)

Keyboard, video, mouse (KVM)


Apply on primary

No downtime

Database object reorganization or redefinition Online object reorganization with

DBMS_REDEFINITION (see Data

Reorganization and Redefinition)

No downtime

Database platform or location maintenance Database Platform or Location Migration < 5 minutes

Note: Refer to patch README to determine if a patch is RAC Rolling Upgradeable or eligible for Standby-First patching. If not, then use either Data Guard database rolling upgrades using the transient logical standby process [7], or Oracle Goldengate.



http://download.oracle.com/docs/cd/B28359_01/server.111/b28282/schedule_outage.htm#CACFFIDD�


22

Appendix 2: Exadata MAA Performance

Exadata MAA is designed for OLTP and Data Warehouse applications, and for database consolidation. Exadata Database Machine has been engineered for very high performance and is able to support the most demanding applications where traditional architectures typically fall short.

Exadata MAA performance tests have demonstrated substantial performance improvements over non-Exadata configurations. Test results for archiving, standby redo apply, and disk backup/restores, are provided in Table 4. The Exadata configuration included: Oracle RAC (w/forced logging and archiving), ASM, RMAN, Flashback, and Data Guard (ASYNC redo transport).

TABLE 4. KEY PERFORMANCE IMPROVEMENTS FOR EXADATA DATABASE MACHINE

PERFORMNCE

TEST

IMPROVEMENT OVER

NON EXADATA

CONFIGURATIONS

NON EXADATA EXADATA DATABASE MACHINE X2-2

FULL RACK (RELEASE 11.2.0.2)

Archiving 5 X Faster 40 MB/sec per instance 214 MB/sec per instance

5.8 TB/hour for full rack

Redo Apply 6 to 12 X Faster 25 MB/sec OLTP

100 MB/sec Direct Logged

Loads

Up to 300 MB/sec OLTP

600+ MB/sec Direct Logged Loads

Local Disk

Backups and

Restores

5 to 6 X Faster Less than 5 TB/hour

depending on the network

bandwidth, disk, controller, and

ability to scale out

Up to 18 TB/hour for full disk backups

Up to 46 TB/hour for incremental backup

14 TB/hour for full disk restores to high

redundancy disk group

Additional MAA tests evaluated the ability of an Exadata MAA configuration to sustain high load rates. These tests are especially applicable for Data Warehouse applications that require batch cycles to complete within a limited window of time. Experience has shown that users of non-Exadata systems will disable important high availability features such as logging and archiving, or database flashback logging, due to concern for their impact on performance. The high performance of the Exadata Database Machine, however, enables users to run with an Exadata MAA configuration and still meet their performance objectives.


23

The following load rates were achieved with a fully compliant MAA configuration (archivelog mode, database forced logging, Flashback Database, all best practices for database corruption settings, Oracle RAC, Oracle ASM redundancy, and Data Guard with asynchronous redo transport):

● 3 TB/hour load rates with Oracle Database 11g Release 2 (11.2.0.2)

● 1.5 TB/hour load rates with Oracle Database 11g Release 1 (11.2.0.1)

Note that other internal Oracle load tests on Oracle Database 11g Release 2 utilizing Oracle compression technologies achieved up to 5 TB/hour load rates. The Exadata MAA load test reported above did not utilize compression, nor was it designed to be a comprehensive performance benchmark – its purpose is to demonstrate the performance potential of Exadata Database Machine using a simple parallel direct loader test with all HA best practice recommendations implemented.


24

References

1. Oracle Maximum Availability Architecture http://www.otn.oracle.com/goto/maa

2. Oracle Database High Availability Overview http://otn.oracle.com/pls/db112/db112.to_toc?partno=e17157

3. Oracle Database High Availability Best Practices http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28282

4. Backup and Recovery Performance and Best Practices for Exadata Database Machine, Oracle Database 11.2.0.2 http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-final-129256.pdf

5. Backup and Recovery Performance and Best Practices for Exadata Database Machine, Oracle Database 11.2.0.1 and prior releases http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-final-129256.pdf

6. Oracle MAA white paper: “Oracle Data Guard: Disaster Recovery for Exadata Database Machine and Exadata Cell” http://www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf

7. Database Rolling Upgrades Made Easy http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-made-easy-131972.pdf

8. Data Guard Concepts and Administration http://download.oracle.com/docs/cd/E11882_01/server.112/e17022/toc.htm

9. Client Failover Best Practices for Data Guard 11g Release 2 http://www.oracle.com/technetwork/database/features/availability/maa-wp-11gr2-client-failover-173305.pdf

10. Oracle Technology Network: MAA Best Practices for Exadata Database Machine http://www.oracle.com/technetwork/database/features/availability/exadata-maa-best-practices-155385.html

11. Data Guard and Storage Remote Mirroring http://www.oracle.com/technetwork/database/features/availability/dataguardremotemirroring-086151.html

12. Exadata Database Machine Best Practices – OTN portal http://www.oracle.com/technetwork/database/exadata/index-089737.html

http://www.otn.oracle.com/goto/maa�

http://otn.oracle.com/pls/db112/db112.to_toc?partno=e17157�

http://otn.oracle.com/pls/db111/db111.to_toc?partno=b28282�





http://www.oracle.com/technetwork/database/features/availability/maa-wp-dr-dbm-130065.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-made-easy-131972.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-made-easy-131972.pdf�

http://download.oracle.com/docs/cd/E11882_01/server.112/e17022/toc.htm�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11gr2-client-failover-173305.pdf�

http://www.oracle.com/technetwork/database/features/availability/maa-wp-11gr2-client-failover-173305.pdf�

http://www.oracle.com/technetwork/database/features/availability/exadata-maa-best-practices-155385.html�

http://www.oracle.com/technetwork/database/features/availability/exadata-maa-best-practices-155385.html�

http://www.oracle.com/technetwork/database/features/availability/dataguardremotemirroring-086151.html�

http://www.oracle.com/technetwork/database/features/availability/dataguardremotemirroring-086151.html�

http://www.oracle.com/technetwork/database/exadata/index-089737.html�

Deploying Oracle Maximum Availability Architecture with Exadata Database Machine and Oracle Exadata June 2011 Author: Lawrence To Contributors: Joe Meeks, Dan Norris , Doug Utzig, Mike Nowak, Mike Smith, Ron Weiss, , Viv Schupmann Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000

Copyright © 2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

0109

Date post:	30-Jan-2018
Category:	Documents
Upload:	duongnhan
View:	263 times
Download:	0 times

Maximum Availability Architecture - ITtoolboxhosteddocs.ittoolbox.com/db_us_en_wp_maxyexa.pdf ·...

Documents