+ All Categories
Home > Documents > H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS...

H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS...

Date post: 04-Jun-2018
Category:
Upload: dodung
View: 228 times
Download: 0 times
Share this document with a friend
36
EMC Global Solutions Operations EMC Corporation Corporate Headquarters Hopkinton MA 01748-9103 1.508.435.1000 www.EMC.com EMC Solutions for Enterprises Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture
Transcript
Page 1: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

EMC Global Solutions OperationsEMC Corporation

Corporate HeadquartersHopkinton MA 01748-9103

1.508.435.1000www.EMC.com

EMC Solutions for EnterprisesOracle Database 10g Solaris OS

RAC/Veritas SFRAC on DMXConsolidation and Layout

Reference Architecture

Page 2: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Copyright and Trademark Information

2

Copyright © 2007 EMC Corporation. All rights reserved.

Published November 2007

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated.

All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly.

EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.

No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture

P/N H4047.1\

Page 3: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

About this DocumentPurpose .................................................................................................. 5Audience................................................................................................ 5Scope ..................................................................................................... 5

Chapter 1 Solution OverviewThe business challenge .......................................................................... 7The technology solution ........................................................................ 7Solution components ............................................................................. 8

EMC Symmetrix DMX .................................................................. 8EMC ControlCenter ....................................................................... 9EMC Generic SafeWrite ................................................................ 9Oracle Database 10g RAC ........................................................... 10Symantec Veritas SFRAC............................................................ 10Infrastructure validation using Oracle Orion ............................... 11

Hardware and software resources ........................................................ 13Environment profile ............................................................................ 15

Chapter 2 Solution DetailsNetwork and cluster considerations ..................................................... 17DMX storage design and configuration .............................................. 19

Alternative storage volume configuration with metavolumes ..... 23Database design ................................................................................... 23

Contents

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 3

Page 4: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Contents

4

Chapter 3 Testing Overview and ObservationsWorkload characterization ................................................................... 25Workload generation tools ................................................................... 26Environment monitoring applications ................................................. 26Testing methods ................................................................................... 29Observations ........................................................................................ 30

Average number of users ............................................................. 31Average number of transactions per minute ................................ 31Phase 1 testing conclusions.......................................................... 32Oracle Flashback and Generic SafeWrite .................................... 32

Chapter 4 Summary

Contents

Page 5: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

This document describes the reference architecture of an EMC enterprise solution tested and validated by EMC Global Solutions Operations. It was produced as part of the TCE program by the GSO TCE Customer Integration Labs, working collaboratively with Technical Field Consultants, Engineering, and the Oracle Solutions Practice.

Purpose The purpose of this solution is to demonstrate the capabilities of EMC hardware and software in a large Oracle Database 10g RAC environment where an EMC Symmetrix is used for storage and consolidation.

Audience The document is intended for technical staff interested in deploying a new or an existing EMC solution for Oracle database application deployment in a Solaris environment using Oracle RAC and Veritas technologies. Executives evaluating such a solution will also find this document useful.

Scope This document provides an overview of a solution that incorporates emerging requirements and best practices for Solaris/Oracle/RAC/Veritas deployments with EMC technologies. An architectural overview and descriptions of the hardware and software components used in the solution are also included.

Note: This document describes only the features and methodologies specific to this solution. For more detailed information on Symmetrix best practices for Oracle RAC/Veritas, Oracle 10g Database, specific components of this solution, or other EMC solutions, consult the appropriate EMC and Oracle documentation.

About thisDocument

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 5

Page 6: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

6

About this Document

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 7: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

1

The business challengeWith limited resources and increasing demands, today's enterprises must manage IT costs and reduce the risk of business disruption in their Oracle environments. But they must also meet the highest performance and scalability requirements and ensure the highest availability of key applications. To meet these challenges, enterprises need proven solution architectures.

The technology solutionEMC empowers enterprises to address these pressing challenges by offering customers cost-effective, easy-to-use solutions that are backed by rigorous validation and testing in real-world conditions and workloads.

To help enterprises reduce cost and mitigate risk in their Oracle environments while meeting performance and scalability requirements, EMC has developed, fully validated, and documented proven solution architectures that incorporate best practices for deploying Solaris/Oracle/RAC/Veritas.

SolutionOverview

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 7

Page 8: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

8

Solution componentsThe Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout solution is based on an OLTP/TPC-C-like workload simulation of an enterprise with a 1 TB database. The solution includes the following key components:

• EMC® Symmetrix® DMX-3 2500

• EMC ControlCenter®

• Oracle Database 10g Enterprise Edition

• Oracle Database 10g RAC

• Sun servers/Solaris software

• Symantec Veritas Storage Foundation for RAC

For details on all of the components that make up the solution, see “Hardware and software resources” on page 13.

EMC Symmetrix DMXThe Symmetrix DMXTM systems are EMC's family of high-end storage solutions. The DMX-3 model establishes a new performance and capacity trajectory for the highest of the high-end enterprise systems. The DMX-3 fully leverages the EMC industry-leading storage management functionality and introduces the economic benefits of scalable packaging to the high-end storage market.

The Symmetrix® DMX-3 is incrementally scalable, supporting from 96 to 1,920 2 Gb/s high-performance Fibre Channel disk drives, providing a maximum raw capacity of approximately 1 PB. To support the massive scalability of DMX-3 configurations, the DMX architecture has been expanded and improved to deliver higher throughput (1 GB/s links) and increased I/O performance (four dual 1.3 GHz PPC processor complexes per director).

Solution components

Page 9: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

EMC ControlCenterEMC ControlCenter (ECC) is designed to be the centralized control center for an entire distributed storage environment. It is a powerful, flexible, unified framework and suite of tools that provides end-to-end management of storage networks, storage devices, and other storage resources. ECC consists of:

• SRM Device Management, which configures and optimizes EMC storage devices, using Symmetrix Manager and Symmetrix Optimizer.

• SRM Monitoring and Reporting, which monitors and reports on multivendor, tiered storage, using ControlCenter Performance Manager, ControlCenter StorageScope, and ControlCenter StorageScope File Level Reporter.

• SRM Planning and Provisioning, which designs, plans, and provisions multivendor, tiered storage, using ControlCenter Automated Resource Manager, ControlCenter SAN Advisor, and ControlCenter SAN Manager.

EMC Generic SafeWriteEMC SafeWrite is a feature, within the Solutions Enabler product, that helps protect critical applications from incurring an incomplete write (resulting in torn pages) due to a failure with a component connected to the DMX front end channel adapter.

Generic SafeWrite is most often used to protect against corruptions due to HBA and link failures including server crashes, where it will help protect against fractured writes that can occur before the data reaches the DMX.

Note: A Relational Database Management System (RDBMS), such as Oracle, structures data within database files using pages (also referred to as blocks). Pages within a database are the smallest allocation unit size possible for a database object (such as a table or a row). For example, the page size for Oracle, is 8 KB. Although this size is configurable, it is usually set to 8 KB. If an incomplete page is written to a database file, a corruption to the database will occur. The resulting corruption is commonly referred to as a torn page.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 9

Page 10: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

10

Oracle Database 10g RACOracle Real Application Clusters (RAC) is an optional feature of Oracle Database 10g Enterprise Edition. Oracle RAC supports the transparent deployment of a single database across a cluster of servers, providing fault tolerance from hardware failures or planned outages. Oracle RAC running on clusters provides Oracle's highest level of capability in terms of availability, scalability, and low-cost computing. Oracle RAC supports mainstream business applications of all kinds. This includes OLTP and DSS.

Oracle Flashback The Oracle Database architecture leverages the unique technological advances in the area of database recovery. Oracle Flashback technology provides a set of new features to view and rewind data back and forth in time. The Flashback features offer the capability to query historical data, perform change analysis, and perform self-service repair to recover from logical corruptions while the database is online.

Oracle Database 10g extends the Flashback technology to provide fast and easy recovery at the database, table, row, and transaction level. Flashback technology revolutionizes recovery by operating just on the changed data. The time it takes to recover the error is now equal to the same amount of time it took to make the mistake.

Symantec Veritas SFRACSymantec Veritas Storage Foundation for Oracle RAC (SFRAC) includes storage-management and high availability technologies allowing you to implement robust, manageable, and scalable Oracle Real Application Clusters.

SFRAC provides a complete I/O and communications stack to support Oracle RAC. This software also monitors and manages the startup and shutdown of instances.

Solution components

Page 11: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

Infrastructure validation using Oracle OrionThe Orion Workload tool is used for predicting the performance of an Oracle database without having to install or create a database. Unlike other calibration tools, Orion is designed for simulating Oracle database I/O workloads using the same I/O software stack as Oracle.

Oracle administrators can use Orion to help design the SAN infrastructure. By generating a typical OLTP-type load it is possible to identify:

• Potential bottlenecks

• I/O profiling

• SAN port utilization

• Throughput and load balancing on the HBAs

Orion also provides early warnings on any hardware faults, either server related or otherwise, prior to installing any applications or databases.

Performance andscalability

Orion provides operational benefits in the areas of performance, scalability, and reliability:

• When used to simulate a real-world Oracle workload, Orion makes it possible to plan future infrastructure configurations based on performance results.

• Orion provides a picture of how I/O patterns scale when workloads change. Orion relates growth in workload to performance, giving a clear picture of the scalability of the system.

When scalability and performance targets are met, higher levels of reliability are experienced in the storage array. By performing Orion benchmarks, it can be verified that the configuration is able to meet or exceed the I/O requirements of an application mix.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 11

Page 12: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

12

Figure 1 illustrates the overall physical architecture of the solution.

Figure 1 Overall physical architecture

Solution components

Page 13: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

Hardware and software resourcesThis section describes the hardware and software resources.

Table 1 lists the hardware resources used in the solution.

Table 1 Hardware resourcesEquipment Quantity Configuration

Storage 1 EMC Symmetrix DMX-3 2500:• 64 GB MEM• 16 DAEs• 146 GB, 15k

SAN 2 Brocade 4920, 64 ports

Host 4 Sun Fire V490Four 1.86 GHz CPUs 32 GB RAMTwo 73 GB 10k internal SCSI disksSix GigE NICs4 Gb Emulex LP11002-E HBA

Network switch 2 Cisco Catalyst 3750G-48TS

Load generation server 4 Dell 6850Four Dual-Core Intel Xeon 3.4 GHz CPUs16 GB RAMTwo 146 GB 10k internal RAID disksTwo dual-port NICs4 Gb Emulex LPe11002-E HBA

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 13

Page 14: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

14

Table 2 lists the software resources used in the solution.

Table 2 Software resourcesTitle Version Comment

Solaris 10 OS for database server

iostat, vmstat, mpstat, iorate Solaris 10 O/S tools

SYMCLI 6.4.05

Microsoft Windows 2003 SP1 R2 32-bit

EMC ControlCenter 6.0 • SAN configuration• DMX masking• Infrastructure alert management• Symmetrix Management Console• Performance Manager

Grid Control (Oracle) 10g Oracle tool

PowerPath® 5.0.2 HBA load balancing and redundancy

STP - statistics collection daemon storstpd daemon, run in conjunction with EMC Solutions Enabler 6.4

Hardware and software resources

Page 15: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

Environment profileTable 3 lists the parameters and assumptions for the solution.

Table 3 Environment profileParameter Value

Database type High-volume OLTP

Database size 1 TB

Number of concurrent users ~15,000

Oracle RAC nodes 4

Storage array EMC DMX-3 2500

Drive size and speed 146 GB 15k

Veritas disk groups Four: DATABASE, REDO, TEMP, FRA

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 15

Page 16: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Overview

16

Environment profile
Page 17: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

2

The following sections provide additional details about the solution.

Network and cluster considerationsFor this Use Case two Cisco WS-C3750G-48TS (IOS v.12.2) switches were used in a GigaStack configuration (32 Gbps, high-speed stacking bus). The GigaStack technology allows physical switches to be joined together to form one logical switch configuration.

VLAN (Virtual Local Area Network)Three primary VLAN were configured on the switch to facilitate the following communications:

1. VLAN A. Production network, used for client connections to the RAC cluster.

2. VLAN B. Private network for Low Latency Transport (LLT) communications, which supports the Veritas cluster interconnect communication channel.

3. VLAN C. Private network for Low Latency Transport (LLT) communications, which supports the Veritas cluster interconnect communication channel.

Solution Details

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 17

Page 18: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

18

Low Latency Transport protocolLLT provides fast, kernel-to-kernel communications and monitors network connections. LLT functions as a high performance replacement for the IP stack and runs directly on top of the Data Link Protocol Interface (DLPI) layer.

All production and cluster network connections are dual homed or distributed across the physical switches to provide a layer of physical redundancy in the event of a switch failure.

Oracle Cluster Ready Services (CRS)Oracle also requires the use of a private network, or networks, for the CRS daemon to enable communication between the nodes and for heartbeating (verifying the health of the nodes).

In this particular Use Case one of the existing network interfaces that is assigned to LLT was used for CRS. This involved configuring the interface with an IP address/netmask.

Although there is a second network interface configured for LLT, it can not be used as a second/redundant link for CRS as it is on a separate VLAN. If additional redundancy is desired for CRS, an additional network interface should be provided which is connected to the same VLAN.

Network Time Protocol (NTP)Another important aspect of the network is ensuring that each node in the cluster is set to the same time. The best way to achieve this is by using NTP to synchronize the clocks on each node.

EMC PowerPathAfter careful consideration, EMC PowerPath was chosen over Veritas DMP or Dynamic Multipathing for this Use Case. PowerPath works with the storage system to intelligently manage I/O paths. PowerPath supports multiple paths to a logical device. PowerPath manages four I/O paths in this solution and provides:

• Automatic failover in the event of a hardware failure. PowerPath automatically detects path failure and redirects I/O to another path.

• Dynamic multipath load balancing. PowerPath distributes I/O requests to a logical device across all available paths, thus improving I/O performance and reducing management time and downtime by eliminating the need to configure paths statically across logical devices.

Network and cluster considerations

Page 19: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

DMX storage design and configurationSymmetrix DMX-3 2500 provides an enterprise-level platform for large-scale mission-critical applications by allowing tiered storage, RAID protections, enhanced performance, and many business continuity features. Although up to 240 number of drives can be supported in a DMX-3 2500, only 40 disks were used to reach the required baseline criteria of the Use Case, using the disk layout described below.

For the 1 TB database, using Veritas, the optimal recommended Symmetrix back-end configuration used hypervolumes. For this reference architecture, each physical spindle is divided into six hypervolumes on RAID 1 disks, described in Table 4. TimeFinder clones were created on the 300 GBRAID 5 disks.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 19

Page 20: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

20

Figure 2 Oracle database layout for RAID 1 disk groups

DMX storage design and configuration

Page 21: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

The DMX-2500 used in this environment has one drive bay configured that can house a maximum of 240 drives. This is subdivided into two areas where the disks are controlled by the DFs (Fibre Channel back-end disk director) in the system bay. Figure 3 shows two physical devices that are RAID-1 protected and are located on DF 2bC0 and also DF 1bC1. The hypers listed on one DF are the same as the other DF.

Figure 3 Disk mirroring layout for RAID 1 disk groups

The layout of these devices is taken from the SMC (Symmetrix Management Console) which shows the protection as 2-Way Mir.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 21

Page 22: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

22

Oracle database performance and hypervolumesRedo logs and datafiles are the most active database files and demand the best response times. Writes to the redo logs and reads from datafiles and indexes are most response time-sensitive. Writes to the datafiles are asynchronous and are, therefore, not as sensitive. The Symmetrix back-end configuration addresses these requirements. The disks are divided into six hypervolumes, as shown in Table 4.

The most active database files (the datafiles and the redo log files) are placed close to each other. This approach minimizes the disk head movement and, therefore, reduces seek time.

The hypervolume that holds the redo logs on each disk is 400 MB in size. The redo log Veritas disk group is built using these Symmetrix volumes. This ensures that redo log I/O is spread across all physical drives used by the database and that the redo log volumes have a very low probability of reaching the Symmetrix write-pending limits. This approach is different from other approaches that recommend that physical disks should be dedicated to redo log files.

Disk capacities are rapidly increasing; it is now typical to have disks that are 146 GB or larger. It is not practical to dedicate entire disks to redo logs since redo log files are typically small (usually 1 GB or less). The redo log placement used in this solution ensures an optimal use of disk space while satisfying the performance requirements of the redo log files.

Table 4 Hypervolume designations

Hypervolume number Purpose Size

1 Data 32 GB

2 Redo 400 MB

3 Data 32 GB

4 FRA 24 GB

5 Undo/Tempa 10 GB

6 FRA 24 GB

a. Ten disk pairs contain Undo files; the other ten pairs contain Temp files

DMX storage design and configuration

Page 23: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

Alternative storage volume configuration with metavolumesThere is an upper limit on the number of LUNs that a host can address. This limit depends on operating system and HBA driver constraints. Typically, the number of LUNs that a host can address ranges from 256 to 1024 per HBA. When these limits are reached, Symmetrix metavolumes are a convenient way to access more Symmetrix hypervolumes than can be addressed individually. Symmetrix metavolumes allow multiple Symmetrix hypervolumes to be presented to hosts as one logical volume.

In Oracle OLTP environments, where the I/O size is typically a single Oracle database block (typically 8 KB) and where the access pattern is random, both Symmetrix metavolumes and hypervolumes are viable options. For this solution, hypervolumes are used instead of metavolumes because there is no LUN limitation.

Database designA single database was clustered across four nodes, such that each cluster node resided on its own server. This is typical Oracle RAC configuration.

Table 5 Database tablespaceDatabase objects Object names

Tablespaces TPCCINDX TPCCDATA SYSAUX UNDOTBS1 UNDOTBS4 USERS SYSTEM UNDOTBS3 UNDOTBS2

Redo logs Four 750 MB redo logs per instance

Veritas disk groups DATADG UNDODG/TEMPDG REDODG FRADG

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 23

Page 24: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Solution Details

24

Database design
Page 25: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

3

Workload characterizationThe reference architecture workload represents transaction processing in an order entry environment. The transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at inventory warehouses. While the workload portrays the activities of a wholesale supplier, the workload is not limited to the activity of any particular business segment, but rather represents any industry that must manage, sell, or distribute a product or service.

The OLTP transaction types are as follows:

• New order: Enter a new order from a customer

• Payment: Update a customer balance to reflect a payment

• Delivery: Deliver an order (processed as a batch transaction)

• Order status: Retrieve the status of a customer's most recent order

• Stock level: Monitor warehouse inventory

Transactions operate against a database of nine tables. Operations at the database level include update, insert, delete, abort, and primary and secondary key access.

Testing Overviewand Observations

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 25

Page 26: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

26

Testing Overview and Observations

Workload generation toolsThe following tools were used to generate the workload used to validate the solution:

• An Oracle TPC-C-like toolkit was used to simulate an OLTP environment and drive the transaction workload, through client connections. See “Workload characterization” on page 25.

• Orion. The Oracle I/O Numbers Calibration Tool is designed to simulate Oracle I/O workloads.

Environment monitoring applicationsThe following applications were used to monitor and validate the solution:

• ECC 6.0 Performance Manager. An EMC toolkit that presents a visual display of detailed Symmetrix performance data.

• EMC ControlCenter Workload Analyzer: an EMC toolkit that presents a visual display of detailed Symmetrix performance data.

• AWR (Oracle Automatic Workload Repository). A built-in repository that exists in every Oracle database. At regular intervals, the Oracle database takes a snapshot of all of its vital statistics and workload information and stores them in the AWR.

• Solaris kernel utilities, including vmstat, iostat, iorate, sar -d, and mpstat.

• Symmetrix Management Console. SMC is a simple, intuitive, browser-based user interface for the configuration and management of Symmetrix arrays.

• Oracle Enterprise Manager 10g Grid Control 10.2.0.3. A critical element of the Oracle Grid platform, manages service level performance from a business perspective.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 27: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Environment monitoring applications

• EMC ControlCenter - The EMC ControlCenter suite of storage resource management (SRM) and device management software simplifies the storage provisioning in the grid environment. It is a client-server application that consists of the following components. (Figure 4 depicts an overview of the EMC ControlCenter architecture.)

• Infrastructure (comprising Server, Repository, and Stores)• ControlCenter Agents• Client Console

Figure 4 EMC ControlCenter architecture

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 27

Page 28: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

28

Testing Overview and Observations

Figure 5 is from the EMC Control Center 6.0 infrastructure, which is used to both monitor and configure the environment. This shows the relationship view from the DMX device 40 back to one of the nodes. It shows the Oracle database instance, EMC PowerPath pseudo name, and displays the redundancy (two paths) to the DMX-3 array.

The infrastructure is constantly monitoring the Oracle 10g database through the EMC Oracle agent that is installed on each Solaris server.

Figure 5 ECC relationship view of DMX device

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 29: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Testing methods

Testing methodsTesting was conducted using a Solaris 10 O/S and Oracle Database 10g Enterprise Edition R2 (64-bit). The testing team identified and optimized the baseline operating system and Oracle database settings. Specific tuning parameters for Solaris, Veritas SFRAC, and Oracle were fine tuned to support performance and scalability testing.

The size of the database and the active user count necessitated an Oracle System Global Area (SGA) memory size of 14 GB. Note that all tests were run a minimum of two times to guarantee consistency and validation. The following list outlines the tests run:

• Tests measured performance characteristics using a baseline of 40 mirrored disks (80 disks total).

• Oracle Flashback recovery was enabled and, from a performance perspective, the impact that it had on the baseline results was noted.

• Then the same test was repeated with Flashback off and EMC SafeWrite on.

• The final test ran the same load against 48 disks, as shown in Figure 6.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 29

Page 30: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

30

Testing Overview and Observations

ObservationsThis Use Case was focused on building and validating an Oracle 10g RAC infrastructure on an EMC DMX-3 2500 using Veritas SFRAC 5 (MP1) on Solaris 10. The database was 1 TB in size and it involved generating an OLTP-type load with a user count of upwards of 15,000. The build used the most current releases of EMC Solutions Enabler, EMC Control Center, EMC PowerPath, Oracle, Veritas, and Solaris.

For the initial baseline, a highly random read/write OLTP load was generated against the database from client machines. Disk utilization was approximately65 percent with an average IOPS of 128. A user count of 16,900 users was reached.

Note: One user equals 1.2 tpmC. Therefore, 16,900 users equate to a TPC-C of 202079.56.

For the second test, EMC Generic SafeWrite was implemented against the baseline configuration and a slight drop in user count of less than 2 percent was noted, while still maintaining similar disk utilization and IOPS as in the first test.

For the third test, Oracle Flashback Recovery was enabled separately against the baseline configuration and a drop in user count by just under 8 percent was observed, while still maintaining similar disk utilization and IOPS as in the first two tests.

For the final test, the procedure of adding disks to the baseline configuration was documented, and again the results were noted against the previous load generated, as in the first three tests. The number of disks added was eight. After running tests the following was observed: an increase in user count by 8 percent, a drop in disk utilization, with a decrease in IOPS to 117.

EMC reference architectures are the basis for building and validating integrated EMC/Oracle solutions. This allows EMC to understand the effects of adding additional components to these type of environments and their relationship to a performance baseline.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 31: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Observations

Average number of usersTable 6 shows the average number of users supported, assuming an average of 1.2 transactions per minute per user, for the 40 and 48 mirrored-pair disk configurations. All configurations used RAID 1.

Average number of transactions per minuteFigure 6 shows a graph of the average number of transactions per minute supported, and number of users, corresponding to the data in Table 6.

Figure 6 Average number of transactions per minute

Table 6 Average number of users

Disks (mirrored)

Devices visible to Veritas (six

hypervolumes on each mirrored pair)

Average user count (transactions per

minute / 1.2)

40 (mirrored) 240 16,900

48 (mirrored) 286 18,252

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 31

Page 32: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

32

Testing Overview and Observations

Phase 1 testing conclusionsIn general, performance improves with more disks and degrades with fewer disks. Disk for disk, the rate of degradation exceeds the rate of improvement. With 20 percent more disks than the original number of disks (80 plus 16 disks (or 40 plus 8 mirrored pairs) with 240 plus 48 hypervolumes) there is an 8 percent increase in performance (more users supported, more transactions supported).

Oracle Flashback and Generic SafeWriteOracle Flashback can be an extremely useful option available to users of an Oracle database. As mentioned earlier in the Reference Architecture, Flashback can assist in recovering from logical database corruptions and provide historical reporting, along with other capabilities; however from this graph it is clear this is not without some cost to performance.

Flashback will impact performance on average, about 8 percent, though at some points during the testing the impact was well over 15 percent. The impact will be based upon the workload so it is important to remember that these numbers demonstrate an example of the performance cost and not a definitive conclusion in all environments. Proper testing of this feature in a particular customer’s environment is one essential piece of data to obtain before making a decision as to whether to enable Flashback in the database.

Figure 7 Flashback and SafeWrite results

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 33: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Observations

Unlike Oracle Flashback, Generic SafeWrite is not an option within the Oracle database; rather it is an EMC capability that safeguards application against incomplete writes. Generic SafeWrite is most importantly generic, that is, it is not bound by OS or application. It is implemented through the EMC software Solutions Enabler, and is therefore available for more than just Oracle databases. This is very important since most enterprise customers run more than just Oracle on their Symmetrix infrastructure.

The results of the Generic SafeWrite test are very encouraging. In fact, as the average impact is less than 2 percent, it is reasonable to declare that the performance impact of Generic SafeWrite is negligible, or even non-existent. The results should encourage customers to strongly consider implementing this feature to protect their critical applications from incomplete writes, however unlikely their occurrence.

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 33

Page 34: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

34

Testing Overview and Observations

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture

Page 35: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

4

EMC can help accelerate assessment, design, implementation, and management while lowering the implementation risks and cost of a consolidation solution for an Oracle Database 10g RAC environment. This reference architecture depicts a validated design using an EMCDMX-3 2500 storage system for consolidation.

Faced with complex infrastructure and escalating costs, many companies are looking to consolidate databases. To achieve even greater flexibility, many companies are implementing database grid environments. This strategy, while effective, is only one piece of the equation. To successfully deploy an Enterprise Grid environment, a thorough understanding of it components is essential - including the impact on the underlying storage infrastructure. A successful Grid implementation starts with a robust foundation that is built for growth and exceptional performance.

EMC has invested in the discovery and identification of how well various storage and server combinations scale and perform based on typical Oracle OLTP workloads. The EMC Consolidation for Oracle 10g solution provides customers with a recommended architecture and best practices for implementing Oracle 10g databases on DMX storage to achieve optimum performance and scalability. This solution is supported by validation testing of a typical, large enterprise Oracle 10g Real Application Cluster (RAC) environment on a Symmetrix DMX platform.

This solution illustrates the performance of various storage configurations.

Additionally, EMC has developed best practices to address Oracle, Veritas, and EMC technologies.

Summary

Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Reference Architecture 35

Page 36: H4047.1-Oracle Database 10g Solaris OS RAC/Veritas … · Oracle Database 10g Solaris OS RAC/Veritas SFRAC on DMX Consolidation and Layout Reference Architecture. ... 11 Hardware

Summary

36

Since 1995, EMC and Oracle have invested to jointly engineer and integrate their technologies. The combination of Oracle and EMC software and best practices used in an integrated fashion can greatly reduce the cost of designing, implementing and operating your IT infrastructure.

To learn more about this and other solutions contact an EMC representative or visit www.EMC.com/solutions/oracle.


Recommended