Page | 1 A Dell Technical White Paper Dell U2L Platform Migration Services By Mahesh Pakala Global Infrastructure Consulting Services May 2012 Customer Success Story: Migrating a Traditional Physical Server Configuration to a Dell SAP High Availability (HA) Tiered Private Cloud Architecture with Minimal Downtime SAP, Redwood CPS, Enqueue Splitting, Oracle RAC 11gR2, Oracle ASM, Workload Management, Compression, VMware HA/ FT & Cross Platform Migration May 2012
Transcript
Page | 1
a
A Dell Technical White Paper
Dell U2L Platform Migration
Services
By Mahesh Pakala
Global Infrastructure Consulting Services
May 2012
Customer Success Story: Migrating a Traditional Physical Server
Configuration to a Dell SAP High Availability (HA) Tiered Private Cloud Architecture with Minimal Downtime
3.1.1 Migration approach ............................................................................................................... 12 3.1.2 Migration Steps on the Source System .............................................................................. 14 3.1.3 Migration Steps on the Target System ............................................................................... 15
1.8. SAP EXPORT/ INSTALLATION PROCESS ......................................................................................... 40 5.1.1 Target System CI (*Interim CI before ASCS split) Installation steps: ............................ 41 5.1.2 Target System DB Installation steps: ................................................................................. 45 5.1.3 ABAP SAP Central Services(ASCS) split steps: .............................................................. 52 5.1.4 Target System Dialog Instance Installation steps: ........................................................... 55
1.9. CREATE RFC DESTINATIONS USING LOGON GROUPS ................................................................... 59 1.10. SET UP TOPCALL FAX SERVER RFC DESTINATION .................................................................... 64 1.11. SET UP TMS IN HA ENVIRONMENTS ........................................................................................... 64
1 Introduction This paper describes the innovative solution, forward-thinking technologies and best practices used
by Dell Global Infrastructure Consulting Services (GICS) to successfully transform a highly transactional and complex SAP implementation in the Oil & Gas industry to a Cloud architecture.
The solution was designed utilizing a tiered architecture running VMware vSphere and Oracle
11gR2 Real Application Cluster (RAC) with Automatic Storage Management (ASM). Using VMware Fault Tolerance (FT) and High Availability (HA) features we delivered iron clad protection to our
customer‟s critical SAP application services providing zero downtime. The solution was delivered to a fortune 100 company supporting a 10TB SAP database running SAP‟s R/3 mission critical
applications.
A consistent challenge faced by IT organization today is the need for cost containment while
dealing with increasing demands for computing capacity and performance. The rapidly increasing capability of Intel x86_64 based servers allows a new model for the creation of agile, low-cost
infrastructures based on the Linux operating system. This model‟s use of lower-cost x86_64 based commodity servers running Linux, combined with Oracle‟s load balancing & fault tolerant Oracle
RAC database provides a resilient, scalable, platform for enterprise solutions. SAP Application
software, can make use of this platform to deploy large mission critical systems that are fault tolerant, scalable and provide high performance.
Dell‟s, SAP‟s and Oracle‟s close engineering partnerships with major Linux distributors, such as Red
Hat, results in enhancements to the Linux solution stack. They provide a joint support model that allows SAP & Oracle RAC on Linux to become an ideal choice of platform.
SAP was the first software vendor to recognize the applicability of Linux to support mission-critical enterprise applications as early as 1996. Today, Linux is one of two reference platforms for SAP
development. SAP business solutions running on Linux conform to the same high standards as they do on every other operating system on which SAP enterprise systems run. SAP solutions on Linux
go through stringent quality assurance and certification processes performed by SAP and its
partner Dell to make SAP business solutions a real option.
Some highlights of Oracle‟s and Dell‟s commitment and success on Linux include:
Release of the first commercial Database for Linux in 1998
Availability of all of Oracle software products on Dell/Linux.
Contributions to the Linux kernel in the form of stability and scalability enhancements
including the Clustered File System.
Integrated and code-level support for all Oracle software and the Linux OS
Widespread internal use of Linux to run mission-critical systems, such as Oracle
Outsourcing, Oracle/Dell E-mail systems, Oracle/Dell Web sites and Oracle/Dell Data centers.
Freedom and flexibility - Running SAP enterprise applications on Linux puts you in control of your IT infrastructure because you‟re not constrained by technology, you can make decisions based on
your business objectives. Linux decouples the operating system from the underlying hardware, thereby increasing your choice of infrastructure vendors.
Savings in IT operating costs – Dell Customers report that SAP applications on Linux deliver
significant direct and indirect savings in their IT operations with lower administration and operating costs compared to non-open-source systems. The platform is extremely reliable so less system
Page | 4
redundancy and maintenance are required than for most traditional installations. Customers enjoy
savings on expensive license fees and are able to use commodity server hardware, which performs well under Linux but at a fraction of the cost.
The power to innovate - By streamlining your IT operations, you can reallocate resources, accelerate business process innovation, address new business requirements, or simply improve
your bottom line.
The migration of database applications from legacy platforms to Linux on x86_64 platforms is a
process many customers are undertaking in order to realize its cost and performance benefits. Transferring from a legacy UNIX system to another platform will have migration costs but with
proper planning, education, tools and services the time & costs can be significantly reduced.
The core strategy was to implement a PRIVATE CLOUD/GRID as an integrated structure of
components providing application services and integrating a variety of resources such as applications, databases, servers, storage systems and networks. The prime objective was to
implement the integration in such a way that resources can be managed as a single system image and provisioned on demand based on the service level requirements of each application.
All resources are physically connected in the form of large pool of servers and storage network. Logically, the machines function as a pool of application servers running VM based App servers and
database servers clustered with Oracle Database 11gR2 RAC. The storage network provides access to shared file system storage for the application and database server software and data storage.
The storage used by different databases managed by a single ASM cluster, which maintains disks and recovery groups for each database.
The implementation of a PRIVATE CLOUD/GRID provides the following advantages –
Increases the total computing power available to applications
Exploits low-cost, high performance technologies as building blocks in flexible infrastructure
Benefits from easy deployment and integration of different types of storage systems
Can grow the total computing power and storage infrastructure in small increments
Provides on-demand adjustments of computing and storage resources based on the short-term or long-term needs of application
Allows easy and flexible migration and movement between clusters, databases or storage classes
Covers availability objectives for planned and unplanned outages by rebalancing workloads
Can be managed using a single set of controls
This document highlights the steps involved in the process that was adopted at a large O&G
Company to address the RAMP (Reliability, Availability, Manageability & Performance) requirements for the SAP implementation on x86_64 platforms.
2 Architecture & Design Considerations SAP provides a proven, scalable, fault-tolerant, multi-tier architecture technology. The individual
components can be protected either by horizontal scalability – that is, the use of multiple
Page | 5
components that tolerate the failure of individual components – or by cluster and switchover
solutions. All SAP partners provide their own proven solutions, which enable SAP applications using additional software and hardware to achieve high availability.
The layers below the business applications are generally transparent to these applications.
However, in the event of errors, they can affect the availability of the business applications and
you must therefore protect them. Partners offer a number of proven solutions for this purpose. The most important mechanisms are described briefly below to address the requirements to
migrate the customer legacy systems to Dell Hardware.
Source system (HP):
Processor: PA-RISC 8900
DBMS: Oracle 9.2.0.7 with HP Service Guard
File system: Veritas vxfs
SAP kernel Release : 6.40
Application server platform: HP-UX
Target system (Dell):
Processor: Intel X7560
DBMS: Oracle 11.2.0.2 with Oracle RAC (Real Application Clusters)
File system: Oracle Cloud File System (ASM/ACFS)
SAP Kernel release 6.40 EX2 /VMware
Application server platform: RedHat Linux in VMWare ESXi 4.1
Application Server Platform : RHEL AS5U6
VMware configuration (Dell):
Specification : 2 VMs with 8vCPUs/48GB, 1 VM with 1vCPU/4GB
SAP Application Servers : Dell00748 (ABAP SAP Central Services), dell00746 (Dialog instance), dell00747 (Dialog instance)
CPU Reservation : No reservation
Memory Reservation : 48GB each
The high availability architecture for the SAP solutions for this particular customer was architected
to run on vSphere and Oracle 11gR2 RAC with ASM. The new VMware Fault Tolerance (FT) feature provides further protection to critical SAP application services with zero downtime, enhancing the
high availability provided by the VMware® High Availability (HA) feature introduced in earlier
releases.
Page | 6
SAP Architecture takes advantage of Fault Tolerance because the SAP Central Services are mission
critical, requiring maximum uptime and protection against single points of failure.
The architecture addresses four SAP single points of failure (SPOF) to provide high availability solutions with different degrees of fault tolerance and uptime. The following single points of failure
are identified in the SAP architecture and addressed:
1.1. Oracle RAC Database:
Oracle Real Application Clusters allows the Oracle databases to run SAP or custom application unchanged across a set of clustered nodes. This capability provides the high availability for node
failures, instance failures, and most planned maintenance activities while providing flexible
scalability. If a clustered node fails, the Oracle database continues running on the surviving nodes. When more processing power is needed, another node can be added without interrupting user
access to data.
Oracle Clusterware is a cluster manager that is designed specifically for Oracle databases. In an
Oracle RAC environment, Oracle Clusterware monitors all Oracle resources (such as database instances and listeners). If a failure occurs, then Oracle Clusterware automatically attempts to
restart the failed resource. During outages, Oracle Clusterware relocates the processing performed
by the inoperative resource to a backup resource. For example, if a node fails, then Oracle Clusterware relocates the database services being used by the application to a surviving node in
the cluster.
Oracle Real Application Clusters includes a highly available (HA) application framework that
provides the necessary service and integration points between RAC and SAP applications. One of the main principles of a highly available SAP application is for it to be able to receive fast
notification when something happens to critical system components (both inside and outside the cluster). This allows the SAP work processes to execute event-handling programs. The timely
execution of such programs minimizes the impact of cluster component failures, by avoiding costly connection time-outs, application timeouts, and reacting to cluster resource reorganizations, in
both planned and unplanned scenarios. The features of Oracle Real Application Clusters 11g are
Page | 7
used by SAP applications to provide the best possible service levels by maximizing throughput,
response time and minimizing the impact of cluster component failures
Each ABAP work process initiates a private connection to the database via a service and if the connection is interrupted due to an instance failure the work processes failover to a surviving RAC
node.
This will set up a new connection and changes to "database connect" state to the surviving database instance. User sessions with SELECTS will continue but users in the middle of DML
database activity will receive SQL error messages but their logged on sessions are preserved on the application server.
Traditionally, an Oracle database provided a single service and all users connected to the same
service. A database will always have this default database service that is the database name. This
service cannot be modified and will always allow you to connect to the database. With Oracle Database 11g, a database can have many services (up to a maximum of 115 per database). The
services hide the complexity of the cluster from the client by providing a single system image for managing work. SAP applications and mid-tier connection pools select a service by using the
Service Name in their connection data.
Oracle Data Guard
Oracle Data Guard provides the management, monitoring, and automation software infrastructure
to create and maintain one or more standby databases to protect Oracle data from failures, disasters, errors, and data corruptions. SAP supports Oracle Data Guard physical and logical
standby databases, Oracle Active Data Guard, and snapshot standby databases.
See the following documents for complete information:
Oracle Data Guard Concepts and Administration for complete details about Oracle Data Guard and standby databases
Oracle Data Guard Broker for information about broker management and fast-start failover
My other Dell Paper : Implementation of SAP Applications Using Oracle Maximum Availability Architecture Best Practices
Oracle Flashback
Oracle Flashback quickly rewinds an Oracle database, table or transaction to a previous time, to
correct any problems caused by logical data corruption or user error. It is like using a 'rewind button' for your database. Oracle Flashback can also quickly returns a previously primary database
to standby operation after a Data Guard failover, thus eliminating the need to recopy or reinstantiate the entire database from a backup. See also Oracle Flashback Technology.
Oracle Automatic Storage Management (Oracle ASM)
Oracle Automatic Storage Management provides a vertically integrated file system and volume
manager directly in the Oracle kernel, resulting in:
Significantly less work to provision database storage
Higher levels of availability
Elimination of the expense, installation, and maintenance of specialized storage products
Unique capabilities for database applications
Page | 8
For optimal performance, Oracle ASM spreads files across all available storage. To protect against
data loss, Oracle ASM extends the concept of SAME (stripe and mirror everything) and adds more flexibility in that it can mirror at the database file level rather than the entire disk level.
1.2. SAP Application Components
SAP application consists of one or more instances of an application server. Each instance can run on a separate server, but it is also possible to operate multiple instances on one host. An SAP
instance can provide different service types. The standard SAP services that can be configured on all instances of the SAP component are dialog, batch, update, and spool. The failure of an SAP
instance on which these standard services are configured causes all the transactions processed on it to be terminated and rolled back. Database consistency is guaranteed at all times. Terminated
transactions can be repeated on one of the instances still available.
The key components that need to be protected are -
SAP Message Service:
The SAP Message Service is used to exchange and regulate messages between SAP instances in a
SAP network. It manages functions such as determining which instance a user logs onto during
client connect and scheduling of batch jobs on instances configured for batch. The VMware HA & VMware FT will protect SAP central services(ASCS + Message Server) by failing them over to the
surviving node.
SAP Enqueue Service:
The enqueue service is a critical component of the SAP system. It administers locking using objects within SAP transactions that can be requested by applications to ensure consistency within the SAP
system. The enqueue service component can be protected with either of the two available options
for Oracle: Oracle Clusterware Based sapctl or VMware‟s Fault Tolerance ( FT). VMware FT was chosen for this implementation for two main reasons: SAPCTL 11gR2 product was not certified for
an older version of SAP and VMware FT was easy to implement right out of the box.
The enqueue service manages locking of business objects at the SAP transaction level. Locks are set in a lock table stored in the shared memory of the host on which the enqueue service runs.
Failure of this service has a considerable effect on the system because all the transactions that
contain locks would have to be rolled back and any SAP updates being processed would fail. This critical service is protected the VMware HA & VMware FT . The VMware FT provides continuous
availability for applications in the event of server failures, by creating a live shadow instance of a virtual machine that is in virtual lockstep with the primary instance. By allowing instantaneous
failover between the two instances in the event of hardware failure, FT enables zero downtime for
the application.
In the SAP Appserver the message and enqueue processes have been separated from the CI and grouped into a standalone service. The ABAP variant is called ABAP SAP Central Services (ASCS).
The isolation of the message and enqueue service from the CI helps to address the high availability requirements of these SPOFs. The central services component is "lighter" than the CI and hence is
much quicker to start up after a failure. But in our case we have FT to make sure it does not fail and hence 1CPU is sufficient to address the requirements.
Page | 9
Logon Load Balancing
The assignment of user logons to application servers occurs during logon. Application servers can be combined in logon groups, whereby logon assignment depends on the load. In the event of
application server failure, the user can log on to another application server in the logon group.
However, the data from the user session on the original application server is lost.
The same procedure is also possible for linking systems by remote function call (RFC), where
application servers can be combined in RFC groups.
1.3. VMware Components
VMware FT relies on deterministic record/replay technology described above. When VMware FT is
enabled for a virtual machine (“the primary”), a second instance of the virtual machine (the “secondary”) is created by live-migrating the memory contents of the primary using VMware®
VMotion™. Once live, the secondary virtual machine runs in lockstep and effectively mirrors the guest instruction execution of the primary.
Page | 10
The hypervisor running on the primary host captures external inputs to the virtual machine and
transfers them asynchronously to the secondary host. The hypervisor running on the secondary host receives these inputs and injects them into the replaying virtual machine at the appropriate
execution point. The primary and the secondary virtual machines share the same virtual disk on shared storage, but all I/O operations are performed only on the primary host. While the
hypervisor does not issue I/O produced by the secondary, it posts all I/O completion events to the
secondary virtual machine at the same execution point as they occurred on the primary.
When a host running the primary virtual machine fails, a transparent failover occurs to the corresponding secondary virtual machine. During this failover, there is no data loss or noticeable
service interruption. In addition, VMware HA automatically restores redundancy by restarting a
new secondary virtual machine on another host. Similarly, if the host running the secondary virtual machine fails, VMware HA starts a new secondary virtual machine on a different host. In either
case there is no noticeable outage by an end user.
The actual logging time delay between the Primary and Secondary Fault Tolerance virtual machine
is based on the network latency between the Primary and Secondary. vLockstep executes the same instructions on the Primary and Secondary, but because this happens on different hosts,
there could be a small latency, but no loss of state. This is typically less than 1 ms. Fault Tolerance includes synchronization to ensure that the Primary and Secondary are synchronized.
The failover from primary to secondary virtual machine is dynamic, with the secondary continuing execution from the exact point where the primary left off. It happens automatically with no data
loss, no downtime, and little delay. Clients see no interruption. After the dynamic failover to the
secondary virtual machine, it becomes the new primary virtual machine. A new secondary virtual machine is spawned automatically.
Page | 11
1.4. Bolt-on Third Party Software Architecture
Various Bolt-on applications were migrated and each of these components new architecture supported HA & FT. The third party bolt-on applications that were migrated along with the SAP
Applications were –
Redwood CPS : Job Scheduling
Connect:Direct : Secure FTP, Telnet
TIBCO Servers : Integration
TOPLINK : FAX Services
Printing Services
FTP and
Scripts
The TIBCO servers have their own independent server for processing. They interface with the SAP
ASCS over RFC and flat files mounted using NFS.
1.5. REDWOOD CPS Servers
The Redwood HA is supported by a TWO-Tier Architecture with the repository DB as the backend to the Redwood Process servers in the front end. –
- The Redwood repository is protected by Oracle RAC and which is highly available and scalable.
- The Redwood process servers are running on the VM’s supported by HA and these schedulers can be restarted on the failed over VM. The database agents are sent (in a round-robin) to all RAC instances. At least one database agent is started against all instances even if that number is higher than the SERVERS setting. If an instance has one database agent, and that agent becomes the dispatcher agent, an extra database agent is started. Therefore, it is imperative that you can start the process server on all instances, even if you normally connect users (and the process server) to only one of the nodes.
3 Platform Migration Strategy The platform migration is a heterogeneous system copy to create a copy of an already existing R/3 system on the platform where either operating system, or database, or both differ from the
operating system/database of the source system.
The whole migration process consists of five main steps:
1. Preparation steps on the source system 2. Export of the source system data into database-independent format or in a format that is
acceptable by the source database 3. Transfer of the data made during the export, 4. New system installation together with data import. 5. Post-processing steps within the target system
Page | 12
3.1.1 Migration approach The main tools used for the migration are „R3SETUP‟ or „sapinst‟ depending on the kernel release. These tools call and execute installation templates, command files each with extenstion of R3S
(R3SETUP) or xml (sapinst) dependant on the tool used.
For the kernel-tools used during a migration some rules should be followed:
a) Tools used for export on the source system must have the same version as on the target system.
b) Tools used must all have the same kernel-version. (Do not mix up kernel-tools of different releases)
c) Tools must have the same kernel release as the system which is migrated. d) The java system copy tools do not depend on the kernel-version and you can always use
the latest version of these tools.
The SAP supported approach to move SAP systems to a different operating system is to perform a heterogeneous system copy. For moving an Oracle database used by SAP applications to a
different OS, SAP supports a few different approaches and each of them are listed below with their pros and cons :
Tool Options Pros Cons
R3Load Very Reliable
Easy to Upgrade to newer DB versions
Easy to Implement when used with standard SAP installation tools
Compress Data
Allow Unicode Conversion if needed Oracle 11g advanced compression in the target database is possible
**May require longer downtime, full
support at kernel 7.0 or higher
Time-consuming if used without complex tuning techniques
Multiple test runs are usually required with multi-terabytes databases when complex tuning techniques are required
Downtime : >32 HRS* * Minimum downtime is heavily
dependent on available CPU, memory, and IO capability of the DB server
RMAN
DUPLICATE/RESTORE/ RECOVER Mixed
Platform Support [ID
1079563.1]
Solaris-x64 Linux-x64 : Not
Applicable
RMAN Backup does not support
cross endianness configuration.
Data Guard Support
for Heterogeneous
Primary and Logical Standbys in Same Data
Guard Configuration [ID 1085687.1]
Solaris-x64 Linux-x64 : Not
Applicable
Data Guard does not support cross
endianness configuration.
Not Applicable
Data Guard Support
for Heterogeneous Primary and Physical Standbys in Same Data
Solaris-x64 Linux-x64 : Not
Applicable
Data Guard does not support cross
endianness configuration.
Not Applicable
Page | 13
Guard Configuration
[ID 413484.1]
Cross Platform Transportable
Tablespaces
Reliable Process
Fast based on the data move speeds compared to R3Load
FREE no additional license required
Easy process
Source Database has to be > 10.2.0.4
Need a few runs
Downtime : >12 HRS*
No Failback
Cannot Compress Database * Minimum downtime is dependent
on available CPU, memory, and IO capability of the DB server and DB Size
Oracle Goldengate
or ooo Reliable Process
Zero Downtime
Failback Possible
Both Target/Source can be active
Not Easy to Implement
Very Expensive
License Costs
Standard R3load-based system copy:
The data is exported from the source database to a staging area and imported into the target
database using SAP‟s R3load tool. This is the standard approach for installing new SAP systems and the SAP recommended approach for platform and Unicode migrations.
This is a new approach available for databases running on Oracle 10g and higher. The data files of user tablespaces are copied to the target database server using various approaches such as NFS or
FTP, and then adapted to the target environment using Oracle tools.
Recommended Approach
For migrating the current SAP R/3 4.7 system, we recommend the R3load-based approach because of the following reasons:
The current system is running on Oracle 9.2. Data can be exported from Oracle 9.2 and directly imported into Oracle 11.2, thereby completing the database upgrade without changing the source environment.
Using the R3load method to migrate the database would reorganize the database in the process; this would remove any existing fragmentation and potentially reduce the space used. Additionally, advanced features such as compression will be implemented in the process.
Page | 14
3.1.2 Migration Steps on the Source System
Step 1 Preparation
Perform database consistency checks, verify relevant SAP configurations and correct any
inconsistencies. Data reduction methods such as SAP data archiving can be utilized to reduce the amount of data
to be exported during this phase. SAP installation CDS, Java SDKs, and migration tools should be downloaded. Analyze the database to determine the best optimization strategy to reduce
downtime.
Local storage space should be allocated on all participating app servers. On the target database server, at least the same amount of the space as on the source database server should be
allocated.
An NFS share should be created on the source database server as the central directory for
Distribution Monitor and shared by the participating servers and the target database server.
Due to the large amount of network traffic during the migration, jumbo frames should be enabled.
During the production cutover weekend, workload on the SAN storage should also be minimized if possible in order to meet the high IO requirements during the downtime. This would require
advanced planning and scheduling.
To prepare for the migration of the productive system, the SAP OS/DB Migration Check should be
ordered in advance. The service can be paid for by using the free service credit that is included as part of the SAP Standard Support or SAP Enterprise Support contract.
Step 2 Generate Control Files for R3load
The control files for R3load can be generated by using either SAPINST or R3ldctl. (R3Load Control)
Those files will contain definitions of the database objects in the source database.
Step 3 Split large tables and distribute packages to participating application servers
Largest tables in the database can be divided into smaller chunks and assigned to participating application servers by using Distribution Monitor and other SAP tools. This will allow multiple
R3load processes to run on multiple application servers in parallel to take advantage of all the
available processing power and reduce the runtime of the migration process.
Step 4 Optimize database parameters for the export
Page | 15
Create a temporary Oracle SPFILE or PFILE specifically to be used during the export process. The
parameters should be adjusted based on the most up to date SAP recommendations and calculated based on the number of CPUs and the amount of memory. Those parameters will only be used
during the export and should be changed back after the migration.
Step 5 Perform parallel export using Distribution Monitor
Start the export processes using Distribution Monitor on all of the participating servers. Carefully monitor the export process and resolve any problems during the export.
3.1.3 Migration Steps on the Target System The migration steps on the source system can be executed once the R3load control files and the
database size file generated by R3szchk become available on the disk
Step 1 Get migration key
The migration key can be generated well in advance at the SAP Service Market place using a
customer “S number” with the proper authorizations. It will be used in the import process for
changing operating system, database, or both.
Step 2 Install SAP and database software
SAP binary files and Oracle database software should be installed on the target database server by
using SAPINST and Oracle Universal Installer.
Step 3 Create database and empty user tablespaces
SAPINST will create the target database and user tablespaces. To speed up the process, data files can be added by using sqlplus in parallel. At least the same amount of space as in the original
database should be allocated.
Step 4 Optimize database parameters for the import
Create a temporary Oracle SPFILE or PFILE specifically to be used during the import process. The parameters should be adjusted based on the most up to date SAP recommendations and calculated
based on the number of CPUs and the amount of memory. Those parameters will only be used during the import and should be changed back after the migration.
Step 5 Perform parallel import using Distribution Monitor
Start the import processes using Distribution Monitor on all of the participating servers.
Carefully monitor the import process and resolve any problems during the import. The import and export processes can be started at the same time. Packages that have been
exported completely will be picked up by available import processes automatically.
Step 6 Perform post import technical steps
Continue SAPINST to complete the post-import verification. SAPINST will also execute required batch jobs and create a temporary SAP license for the target system. New license has to be
installed as soon as possible before the temporary license expires. Database statistics can be created using multiple CPUs in parallel. Follow up steps in the SAP System Copy Guide should be
executed at this time.
Page | 16
ABAP loads for the new platform should be generated using all application servers now to avoid
compiling when end users use the system.
Step 7 Post migration test activities
After the first successful migration, functional validation should be performed in the migrated
system. The performance of the most time-critical transactions should be tested and optimized for
the new platform. SAP, Database, and Operating system parameters should be adjusted accordingly to ensure optimal performance. Interfaces and custom programs should also be
tested. Any issues discovered should be resolved before the final test run and production cutover.
After the production cutover, a small set of validation tasks should be executed to validate the
most business critical functions before Go-live.
Four weeks after the production cutover, SAP will perform a Migration Check Verification session
remotely and provide additional recommendations for the new platform.
A separate Golive cutover will be executed for each of the following environments:
Technical Sandbox
Functional Sandbox
Development
Quality Assurance
Pre-Production
Production
In order to support current implementations in SAP systems to be migrated, different transport groups will be set up to support the synchronization process between two data centers so that interruption to current projects can be kept to a minimum.
Page | 17
4 Implementation Steps (VMware & Oracle)
This section is an example of the installation & configuration steps performed at the customer site
for one of their environments. It is required that you must have performed a VMware, Oracle RAC & SAP installation prior to this section.
The implementation steps at a high level includes the following stages –
Install the Operating System on all the Servers (Not Covered)
Install VMware on the ESX servers (Not Covered)
Configuration of VMware
Create the ASM LUNS
Install the Oracle Clusterware
Create ASM Disk Groups
Install the Oracle Software (Shared ACFS Disk)
Create the Database
Install the SAP software
Perform the DB Migration of SAP & Redwood CPS
1.6. System Administration Tasks (OS, Storage, VMware)
4.1.1 VMware vSphere two node Cluster
Two Node ESX clusters were created to support the app servers. Connectivity for each vSPhere
ESX 4.1 node consists of 8 NICs partitioned with Virtual LANs and 1 NIC for out-of-band
management (DELL iDRAC).
To achieve load balancing and fault tolerance the standard Vmware vswitch capabilities are used. The table below depicts the network configuration of each node:
vswitch description vmnic phisical nic Connection Type
Page | 18
vswitch0 global vswitch vmnic0 embedded port 1 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic1 embedded port 2 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic2 embedded port 3 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic3 embedded port 4 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic4 slot 5, port 1 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic5 slot 5, port 2 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic6 slot 3, port 1 Switched 1000Base-T - switchport Trunk 802.1q Mode
vswitch0 global vswitch vmnic7 slot 3, port 2 Switched 1000Base-T - switchport Trunk 802.1q Mode
To support this configuration each switch port connected to ESX must be configured with 802.1q encapsulation to include in the Ethernet frame the VLAN Tag.
The Virtual configuration is described in the following picture:
Port groups configuration and balancing logics:
Port Group Description Active Vmnic StandBy vmnic Balancing type VLAN ID
SAP SAP Application connectivity 0,1,2,4,6 7,5,3
port Id 96
Management Management interface 3 0,1,2,4,7,5,6
port Id 96
vMotion vMotion traffic 5 0,1,2,4,7,3,6 port Id 12
FT Fault Tolerance Traffic 7 0,1,2,4,6,3,5
port Id 12
vSwitch0 configuration:
Attribute Specification
Network Failover Detection Beacon Probing
Notify Switch Yes
Page | 19
FailBack Yes
Oracle nodestorage allocations
Local storage
Local disks are configured in RAID 1 (mirror) protection.
OS filesystem:
mount point Size (GB)
/tmp 5
/home 5
/ 13
/usr 10
/var 5
Swap 48
/u01 25
/local/Oracle 25
SAN storage:
DB # of LUNs
LUN Type
LUN Size (Gb)
Initial Go Live Storage Allocated (Gb)
ASM Disk Group Comments
LP3 5 Fiber 2 10 OCR Voting & Ocr
LP3 2 Fiber 100 200 ACFS ORACLE_HOME
LP3 30 Fiber 400 12000 DATA_DG1 Redo, Control files(Copy 1), system, temp, undo and application data
The Oracle Clusterware installation is an independent task to the SAP software installation. The following task tasks must be completed prior to installation of the Oracle GRID –
Page | 23
Installation of the REDHAT Linux AS5U6 with all required packages as per Dell best practices
o Kernel Parameters o IP address setup o User equivalence o Multipathing of HBAs o Teaming of NICs
Creations of LUNS
ASMLIB installation
4.1.1.1 Oracle Clusterware Installation
Once the above tasks are completed then we initiate the installation process of the Oracle
Clusterware.
**PLEASE NOTE MANY SCREENSHOTS HAVE BEEN REMOVED IN THE STEP SEQUENCE **
./runinstaller
Page | 24
Page | 25
Page | 26
Page | 27
4.1.1.2 ASMCA Tool for ASM Disk Group
It is recommended to perform all ASM related tasks using the Oracle supplied ASMCA tool.
After Oracle clusterware has been installed, create the required ASM DISK groups and shared file system that will be used to hold the Oracle Software and home directory (ORACLE_HOME).
Cd $ORACLE_HOME
Cd bin
./asmca
Page | 28
Create a ACFS Shared Oracle Home which is shared across the cluster.
Page | 29
Page | 30
Page | 31
Page | 32
4.1.1.3 Oracle Database Software Installation ONLY The Oracle Software is only installed to enable the SAPINST for creating the SAP SEED database.
Page | 33
Page | 34
Page | 35
******
Page | 36
4.1.1.4 Oracle RAC Database Creation
Create ORACLE RAC database using DBCA but with OHOME set to 112_64
This will create all the necessary Oracle Database Services.
Page | 37
Page | 38
Page | 39
Page | 40
Shutdown the DB
$Srvctl stop database –d <DB NAME>
Then convert it into SINGLE INSTANCE for using the SAPINST LOAD. The conversion is a simple exercise of starting the database with a new pfile where couple of init.ora parameters are modifed.
Start the database using a pfile which has the following changes –
After the LOAD and all SAPINST activities are completed shutdown the databases and start the
database in RAC mode using –
$Srvctl start database –d LS6
5 Implementation Steps (Applications) 1.8. SAP Export/ Installation Process
Check tool versions:
Migration Monitor
- Download the file migmon.sar from SAP Service Marketplace from the following path: http://service.sap.com/swdc -> Download -> SAP Support Packages and Patches ->
Entry by Application Group -> Additional Components -> SYSTEM COPY TOOLS ->
SYSTEM COPY TOOLS 7.00 -> #OS independent -> MIGMON_3-20001410.SAR. - Download the latest version of R3load
Export steps:
- Check SAP system and execute pre-export steps as described in System Copy Guide
Page | 41
- Wait for SAP shutdown
- Ensure application servers are shut down permanently except for contingency scenarios
- Logon as ls6adm into the source system database server (dell00t6) - Set JAVA_HOME variable setenv JAVA_HOME /opt/java1.4
- Generate R3load control files:
/sapmnt/LS6/exe/R3ldctl -l ./R3ldctlExport.log -p /dell/export/DATA/ - Create separate packages for tables and packages
Reference: SAP Note 943334 - TMS setup in high availability systems for technical details.
Instruction:
In transaction SE38, Execute Report TMS_MGR_LOADBALANCING:
Page | 65
Enter the SYSTEM(s) or DOMAIN(s) to be enabled or disabled for Load Balancing using logon
group.
Check the box “LOADBLON” to enable logon balancing. Uncheck it to disable it.
Execute.
Double click on the system to be updated (for example: LP3)
Page | 66
Click on change.
Update the target host and system number for the new configuration.
Save and distribute the configuration
Page | 67
6 Dynamic Workload Management Oracle RAC enables enterprise GRIDS. Enterprise GRIDs are built from standardized commodity-
priced processing, storage and network components. The enterprise GRID enables you to consolidate applications into a cluster with Oracle clusterware allocating the resources to the server
pools running the components of the application such as the SAP application server and the Oracle
RAC database.
In the example below the workload will be managed across the 4 cluster nodes using the database services. Services represent groups of applications with common attributes, service level
thresholds, and priorities. Application functions will be divided into workloads identified by services
either for Batch Jobs or OLTP Trxs based on requirements.
Page | 68
The Oracle RAC provides DB services to perform workload management. Workload management relies on the use of Services, a feature of Oracle database services to hide the complexity of a RAC
database by providing a single system image to manage workload. Services allow applications to benefit from the reliability of a cluster.
An SAP application consists of one or more instances of an application server. Each instance can run on a separate server, but it is also possible to operate multiple instances on one host. An SAP
instance can provide different service types. The standard SAP services that can be configured on all instances of the SAP component are dialog, batch and update. The failure of an SAP instance on
which only these standard services are configured causes all the transactions processed on it to be terminated and rolled back. Database consistency is guaranteed at all times. But long running SQL
SELECT queries are transparently failed over to the surviving nodes. Terminated transactions can
be repeated on one of the instances still available.
The use of Services is required to take advantage of the load balancing advisory and runtime connection load balancing features of Oracle RAC. When a service is created, a service level
goal is defined.
CLB_GOAL
Long(2). Use the LONG connection load balancing method for applications that have long-lived connections. This is typical for connection pools and SQL*Forms sessions. It does not matter if GOAL is set or not for this condition as the point behind this setting is to balance based on number of sessions. LONG is the default connection load balancing goal.
Short(1).Use the SHORT connection load balancing method for applications that have short-lived connections. The database uses first the GOAL setting to have PMON tell the Listener which node to prefer
GOAL
None(0):When set to 0(NONE), this disables the ONS notifications to the Client as to the load of the various nodes.
Service Time (1)Attempts to direct work requests to instances according to response time. So if one node takes longer to do the same work, the client can be informed of this load difference, so it can now direct further work to the node that is taking less time. Load balancing advisory data is based on elapsed time for work done in the service plus available bandwidth to the service.
Page | 69
Throughput(2):Attempts to direct work requests according to throughput. The load balancing advisory is based on the rate that work is completed in the service plus available bandwidth to the service. Instead of figuring out how long something takes, it is the frequency this work occurs that is used. So if node one is able to handle 10 transactions, while node two can handle 12, in the same amount of time, then the client will be told to go to node two. So even if node two will take longer to handle a specific job, it can handle more jobs at one time then node.
srvctl add service -d LP3 -s LP3_D01 -r LP31 -a LP32 -P BASIC -e SELECT -m BASIC -j LONG -B
SERVICE_TIME -z 10 -w 2
srvctl add service -d LP3 -s LP3_D02 -r LP32 -a LP31 -P BASIC -e SELECT -m BASIC -j LONG -B
SERVICE_TIME -z 10 -w 2
srvctl status service -s LP3_D01 -d LP3
Service LP3_D01 is not running.
srvctl start service -s LP3_D01 -d LP3
srvctl status service -s LP3_D01 -d LP3
Service LP3_D01 is running on instance(s) LP31
srvctl start service -s LP3_D02 -d LP3
srvctl status service -s LP3_D02 -d LP3
Service LP3_D02 is running on instance(s) LP32
[oracle@dell00755 ~]$ srvctl config service -d LP3
Please setup the environment as per SAPNOTE#1598594: BR*Tools configuration for Oracle
installation under "oracle" user.
Create Control file alias :
- alter diskgroup OLOG add alias '+OLOG/lp3/cntrlLP3.dbf' for '+OLOG/lp3/controlfile/current.261.760059081'
- alter diskgroup MLOG add alias '+MLOG/lp3/cntrlLP3.dbf' for '+MLOG/lp3/controlfile/current.261.760059081'
This procedure combines Storage, EMC related actions, and Database Administration actions. All
Storage related actions are highlighted to differentiate them.
On production we have a set of scripts for each step that are run from the enterprise scheduler.
Steps in Brief:
1. Synchronize the Device Groups (establish) 2. On ASM set balance power = 0 3. Archive current on both instances 4. Place database on backup mode 5. Perform consistent split for disk group DATADG 6. End backup mode on the database 7. Create control files backup on disk group ARCHDG 8. Perform consistent split for disk group ARCHDG that contain archived logs and control file
copies 9. Optionally backup the snapshot site to tape
Snapshot Procedure:
Page | 71
1. Prepare the remote host Oracle Homes (optional)
You may configure a remote site to mount and check the backup consistency. This step is only required at implementation, afterwards do not require maintenance unless there
are structural changes.
a) copy ASM init.ora parameter file: init+ASM.ora to ASM home, dbs directory b) copy ASM password file orapw+ASM to ASM home, dbs directory c) copy Database init.ora parameter file: initTSTASM1.ora to Oracle home, dbs directory d) copy Database password file orapwTSTASM1 to Oracle home, dbs directory e) update hostname on parameter files to reflect target hostname. f) create dump directories for database and asm instance
2. Identify the LUN's in ASM Datadg disk group:
This information needs to be passed to the Storage Manager in order to configure the Symmetrix
Device Groups and is required only at the implementation stage, afterwards it needs to be updated
only if there are structural changes.
GROUP_NUMBER PATH NAME ------------ ------------------------------ ----------- 1 /dev/emcpowera ARCHDG
3 /dev/emcpowere DATADG 3 /dev/emcpowerg DATADG
3. Create Symmetrix Device Groups and Associate BCV devices:
Two groups are required because the group containing the redo logs require to be split after the group containing the data.
4. Establish (synchronize) The Device Groups:
Make sure de Device Groups are established.
5. Online Backup on Production Host:
Turn rebalance power to 0 on ASM instance
a) ALTER DISKGROUP data REBALANCE POWER 0 WAIT; o Force a log switch using archive log current
b) ALTER SYSTEM ARCHIVE LOG CURRENT o Place the database in backup mode
c) ALTER DATABASE BEGIN BACKUP o select file#, status from v$backup
6. Perform a Consistent Split Snapshot for Database Files:
Perform a point in time snapshot of DATADG
7. End Backup Mode:
d) Take the database out of backup mode e) Force a log switch
8. Create Backup of control files:
Page | 72
f) Create 2 controlfile backups, "control_start" and control_backup on ARCHDG g) Prepare an init ora pointing the control file to control_start on ARCHDG if not ready from
point 1. 9. Perform a Consistent Split Snapshot of the Recovery Area to capture the Archive Log:
h) Perform a snapshot of the volumes containing the archived logs ARCHDG 10. Start ASM instance on Backup Host and perform checks. (Optional)
If you did configure a backup server to check the R2 then follow these steps:
i) Startup the ASM instance on the Backup Host j) Check that all diskgroups are successfully mounted k) Startup and mount the database using the parfile created on "i" , that points to
control_start l) recover and startup readonly
RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL;
alter database open read only;
Optional Backup the Backup Steps:
m) Optionally use Backup EMC snapshots of the r2 copy n) Optionally make EMC snapshots of r2 to be used as report instances, development or
tests environments. o) Optionally backup r2 to tape using RMAN.
8 Reference Oracle Paper : Configuration of SAP NetWeaver for Oracle Grid Infrastructure 11.2.0.2 and
Oracle Real Application Clusters 11g Release 2: A Best Practices Guide
Oracle Paper : SAP Databases on Oracle Automatic Storage Management 11g Release 2
Dell Paper : Implementation of SAP® Applications Using Oracle® Maximum Availability Architecture Best Practices
Homogeneous and Heterogeneous System Copy for SAP Systems Based on SAP Web Application Server 6.20 Document Version 1.25
Migration Monitor User’s Guide
Package Splitter User’s Guide
SAP Notes :
SAP Note Number Title Description
355771 Oracle: Explanation of the new tablespace layout
9 Authors Customer Success Story: Migrating a traditional physical server configuration to a Dell
SAP High Availability (HA) Tiered Private Cloud Architecture
May 2012
Author: Mahesh Pakala
Mahesh Pakala is a highly experienced professional in Information Technology and Media Industry in the Silicon Valley. He has worked in various senior management roles in the industry and is on advisory panel of companies and venture firms as a technology and business consultant.
He works in the GICS (Global Infrastructure Consulting Services) Group of Dell Inc., and assists large enterprise customers with HA architectures for SAAS/Cloud Implementations. He has extensive work experience in areas of – engineering, media and technology with companies, such as Oracle Corporation (System Performance Group & RDBMS Kernel Escalations), Ingres (Computer Associates), Fujitsu and startups. He has been a speaker in the areas of High Availability and System Architecture at various conferences.
Many Thanks to my esteemed colleagues for their in-depth knowledge, contribution and involvement in this project and without which it could not have been successfully completed – Mr. JungFeng Chen(SAP wizard), Matteo Mazzari(VMware wizard) , Alebardi Luca(PM Wizard), Antonio Apollonio(Services wizard) & Emanuele Riva(Fearless Leader) for assisting with the SAP Platform migration implementation on Linux/Dell for a major mission critical application.
Special thanks to our customer for their confidence and trust in our team for the end-to-end delivery capabilities!!
Reviewed By – Jan Klokkers(Oracle Corp), Ron Pewitz, Morten Loderup, Nathan Saunders, Thorsten Staerk & Scott Lee
This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.
SAP, Oracle, VMware, Redwood, Tibco, TOPCALL, and Dell, are registered trademarks of SAP, Oracle Corporation, VMware, Redwood, Tibco and DELL and/or its affiliates. Other names may be trademarks of their respective owners.
This document is intended to address migrating a database only. Regardless of the method chosen to migrate to a new platform, there are additional areas that must be considered to ensure a successful transition to the new platform, such as understanding platform-specific features and changes in the Oracle, SAP and Redwood software. Refer to the platform-specific installation guides, release notes, and READMEs for details.
![f a reassembled PDU] - Texas Instrumentse2e.ti.com/cfs-file/__key/communityserver-discussions-components... · 2050 86.673954000 192.168.1.103 192.168.1.100 ... [TCP segment o f a](https://static.documents.pub/doc/80x56/5b05c4e67f8b9a93418bcf7a/f-a-reassembled-pdu-texas-86673954000-1921681103-1921681100-tcp-segment.jpg)
