SOA Cloud Services Disaster Recovery - oracle.com · SOA CLOUD SERVICES DISASTER RECOVERY 12....

SOA Cloud Service Disaster Recovery

Production and DR in the Cloud

O R A C L E W H I T E P A P E R | A U G U S T 2 0 1 8

SOA CLOUD SERVICES DISASTER RECOVERY

SOA CLOUD SERVICE DISASTER RECOVERY

Table of Contents

Introduction 1

Disaster Protection Considerations for Oracle SOA Cloud Service 3

Service Level Requirements 5

Security Requirements 5

Configuration Requirements 6

SOACS DR Deployment 7

OPTION1 Pre-condition: primary SOACS System already exists 8

1. Create virtual hostname for the frontend OTD and update frontend address

in primary site 9

2. Update t3/RMI urls (if used) 11

3. Set up and verify Data Guard 13

4. Provision SOACS in secondary location pointing to snapshot standby

database 13

5. Create virtual hostname for frontend OTD and update frontend address in

secondary site 14

6. Update t3/rmi urls in the secondary site (if used) 14

7. Copy and update fmwconfig bootstrap artifacts to secondary location 15

8. Replace schema prefix in the standby datasources 15

9. Convert the snapshot standby database back to the physical standby

database 17

10. Switchover Database 18

11. Verify SOACS in secondary site 18


12. Switchback to original site. 18

SOACS DR Lifecycle Procedures 19

Switchover 19

Failover 19

Applying domain configuration changes to the system 20

a) Repeating domain configuration changes in both sites 20

b) Using DBFS for configuration propagation 21

Scaling the SOACSDR system 35

Conclusion 37

Appendix A - OPTION2 38

A. Provision DBaaS in both locations 38

B. Set up and verify Data Guard 38

C. Provision SOACS in primary location 38

D. Create virtual hostname for front end OTD and update front end address in

primary site 39

E. Update t3/rmi urls (if used) 39

F. Switchover Database 39

G. Provision SOACS in secondary location 39

H. Create virtual hostname for front end OTD and update front end address in

secondary site 39

I. Update t3/rmi urls in the secondary site (if used) 39

J. Copy and update fmwconfig bootstrap artifacts to secondary location 39

K. Replace schema prefix in datasources 39

L. Verify SOACS in secondary site 39


M. Switchback to original site. 39

Appendix B - Setting up Data Guard 40

Appendix C - Creating and dispersing wallets in 12c 56

.

1 | SOA CLOUD SERVICES DISASTER RECOVERY

Introduction

Oracle’s Maximum Availability Architecture (Oracle MAA) is the best practices blueprint for data

protection and availability of Oracle products (Database, Fusion Middleware, Applications) deployed

on on-premises, private, public or hybrid clouds. Implementing Oracle Maximum Availability

Architecture best practices is one of the key requirements for any Oracle deployment. Oracle Fusion

Middleware and Oracle Databases include an extensive set of high availability features, such as

process death detection and restart, clustering, server migration, clusterware integration, GridLink,

load balancing, failover, backup and recovery, rolling upgrades, and rolling configuration changes,

which protects application deployments from unplanned downtime and minimize planned downtime.

Oracle Service-Oriented Architecture (SOA) Cloud Service is a Platform as a Service (PaaS)

computing platform solution for running Oracle SOA Suite in the cloud. This computing platform uses

Oracle Compute Cloud Service, Oracle Database Cloud Service and Oracle Java Cloud Service as its

basic infrastructure Oracle SOA Suite requires an Oracle Database to store Oracle Platform Security

Services information, instance tracking, composite and document metadata and other Oracle FMW

Infrastructure schemas. In a typical Oracle SOA deployment the application data (such as application-

specific schemas, jms stores etc.) and the SOA-specific schemas are stored in the same database for

transactional consistency and simplified administration reasons. In a SOA Cloud Service instance an

Oracle Database Cloud Service instance is used to store these schemas. All Oracle SOA deployments

need protection from unforeseen disasters and natural calamities. This protection is also required for

systems deployed in the Cloud and it needs to address both the middle tier (Oracle SOA Suite Cloud

Service) and the data tier (Database Cloud Service). The solution involves setting up a standby system

at a geographically different Oracle cloud data center than the production site. The standby system

may have equal or fewer services and resources compared to the production site. The standby system

is normally in a passive mode and is activated when the primary site becomes unavailable. This

deployment model is sometimes referred to as an active-passive model. This model is usually adopted

when the two sites are connected over a WAN and network latency does not allow clustering across

the two sites.

Oracle SOA Cloud Service (SOACS) provides the best Recovery Time Objective (RTO) and Recovery

Point Objective (RPO) by utilizing high availability and disaster protection capabilities provided by

Oracle Fusion Middleware and Oracle Database. While there are some unique considerations to a

cloud disaster recovery (DR) configuration, it follows the same Oracle MAA best practices as any

Oracle Fusion Middleware (FMW) and Oracle Database deployment. This Oracle MAA blueprint

details Oracle MAA Best Practices and provides a procedural overview for deploying DR for SOA


Cloud Service. Oracle SOA Cloud Service Disaster Recovery solution is achieved by replicating a

limited set of configuration files that are required to bootstrap SOA components. The application may

require additional configuration files to be replicated. Options are provided in this paper to suit different

application paradigms. Disaster protection for Oracle Database Cloud Service used by Oracle SOA

Cloud Service is provided through Oracle Data Guard.

This paper is intended for a technical audience having knowledge of Oracle Weblogic Server, Oracle

FMW SOA, Oracle Database, Data Guard and Oracle Database backup and recovery. This paper also

assumes a basic understanding of services offered on the Oracle Cloud1.

1 https://cloud.oracle.com/home

https://cloud.oracle.com/home


Disaster Protection Considerations for Oracle SOA Cloud Service

Oracle SOA Cloud Service uses DBCS (Oracle Database Cloud Service) to host the metadata and SOA instance

information. There are two options for protecting Oracle Database Cloud Services against disasters:

Data Guard utilizing Enterprise Edition Service or High Performance Service.

Active Data Guard utilizing the Extreme Performance Service or Exadata Cloud Service.

For more information on Data Guard and Active Data Guard please refer to the Data Guard home page on the

Oracle Technology Network and the Active Data Guard white paper2.

Oracle FMW SOA Suite can take advantage of the automatic block repair, fast incremental backups, fast rolling

upgrades, Far Sync, and most features of Active Data Guard. However, SOA servers cannot, however, use Oracle

Active Data Guard ‘s read-only query capabilities. This is because Oracle SOA Servers cannot run in read-only

mode. As soon as a SOA Server starts up, it connects to the database and tries to process business flows (i.e. they

“write” to the database right away). The only way to start the SOA servers in the standby site is by either 1)

performing a switchover/failover of the database or 2) by converting the standby database to a snapshot standby

which makes the standby read-writable temporarily for testing purposes. Redo received from the primary are stored

but not applied. The changes that are done are discarded at the end of testing where it can resume applying the

redo from the primary database. It needs to be noted that additional storage is required for the fast recovery area

when a standby is in snapshot mode (to hold archived redo received from the primary production database for later

use and current redo and flashback logs generated by the snapshot standby).

In a SOACS DR set up, the snapshot standby method is used to replicate configuration changes that were applied in

the production site when WLS domain configuration is not changed too frequently. The procedure (which will be

described later on in detail) consists in converting the standby database to a snapshot standby and applying again

the changes using the WLS Admin Console in the secondary location. This updates the file system artifacts (ear

files, deployment plans etc) to be the same as in the primary site. It is also a best practice to periodically place the

standby in read/write mode (using Data Guard Snapshot Standby) to validate its readiness to support read-write

production workloads. The snapshot standby, however, needs to be used carefully with SOA servers since

undesired processing of pending instances may occur when the SOA servers have access to a dehydration store. A

snapshot standby may also be used for a final level of pre-production functional and performance testing of patches

and upgrades since the DR system is frequently sized similar to the production system

Please refer to Oracle documentation for additional details on Data Guard Snapshot Standby3

Beyond this, the Oracle Cloud provides all backend infrastructure and capabilities required for disaster recovery

should the primary database become unavailable for any reason. This includes:

1. Ability to monitor the standby database and alert on major issues.

2. Ability to activate the standby to validate DR readiness and then convert back to a synchronized standby.

3. Utilization of essentially the same Oracle MAA best practices as on-premises deployments. This paper

describes the few considerations that are specific to cloud deployments.

2 http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf 3 http://docs.oracle.com/database/121/SBYDB/manage_ps.htm#BACIEJJI

http://www.oracle.com/technetwork/database/options/active-data-guard/overview/index.html

http://www.oracle.com/technetwork/database/options/active-data-guard/overview/index.html

http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf

http://docs.oracle.com/database/121/SBYDB/manage_ps.htm#BACIEJJI


4. Ability to switchover (planned event) or failover (unplanned event) production to the standby database in the

cloud during a planned maintenance or an unplanned outage. Once the failed primary database is repaired, it

can be resynchronized with the new production database in the cloud and then roles can be switched back to

the original status.

5. Ability to failover both the SOA/middle tier and database to enable production applications to run in the Oracle

Cloud when there is a complete site outage.

In the middle tier, the architecture of a SOACS DR system consists of two SOACS instances deployed in the same

sites as the DBaaS instances. Each SOACS instance uses a SOA Cluster, an Administration Server and OTD as

front-end load balancer. A single Database (DBaaS) instance is used to host both the SOACS schemas (MDS,

composite deployment, SOAINFRA schemas etc.) the JMS persistent stores, the Transaction Logs persistent stores

and any application specific data. Figure 1 describes this architecture:

Figure 1: SOACS Dr architecture


Service Level Requirements

Oracle SOA Cloud Service is a user-managed environment. The user must determine service level expectations for

availability, data protection, and performance that are practical for a given configuration and application. Service

Levels must be established for each of three dimensions relevant to disaster recovery that are applicable to any

Data Guard configuration:

» Availability: Recovery Time Objective (RTO) describes the maximum acceptable downtime should an outage

occur. This includes time required to detect the outage and to failover the database, the Web tier and SOA

servers so that service is resumed.

» Data Protection: Recovery Point Objective (RPO) describes the maximum amount of data loss that can be

tolerated. In SOA’s case this is especially related to transaction logs, JMS messages and SOA instance

information which all resides in the same database. The actual achievable RPO depends upon:

» Available network bandwidth.

» Network reliability.

» Data Guard transport method used: either asynchronous for near-zero data loss protection, or

synchronous for zero data loss protection.

» Performance: Database and Middle Tier response time may be different after failover if less capacity –

compute, memory, I/O, etc., are provisioned at the standby system than in the primary system. This occurs

when users purposefully under-configure standby resources to reduce cost (accepting reduced service level

while in DR mode). MAA best practices recommend configuring symmetrical capacity at both primary and

standby in the web, application and database tiers so there is no change in response time after failover. Rapid

provisioning available with the cloud can enable a middle ground where less capacity is initially deployed, but

where the new primary is rapidly scaled-up should a failover be required.

Note: Independent of the service levels related to DR, all database instances created in the Oracle cloud conform to

the service descriptions defined by the applicable Database Cloud Service4.

Security Requirements

Oracle MAA best practice recommends using Oracle Transparent Data Encryption (TDE) to encrypt primary and

standby databases at rest. Conversion to TDE enables automatic encryption at rest for all DATA/INDEX tablespaces

and encryption-in-flight of user data redo changes during replication to the cloud. Oracle Net encryption is also

required for encryption-in-flight for other redo changes that are not encrypted by TDE (e.g. data from unencrypted

tablespaces such as SYSTEM and SYSAUX).

Note: Data Guard and Active Data Guard use redo-based replication – a process that transmits redo generated by a

primary database and applies those changes to a standby database using continuous media recovery. This means

that primary and standby databases are block for block identical copies of each other. Using TDE to encrypt a

standby database also requires that the primary database be encrypted with TDE.

4 http://www.oracle.com/us/corporate/contracts/paas-iaas-public-cloud-2140609.pdf


Using TDE to protect data is an important part of improving the security of the system. Users should however be

aware of certain considerations when using any encryption solution, including:

» Additional CPU overhead: Encryption requires additional CPU cycles to calculate encrypted and decrypted

values. TDE minimizes the overhead by taking advantage of database caching capabilities and leveraging

hardware acceleration for AES on Intel and SPARC CPUs. Most TDE users see little performance impact on

their production systems after enabling TDE. Please refer to the Oracle Database Advanced Security Guide5

for more details.

» Lower data compression: Encrypted data compresses poorly because it must reveal no information about the

original plain text data. Thus, any compression applied to TDE encrypted data will have low compression ratios.

Hence, redo transport compression should not be enabled when TDE encryption is used. However, when TDE

is used in conjunction with Oracle database compression technologies such as Advanced Compression or

Hybrid Columnar Compression, compression is performed before the encryption occurs, and the benefits of

compression and encryption are both achieved.

» Key management: Encryption is only as strong as the key used to encrypt. Furthermore, the loss of the

encryption key is tantamount to losing all data protected by that key. If encryption is enabled on a few

databases, keeping track of the key and its lifecycle is relatively easy. As the number of encrypted databases

grows, managing keys becomes an increasingly difficult problem. For users with a large number of encrypted

databases, it is recommended that Oracle Key Vault6 on-premise be used to store and manage TDE master

keys.

When DBFS is used to maintain WLS domain configuration, using the appropriate encryption at the DB level

guarantees also the security of configuration propagation across sites.

Configuration Requirements

Beyond these runtime-related aspects, the following considerations are key to the SOACS DR set up:

A default database instance is automatically created when you create an Oracle Database Cloud Service.

On the standby site, the database initially created cannot be used as a Data Guard standby database. It

can be deleted or reused for a different purpoxse

The SOACS instance name in both primary & standby domains/locations should be the same. The

instance name is used to construct the hostnames, WLS server names and domain name where SOACS

will run. Since the domain configuration is not copied entirely from one site to the other, using the same

instance name guarantees consistency and allows the recovery of JMS messages and Tlogs. It also

simplifies customizations and operations in both sites.

MDS, Composite deployments and policies are automatically synchronized across sites by Data Guard

since they are stored in the database.

The SOACS WLS domain configuration is not globally synced across sites with the following

considerations:

o Each SOACS system will maintain the original JDBC URLs used to connect to the DB even after

the DR set up has completed. Only the schema prefix will be altered so that both locations point

to the same schemas

5 https://docs.oracle.com/database/121/ASOAG/asotrans_faq.htm#ASOAG10544

6 http://www.oracle.com/us/products/database/security/key-vault/overview/index.html

https://docs.oracle.com/database/121/ASOAG/asotrans_faq.htm#ASOAG10544

http://www.oracle.com/technetwork/database/options/compression/overview/index.html

http://www.oracle.com/us/products/database/security/key-vault/overview/index.html


o All the fmwconfig-OPSS in the WLS domain configuration is copied from one site to the other

and the JDBC string is modified after the copy so that OPSS points to the local DB

o Application deployments (workflow task ears, custom ear files, deployment plan updates,

redeployments etc) are not synced automatically across sites. The SOACS DR design offers two

alternatives for addressing this:

Systems with highly frequent domain configuration changes or application deployments

can use DBFS (stored in the database) to replicate the domain artifacts across sites.

Systems with infrequent domain configuration changes or application deployments can

convert the standby database to snapshot and apply the configuration change also in

the secondary site.

Once Data Guard has been configured and for systems in production, conversion of standby to snapshot

should be done only without starting the SOA servers in the standby site. This is because SOA servers will

try to process SOA instances and complete pending work (callbacks, async invocation etc.) that may have

been processed already on the primary site. Only the administration server should be brought up pointing

to a snapshot standby once the system is in production.

SOACS DR Deployment

In this document we assume as a starting point that the primary site, consisting of a single instance DBaaS, SOACS

Cluster and Oracle Traffic Director (OTD) is live and a dual site geographically distributed DR configuration will be

created. Refer to the SOACS Documentation for details about how to provision the initial set up. In summary and

leaving aside the Compute Shapes selected for SOA, OTD and the DB) similar options to the following should be

used:

The SOACS DR configuration makes failover/switchover transparent to end user or systems accessing the services

by making SOA endpoints agnostic to the site that is being used. For this to be possible, the front end address of the

existing SOA Cluster is altered to use a virtual hostname that can be assigned to either DR location’s OTD once the

configuration is completed. Appropriate DNS services (global DNS, local DNS server or manual file hostname

resolution) need to be used for the front end url to be mapped to either site. If any composites are deployed/used

http://docs.oracle.com/cloud/latest/soacs_gs/index.html


before this front end address change is performed, it may be required to alter the endpoint address to match this

new virtual hostname (NOTE: By default end point addresses are constructed based on the SOA/WLS cluster’s

HTTP front end parameter). Since the production SOACS may already be running production workloads, the DR

configuration process should cause as minimum downtime as possible.

There are two deployment options:

1. Option #1 is where a primary SOACS already exists.

2. Option #2 is a tiered approach where the Database tier is set up first and verified for DR and then the

middle tier is added on top. This approach is described in Appendix A.

This section describes the summary steps for set up process for Option #1:

OPTION1 Pre-condition: Primary SOACS System already exists

1. Create virtual hostname for front end OTD and update front end address in primary site

2. Update t3/rmi urls (if used)

3. Provision DBaaS in secondary location

4. Set up and verify DataGuard (service downtime)

5. Provision SOACS in secondary location pointing to snapshot DB

6. Create virtual hostname for front end OTD and update front end address in secondary site


8. Copy and update fmwconfig bootstrap artifacts to secondary location

9. Replace schema prefix in datasources

10. Convert snapshot to standby

11. Switchover Database (service downtime)

12. Verify SOACS in secondary site

13. Switchback to original site (service downtime)

If the primary SOACS site is running it may be the case that callbacks and instances remain pending to be executed

when the secondary site is configured. Follow the indications in the next sections to avoid duplications when using

this approach. As an alternative and for those cases where the standby DR site is being created from scratch (i.e.

the primary SOACS system is not live yet and DR is being implemented at the same time as the main production

system), it is recommended to configure DBaaS and Data Guard first and then provision SOACS in both sites. This

allows performing an initial Data Guard verification without altering the dependent SOACS services and also does

not use the databases on both sites concurrently during the set up. The first option (referred to as “OPTION 1” in this

document) incurs in less downtime during the set up. The following sections describe in detail the set up process.

The second option (“OPTION 2”) provides a tiered approach where the DB tier is set up and verified for DR first and

then the middle tier is added on top. This approach is described in Appendix A.

OPTION1 Pre-condition: primary SOACS System already exists

It is expected that the primary site has already been provisioned following the standard SOACS procedures:


Create an SSH key pair

Create an Oracle Storage Cloud Service Container

Create an Oracle Database Cloud Service instance

Run the SOACS provisioning wizard

It is also expected that the required VNC access has been configured to the VMs in the primary site. This is needed because an X11 session is used for installing Grid Infrastructure in the DBaaS compute. Some additional packages are required to run the installer in graphical mode7. Alternatively, the GI installation can be run in silent mode that does not require a graphical environment. Sample response files for a silent installation are provided in Appendix B - Setting up Data Guard.

Once the above steps have been followed and the SOACS instance is functioning, follow these steps:

1. Create virtual hostname for the frontend OTD and update frontend address in primary site

By default SOACS’s provisioning sets the value of the SOA cluster’s front end address to the IP of OTD’s listen

address. This IP needs to be replaced with a virtual hostname and this hostname needs to be resolvable by

both the external clients and by the WLS servers that OTD is routing requests to. To resolve the virtual

hostname externally a local file or DNS resolution may be used. To resolve the virtual hostname in the scope of

the SOACS instance, a local file resolution must be used.

A. To make the virtual hostname available in the SOACS nodes, follow these steps:

Login to the WebLogic Server Administration Console for your SOACS instance.

In the left pane, choose Environment in the Domain Structure window and then choose

Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.

Select HTTP

Note down the IP used in the Frontend Host filed.

NOTE: Alternatively you can get this IP also from the SOACS instance screen information.

This IP would be listed as the PUBLIC IP for the Load Balancer:

7 These yum commands can be used to add the graphical environment and the vnc server to the DBaaS machine:

yum groupinstall "Desktop Platform" -y

yum groupinstall "General Purpose Desktop" -y

yum install tigervnc-server

https://apexapps.oracle.com/pls/apex/f?p=44785:112:0::NO::P112_CONTENT_ID,P112_PREV_PAGE:10023

https://apexapps.oracle.com/pls/apex/f?p=44785:112:0::::P112_CONTENT_ID:13208


http://docs.oracle.com/cloud/latest/soacs_gs/CSBCS/GUID-51392B25-E93A-43C0-B232-96E66A9A7C08.htm#CSBCS-GUID-51392B25-E93A-43C0-B232-96E66A9A7C08

https://docs.oracle.com/cloud-machine/latest/jcs_gs/JSCUG/GUID-8F5B7EC4-38C8-4C0E-B111-5AB5FE967989.htm#JSCUG-GUID-8F5B7EC4-38C8-4C0E-B111-5AB5FE967989


In a command shell prompt for your SOACS nodes, sudo to the root user and edit

/etc/hosts to map this IP to a virtual hostname (if a DNS is going to be used for the

external resolution of this IP, this hostname will have to be used here. If you are going

to use file-based (/etc/hosts) hostname resolutions, this would be your own choice).

For example:

[root@soacsdr-jcs-wls-1~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

160.34.102.88 soalbrdr soalbrdr.example.com

Repeat these steps in the other SOACS server that are used by the SOACS Cluster

B. To make the virtual hostname available in the external clients either update your DNS server or

modify the hosts file for those clients as above. For example in your on-premises windows client,

edit your C:\Windows\System32\drivers\etc\hosts file as above

NOTE: This configuration for external clients will work if direct connections from the internet are used.

Connections from custom intranets will need to account for this hostname to be added to the required

proxy server used by the browsers or http clients

C. Once the appropriate hostname resolution is in place per the steps above, modify the front end

address in the SOACS Cluster:

Login to the WebLogic Server Administration Console for your SOACS instance.

In the left pane, choose Environment in the Domain Structure window and then choose

Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.

Select HTTP

Set the value for the Frontend Host=virtual_hostname_created, in the example above

this would be soalbrdr


Click Save.

To activate the changes, click Activate Changes in the Change Center section of the

Administration Console.

Restart the SOACS cluster for the front end change to be effective

2. Update t3/RMI urls (if used)

The urls used for RMI invocations in the SOACS cluster need to be agnostic to the IPs or hostnames used by each

site in the SOACS DR configuration. Instead of using the typical host:port,host:port JNDI urls, change them to use

the cluster syntax8. The cluster syntax is as follows: cluster:t3://cluster_name. For example, to modify the JMS

adapter factory properties to use this syntax, follow these steps:

A. Log into your Oracle WebLogic Server Administration Console for your SOACS instance

B. Click Deployments in the left pane for Domain Structure.

C. Click JmsAdapter under Summary of Deployments on the right pane.

D. Click the Configuration tab.

E. Click the Outbound Connection Pools tab and expand

oracle.tip.adapter.jms.IJmsConnectionFactory to see the configured connection factories.

F. Click the specific instance you are using (for example, eis/wls/Queue). The Outbound

Connection Properties for the connection factory opens.

G. Click Lock & Edit.

H. In the FactoryProperties field (click on the corresponding cell under Property value), alter the

java.naming.provider.url filed to use the cluster syntax (leave the rest of the fields as they were):

java.naming.provider.url= cluster:t3://cluster_name;

I. Click Save after you update the properties. The Save Deployment Plan page appears.

J. Enter a location for the deployment plan.

K. Copy the deployment plan from your SOACS node1 to your SOACS node2 in the exact same

directory/location

NOTE: since the ssh key for the system pertains the opc user, yuo need to copy the file as opc user to a tmp

Location in node2 where both opc and oracle users have access rights and then copy form that directory to eh

deployment plan home using the oracle user.

L. Click Save and Activate.

M. Click Deployments.

N. Click Lock & Edit

8 Obviously this is feasible only for inter-domain invocations. T3/rmi clients that are external to the SOACS domain will not be able to use this approach

and will have to either use a tcp load balancer for JNDi resolution or use the appropriate DNS mapping of the host:port list in the secondary site


O. Select the JMS Adapter.

P. Click Update.

Q. Select Update this application in place with new deployment plan changes (A deployment plan

must be specified for this option.) and select the deployment plan saved in a shared storage

location; all servers in the cluster must be able to access the plan.

R. Click Finish.

S. Activate the changes.

Similarly, any other custom JNDI urls used in the SOACS system should also be updated so that when a

switchover/failover occurs in the SOACSDR system, the urls are valid also in the secondary site.


3. Set up and verify Data Guard

The steps to set up Data Guard across two locations are generic for any DbaaS Database. Refer to Appendix B

for details9.

4. Provision SOACS in secondary location pointing to snapshot standby database

Once Data Guard has been configured the secondary SOACS system can be provisioned. The following are

key considerations for this set up:

1. In order to use the standby database as the container for SOACS schemas, the physical standby

needs to be converted to a snapshot standby.

2. Oracle SOACS provisions SOA schemas using a prefix that is specific to each cloud

services/instance. This means that in the initial provisioning the secondary location SOACS servers

will point to the same Database but will use different schemas. This is critical for running systems

because this will prevent the execution of composites/flows by the initial SOACS domain in the

secondary location. It is needed that only one site has active SOA servers pointing to an available

database at any point in time. Otherwise message and callback duplications could occur leading the

SOA system to inconsistencies.

Follow these steps to provision the secondary SOACS system:

A. Convert the physical standby database in the secondary site into a snapshot standby

IMPORTANT: Once the secondary location JDBC strings are updated to point to the same schemas as

production, the SOA servers in the secondary location will see the same data that the production ones were

seeing when the snapshot conversion occurred. If any SOA flows, callbacks etc. are pending, the servers in the

secondary location will try to complete those. Thus, it is important that instances are drained and completed on

the primary site before converting the standby database to snapshot or duplications could occur. Alternatively

the SOA servers in the primary location may be stopped and the database can be switched over to the

secondary location (incurs in higher downtime).

From the primary database host and as user oracle execute the following commands:

[oracle@soacsdb ~]$. ~/script.env

9 Apply fix for bug 24652400 to your specific database version or, as a temporary workaround, adjust the PDB Service to the standby domain after the

first switchover. To do this, connect to sqplus as sysdba on the standby and execute the following: update cdb_service$ set con_id#=3 where

name='pdb1.opcdomain’; where opc_domain is the value returned by “show parameter db_domain” in the standby database. For example:

SQL> show parameter db_domain

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

db_domain string esoracle25875.oraclecloud.inte

rnal

SQL> update cdb_service$ set con_id#=3 where name='pdb1.esoracle25875.oraclecloud.internal';

1 row updated.

SQL> commit;

Commit complete.


[oracle@soacsdb ~]$dgmgrl sys/${passwd}@${A_DBNM}

DGMGRL> CONVERT DATABASE orclb to SNAPSHOT STANDBY;

Converting database "orclb" to a Snapshot Standby database, please wait...

Database "orclb" converted successfully

B. Provision the SOACS instance using the SOACS provisioning wizard:

Follow the steps in the SOACS documentation to create the secondary site SOACS system. Use the EXACT

same name for the SOACS service that you are using in your primary location. The following table summarizes

the wizard options for the set up

SOACS Configuration Primary Site Secondary Site

Oracle Cloud Domain domaina domainb

Oracle Cloud User [email protected] [email protected]

Oracle Cloud Password Acme1234# Acme1234#

SOACS Service Type SOA Cluster SOA Cluster

SOACS Software Version 12.1.3 12.1.3

SOACS Instance Name soacsdr soacsdr

SOACS DB Configuration soacsdb soacsdbB

SOACS Cluster Size 2 2

SOACS Compute Shape OC2M OC2M

WebLogic User weblogic weblogic

WebLogic Password Acme1234# Acme1234#

Load Balancer Provisioning Yes Yes

Load Balancer Policy Least Connection Count Least Connection Count

Load Balancer Compute Shape OC3 OC3

SOACS Backup Container soacsContainer soacsBContainer

Oracle recommends that the exact same capacity and compute configuration is used on both primary and standby

locations for the ideal failover/switchover behavior (otherwise, the required request-throttling in OTD and sizing of

SOACS nodes needs to be done on the secondary location). Once the provisioning process completes the SOA

servers can be sanity verified.

5. Create virtual hostname for frontend OTD and update frontend address in secondary site

Follow the same steps as in section “Create virtual hostname for frontend OTD and update frontend address in

primary site” above using the public IP of the OTD instance in the secondary site to map the virtual host.

NOTE: Despite the hostname alias being the same in both sites, the required trust stores/certificates will have to be

recreated in the standby locations and the standby’s OTD/LBRS certificate will have to be imported in the

appropriate trust store if any SSL invocations are executed from the Cloud servers to the front end LBR. Refer to the

Enterprise Deployment Guide for for Oracle SOA Suite for the required steps

6. Update t3/rmi urls in the secondary site (if used)

Follow the same steps as in section “Update t3/rmi urls in the primary site” above using the same cluster name as in

primary.


http://docs.oracle.com/middleware/12212/lcm/SOEDG/GUID-361A8FF7-5BAD-4773-A9FB-3880BF1C1C8B.htm#GUID-7499718B-A1A1-4403-8051-1995885B3B5C



The FMW configuration in the secondary site needs to be updated so that both locations reuse the same policy and

JPS configuration. For this, it is required to copy the domain_name/config/fmwconfig artifacts from the primary

SOACS Administration servers to the secondary one and update the JPS JDBC url of the primary site with the

address that the WLS servers in the secondary site will use to access the DB. The entire domain does NOT need to

be copied (and, due to firewall and access restrictions across Cloud datacenters, cannot be copied). Follow these

steps:

A. From the primary location’s Admin Server host as user oracle execute the following:

[oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config

[oracle@soacsdr-jcs-wls-1~]$ tar -czvf /u01/data/fmwconfig-site1.gz ./fmwconfig/

[oracle@soacsdr-jcs-wls-1~]$ scp -i key_for_ssh.ppk /u01/data/fmwconfig-site1.gz

secondary_site_public_ip:/tmp/

It is expect that the key_for_ssh.ppk file generated as prescribed in the SOACS Documentation is available in both

locations. The key, can be copied itself from a third party node to the compute nodes themselves to enable ssh

between them.

B. In the secondary’s site, execute the following steps as the oracle user

a. Stop the Administration and Managed servers using the Weblogic Administration Console

b. Move the fmwconfig directory to a backup name

[oracle@soacsdr-jcs-wls-1~]$ mv /u01/data/domains/domain_name/config/fmwconfig/

/u01/data/domains/domain_name/config/fmwconfigbackup

c. Replace the fmwconfig directory with the copy from the primary site.

[oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config

[oracle@soacsdr-jcs-wls-1~]$ tar –xzvf /tmp/fmwconfig-site1.gz

d. Replace the identity domain and database name in the JDBC connect string used by OPSS:

[oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/fmwconfig

[oracle@soacsdr-jcs-wls-1~]$find ./*.xml | xargs sed –i

's/primary_database_node:1521\/PDB1.primary_domain_name.oraclecloud.internal/secondary_d

atabase_node:1521\/PDB1.secondary_domain_name.oraclecloud.internal/g'

Where primary_domain_name and secondary_domain_name are the cloud identity domains of each site. For

example:

[oracle@soacsdr-jcs-wls-1~]$find ./*.xml | xargs sed –i

's/soacsdb:1521\/PDB1.us6z12opcsoa01.oraclecloud.internal/soacdbB:1521\/PDB1.esoracle929

91.oraclecloud.internal/g'

8. Replace schema prefix in the standby datasources

As described in previous sections, each site will use a “local” database address in the JDBC connect string, however

it is required to update the schema names so that both sites use the same schema. Execute these steps to replace

the schema prefix in the secondary site.


https://docs.oracle.com/cloud/latest/jcs_gs/JSCUG/GUID-BD5190E0-5DE4-454E-BAD3-E889C575D432.htm#GUID-BD5190E0-5DE4-454E-BAD3-E889C575D432


A. Make a copy of your DataSource configuration. As the oracle user in the secondary Administration Server

node

[oracle@soacsdr-jcs-wls-1~]$ cp -R /u01/data/domains/domain_name/config/jdbc/

/u01/data/domains/domain_name/config/jdbcbackup

B. Identify the schema prefix in the primary site. As the oracle user in the primary Administration Server node

[oracle@soacsdr-jcs-wls-1~]$ cd /u01/data/domains/domain_name/config/jdbc/

[oracle@soacsdr-jcs-wls-1~]$ cat SOADataSource-jdbc.xml | grep _SOAINFRA

<value>SP266974844_SOAINFRA</value>

The prefix is the value preceding the _SOAINFRA string (in this example SP266974844)

C. Identify the schema prefix in the secondary site. As the oracle user in the secondary Administration Server

node


[oracle@soacsdr-jcs-wls-1~]$ cat SOADataSource-jdbc.xml | grep _SOAINFRA

<value>SP255674113_SOAINFRA</value>

The prefix is the value preceding the _SOAINFRA string (in this example SP255674113 on the standby)

D. Replace the initial schema prefix in the secondary site with the prefix of the primary site. As the oracle user

in the secondary Administration Server node


find /u01/data/domains/domain_name/config/jdbc -name '*.xml' | xargs sed -i

's/secondary_location_prefix/primary_location_prefix/g'

For example (using the values returned above):

find /u01/data/domains/domain_name/config/ *.xml | xargs sed -i 's/

SP255674113/SP266974844/g'

IMPORTANT: Up to this point, the SOA servers in the secondary location have been pointing to “empty” SOAINFRA

schemas with no composites deployed, no policies and no flows pending of execution. Once the secondary location

JDBC strings have been updated to point to the same schemas as production per the above steps, the SOA servers

in the secondary location will see the same data that the production ones were seeing. If any flows, callbacks etc

were pending to be executed; the secondary location servers will try to complete those at this point if started. Thus, it

is important that instances are drained and completed on the primary site before converting to snapshot the standby

database as already indicated above.

E. If DBFS is being used for sharing files across the SOACS node in a cluster, it is needed to recreate the

wallets and tns connect strings used for the local HA dbfs mounts12.

a. As the oracle user in the Admin Server node in the secondary location update the contents of

domain_location/dbfs/tnsnames.ora to match those of Dataguard set up. Make sure the

tnsnames.ora file points to the right name in standby

12 SOACS provides default DBFS mount points to share files across the members of the SOA cluster in the scope of a single Datacenter. These DBFS

mounts are used by components like MFT, B2B and the File Adapter.


cat /u01/data/domains/maa2_domainbackup/dbfs/tnsnames.ora

ORCLB =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = drDB2a)(PORT = 1521))

(CONNECT_DATA =

(SERVER = DEDICATED)

(SERVICE_NAME = PDB1.secondarydomain.oraclecloud.internal)

b. Re-create the wallet

[oracle@soacsdr-jcs-wls-1~]$mv /u01/data/domains/maa2_domain/dbfs/wallet/

/u01/data/domains/maa2_domain/dbfs/wallet_orig

[oracle@soacsdr-jcs-wls-1~]$unset ORACLE_HOME

[oracle@soacsdr-jcs-wls-1~]/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl

/u01/data/domains/maa2_domain/dbfs/wallet –create

[oracle@soacsdr-jcs-wls-1~]/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl

/u01/data/domains/maa2_domain/dbfs/wallet –createCredential ORCLB

primary_location_prefix_DBFS Cloud_Schema_Password13

c. Remount the DBFS directories witht the wallet

[oracle@soacsdr-jcs-wls-1~]fusermount –u /u01/soacs/dbfs_directio

[oracle@soacsdr-jcs-wls-1~]fusermount –u /u01/soacs/dbfs

[oracle@soacsdr-jcs-wls-1~]export ORACLE_HOME=/u01/app/oracle/middleware/dbclient/

[oracle@soacsdr-jcs-wls-1~]$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCLB -o direct_io

/u01/soacs/dbfs_directio

[oracle@soacsdr-jcs-wls-1~]$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCLB /u01/soacs/dbfs

d. Update dbfsMount.sh script

The script dbfsMount.sh, located in domain_location/dbfs, is used to mount the soacs dbfs file

systems when the servers start. The alias used by this script must be the same that has been

configured in the domain_location/dbfs /tnsnames.ora pointing to the standby. Originally, this script

uses these commands:

$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL -o direct_io $MOUNT_PATH_DIRECTIO

&>dbfs.log &

$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCL $MOUNT_PATH &>dbfs.log &

Edit the script and update it accordingly:

$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCLB -o direct_io $MOUNT_PATH_DIRECTIO

&>dbfs.log &

$ORACLE_HOME/bin/dbfs_client -o wallet /@ORCLB $MOUNT_PATH &>dbfs.log &

e. Repeat the steps from a to d in the other SOACS server in the secondary location to update the

content of domain_location/dbfs

9. Convert the snapshot standby database back to the physical standby database

Execute these steps as oracle user in the primary Database host:

13 This is the password provided for the SOACS service creation



[oracle@soacsdb ~]$. dgmgrl sys/${passwd}@${A_DBNM}

DGMGRL> CONVERT DATABASE orclb to PHYSICAL STANDBY;

Converting database "orclb" to a Physical Standby database, please wait...

Oracle Clusterware is restarting database "orclb" ...

Continuing to convert database "orclb" ...


NOTE: Give some time for the redo-apply to catch up on standby before continuing with the

next steps and make sure the configuration is correct in dgmgrl

10. Switchover Database




DGMGRL> switchover to orclb

Performing switchover NOW, please wait...

Operation requires a connection to instance "ORCL" on database "orclb"

Connecting to instance "ORCL"...

Connected as SYSDBA.

New primary database "orclb" is opening...

Oracle Clusterware is restarting database "orcl" ...

Switchover succeeded, new primary is "orclb"


At this point the SOA servers in both sites are pointing to the same schemas and the database is open in the

secondary location. Start the servers and verify composite deployments and composite instances (if any) in the new

active site

a. Start the Administration and Managed servers in the secondary site

a. Logon to the Enterprise Manager FMW Control Console for the secondary SOACS instance and

verify the soa-infra systems in both servers in the cluster. It is a good practice to perform an

endpoint test to confirm the correct behavior of the system

12. Switchback to original site.

As an optional step and to revert the system back to its original state, switchback the database and restart SOA

servers in the primary location.


http://www.oracle.com/webfolder/technetwork/tutorials/obe/cloud/sscs/DeployingComposite_Cloud/soa-comp-cloud-deployment.html


SOACS DR Lifecycle Procedures

Switchover

In a switchover operation an administrator reverts the roles of the two sites in a planned operation. The roles change

from the primary to the standby as well as from standby to primary. To perform a switchover in a SOACS DR

configuration follow these steps:

a. Stop the managed servers in the SOACS primary site.

Use the SOACS documentation to stop the Administration and Managed servers in the primary site

b. If configuration changes have been applied recently to the primary SOA/Weblogic domain, make

sure to propagate those to the standby location (Refer to section “Applying domain configuration

changes to the system” bellow)

c. Perform the required DNS changes to point consumers to the new primary OTD

Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host

resolution to point the front end address of the system to the public IP used by OTD in site2

d. Perform a database switchover using Data Guard Broker

From the primary site’s DBaaS node and as oracle user:


[oracle@soacsdb ~]$ dgmgrl sys/${passwd}@${A_DBNM}









e. Start Managed servers in the new SOACS primary site

Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site. (the

Administration server and Node Manager can be kept up on standby)

Failover

In a failover operation, the primary site becomes unavailable and an administrator fails over the Database and starts

managed servers in the secondary site. You can role-transition a standby database to a primary database when the

original primary database fails and there is no possibility of recovering the primary database in a timely manner. This

is known as a manual failover. There may or may not be data loss depending upon whether your primary and target

standby databases were transactionally consistent at the time of the primary database failure. To perform a

switchover in a SOACS DR configuration follow these steps:

a. Perform the required DNS changes to point consumers to the new primary OTD

Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host

resolution to point the front end address of the system to the public IP used by OTD in Site2




b. Perform a database failover using Data Guard Broker. Start Data Guard broker on the secondary

DBaaS node. As oracle user execute these steps:

[oracle@soacsdbB ~]$. ~/script.env

[oracle@soacsdbB ~]$ dgmgrl sys/${passwd}@${A_DBNM}

DGMGRL> failover to orclb

Performing failover NOW, please wait...

Failover succeeded, new primary is "orclb"

c. Start Managed servers in the new SOACS primary site

Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site

Applying domain configuration changes to the system

The fact that file system synchronization across Cloud datacenters is not allowed, precludes WLS domain config

changes from being automatically propagated to the secondary site (as it would occur in standard on-premise Active

Passive DR deployments). Two main approaches can be used to maintain the same configuration (ear

deployments, WLS domain configuration, deployment plans etc) in both locations. The applicability of each depends

on how frequently this “file-system-resident” configuration is modified.

a) For those SOACS cases where the domain configuration is infrequently altered (notice that composite

deployments, domain and WSM policies and MDS updates do not fall into this category as they are

stored in the Database) it is recommended to simply apply the configuration changes manually twice,

once in production and once in standby.

b) For those SOACS cases where the file system configuration is modified regularly, Oracle Database

File System (DBFS) can be used to synchronize the configuration using Dataguard (DBFS provides a

file system view of data that is stored in the Database). Using DBFS for configuration replication

has implications from the set up, disk space allocation and lifecycle perspective and oracle

recommends using it only when it is strictly necessary to replicate configuration changes very

frequently. There are other alternatives to DBFS such as direct use of rsync across sites, but those

present other risks including lack of transactional control in the copy and possible corruption of the

domain structure in the event of a failure during the copy operations.

Both approaches are described below.

a) Repeating domain configuration changes in both sites

To maintain the file system configuration synchronized by repeating the config change in both sites, follow these

steps:

A. Apply the configuration change normally in the primary site

Use the WLS Administration Console in the primary location to apply the configuration change. Activate the

change, restart the required SOACS servers if needed and verify that the change is working as expected.

B. Convert the standby database to a snapshot standby14

14 Changes to a reduced number of configuration artifacts in SOA and OSB may require the servers to be up in order to be applied, in these cases a

switchover and a start of the managed servers will be needed. Refer to the specific product documentation to identify these artifacts.











C. Start (if it wasn’t started) the Administration Server on the secondary site

Follow the steps in the Oracle Cloud documentation to start the administration server. It is important that ONLY

the administration server and not the managed servers is started on the secondary location. The reason being

that started SOA managed could resume/recover any pending flows, callbacks or and lead to duplicates if the

primary site remains up also.

D. Repeat the configuration change in the secondary site

Use the WLS Administration Console in the primary location to apply the configuration change. Activate the

change, restart the required SOACS servers if needed and verify that the change is working as expected. This

change should not alter any of the configurations in the database, only WLS domain configuration or application

deployments (not ear deployments). Any modifications in the database will be overwritten by the primary when

the Db is reverted to physical standby in the next step.

E. Revert the database to physical standby







b) Using DBFS for configuration propagation

Database File System (DBFS) uses database features to store files and manage relational data and expose them as

a standard file system that can be accessed by any operating system. The Oracle Database File System (DBFS)

creates a standard file system interface on top of files and directories that are stored in database tables. DBFS is

similar to NFS in that it provides a shared network file system that looks like a local file system. Like NFS, both a

server component and a client component are required to run DBFS.

Given the restrictions in file replication across Oracle Cloud data centers, DBFS can be used with Oracle Data

Guard to copy files from a primary site to a secondary site. When the system’s lifecycle involves frequents updates

to the domain file system, DBFS can be used to replicate the WLS domain configuration across sites. However, the

SOACS domain Configuration cannot reside directly on the DBFS mount because that would make the middle tier

dependent on the DBFS infrastructure in order to come up (the dependency is not only on the database but also on

FUSE libraries, mount points etc). Note also that the SOACS Weblogic domain configuration cannot be copied “as

is” in this design since each site in the SOACS DR solution contains references to the Cloud identity domain and the

local DB service in the JDBC connect strings. The configuration has to be modified after it is copied to each site. In

this approach, a directory with a copy of the primary site’s domain configuration is kept up to date with Data Guard in

the standby location. This directory is not available to the standby site unless the DB is open in read-only mode (the



default with the steps provided in this paper), a conversion to a snapshot standby is performed,or a switchover or

failover is executed.The conversion to snapshot will allow mounting the DBFS file system for writes also, but has the

caveat that it will not preclude SOA servers in the secondary location from starting (either accidentally, or by a

reboot) and may cause duplications and re-executions of work already processed by the primary Location. Oracle

recommends, however that a complete start and test of the servers in the secondary location is performed on a

regular basis to verify that the configuration and deployments are being replicated properly and that the Cloud

identity Domain and JDBC connect strings are being replaced correctly. This can be done by doing a switchover or

by converting the secondary database to snapshot standby. In this last cases, Oracle recommends to “drain” first the

Primary Database from any SOA work pending (this includes pending async invocations and JMS messages) to

avoid duplicated executions. Draining may imply blocking new incoming requests and waiting for all pending

messages to be processed/completed.

For application deployment operations, Oracle recommended using the Weblogic deployment “Upload your files”

option in the WebLogic Administration Console so that the deployed files are placed under the upload directory of

the Administration Server (under domain directory/servers/admin_server_name/upload). That way these files will be

copied by just including the domain dir in the DBFS copy scripts

In summary the use of DBFS for domain configuration synchronization requires the following steps:

1. Set up DBFS and create appropriate mount points

2. Create domain configuration replacement scripts

3. Enable domain copy (either by manual execution or by configuring cron jobs)15

The following subsections describe the steps in detail

Set up DBFS and create appropriate mount points

A client command-line interface named dbfs_client runs on each file system middle tier computer that has access

to the DBFS mount points. dbfs_client allows users to copy files in and out of the database from any host on the

network. DBFS is a part of Oracle database installation and requires fuse (FileSytem in Userspace) in the nodes that

access the mount point. It implements simple file system commands such as list and copy in a manner that is similar

to shell utilities. dbfs_client and fuse are already installed in SOACS 12.2.1.2 VMs. In these cases the “Install

the database client” and “Install FUSE” steps bellow can be skipped. The rest of the steps are applicable both to

12.1.3 and 12.2.1.2 SOACS. The DBFS mount points and client configuration created by default in SOACS 12.2.1.2

VMs are used for MFT, B2B and File Adapter cases (to share a common mount point between the members in the

SOA cluster). For performance and administration isolation purposes the dbfs mount point used for replicating the

domain across sites is configured and mounted separately from the default DBFS mount points under /u01/soacs).

This guarantees that configuration operations will not interfere with the system’s runtime behavior (for example,

thanks to this separation, it will be possible to unmount the DBFS configuration directory without affecting the

processing of files by the file adapter in a cluster). There will be then a “configuration” DBFS mount point under

/u02/dbfs (configured in the steps bellow) and a “runtime” DBFS mount point under /u01/soacs (automatically

created/configured with the SOACS system creation). Each one of them uses different tnsnames.ora and sqlnet.ora

configuration for the isolation reasons indicated above.

15 Changes applied to files in different nodes of the SOACS Cluster that do not reside under the domain/config directory, need to be manually synced to

those nodes. The DBFS approach described in this paper replicate the domain configuration to the Administration Server node only.


Install the Database Client

For SOACS 12.1.3 systems follow these steps to install the required binaries and libraries. It is expected that the

required VNC access has been configured in the different middle tier compute nodes. This will be required for

running the Oracle Installer. Download the oracle Client software to the SOACS machines in a readable-writable

directory by the oracle user (for example: /u01/install/). The Oracle Client software is available for download from

Oracle eDelivery16. It is possible to download the software directly to the SOACS instance using the browser in the

SOACS VM. As oracle user in each of the SOACS nodes (not needed in the OTD one):

Unzip the GI software:

[oracle@soacsdr-jcs-wls-1~]$ cd /u01/install

[oracle@soacsdr-jcs-wls-1~]$ unzip V46097-01.zip

Start the installer (remember to allow xhost + for VNC sessions started as opc user):

[oracle@soacsdr-jcs-wls-1~]$ cd client/

[oracle@soacsdr-jcs-wls-1~]$ runInstaller &

Select the following options during the installation:

Install Type: Administrator

Oracle Base: /u01/app/oracle

Software Location: /u01/app/oracle/product/12.1.0/client_1

Inventory Location: /u01/app/oracle/client_oraInventory

Ignore Swap space precheck

Install FUSE:

As opc user in each of the SOACS nodes (not needed in the OTD one)

[opc@soacsdr-jcs-wls-1~]$ sudo usermod -a -G fuse oracle

[opc@soacsdr-jcs-wls-1~]$ sudo yum install fuse-devel -y

Loaded plugins: refresh-packagekit, security

Setting up Install Process

…

Complete!

Create the required tablespaces and database user in the DBaaS node.

As oracle user in the primary DBaaS node (the required datafiles will be propagated to standby automatically

with DG17)


[oracle@soacsdb ~]$ sqlplus -s sys/${passwd}@${A_DBNM} as sysdba <<EOF

alter session set container=pdb1;

create tablespace tbsdbfs datafile '/u02/app/oracle/oradata/tbsdbfs01.dbf' size 1G

autoextend on next 100m;

create user soadbfs identified by soadbfs default tablespace tbsdbfs quota unlimited on

tbsdbfs CONTAINER=CURRENT;

grant connect, create table, create procedure, dbfs_role to soadbfs;

16 Some locations may require appropriate SSO hostname resolution for login into eDelivery. Adding 209.17.4.8 login.oracle.com to the compute

nodes’ /etc/hosts fixes this in most cases. 17 For most cases 1GB should suffice to host the domain configuration excluding log files. This tablespace, however, may need to be enlarged

depending on whether tmp, logs etc will be included also in the replication

https://docs.oracle.com/cloud-machine/latest/jcs_gs/JSCUG/GUID-8F5B7EC4-38C8-4C0E-B111-5AB5FE967989.htm#JSCUG-GUID-8F5B7EC4-38C8-4C0E-B111-5AB5FE967989

https://edelivery.oracle.com/


EOF

Create a filesystem in DBFS.

As oracle user in the primary DBaaS node:



alter session set container=pdb1;

connect soadbfs/soadbfs@pdb1;

@$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem.sql tbsdbfs dbfs;

EOF

Add Oracle Wallet for the dbfs client access in the middle tier:

A DBFS store is mounted using the dbfs_client program. The dbfs_client program does not return until the file system is unmounted. For the most secure method of specifying a password in the mount operation and also for

running the dfbs_client in the background and oracle Wallet is required. To create the required wallet, follow

these steps:

a) Create a directory for the wallet. For example:

[oracle@soacsdb ~]$ mkdir –p $ORACLE_HOME_DBFS18/oracle/wallet

b) Create an auto-login wallet.

/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl

$ORACLE_HOME_DBFS/oracle/wallet -create

c) Add the wallet location in the client's sqlnet.ora file and set wallet override:

[oracle@soacsdb ~]$ cat >> $ORACLE_HOME_DBFS/network/admin/sqlnet.ora <<EOF

WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY =

$ORACLE_HOME_DBFS/oracle/wallet) ) )

SQLNET.WALLET_OVERRIDE = TRUE

EOF

d) Create credentials:

/u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl wallet_location -

createCredential db_connect_string username password

For example:

[oracle@soacsdb ~]$ /u01/app/oracle/middleware/oracle_common/bin/mkstore -wrl

$ORACLE_HOME_DBFS/oracle/wallet/ -createCredential PDB1 soadbfs soadbfs

Copy the tnsnames.ora file from the DB node to the middle tier

18 ORACLE_HOME_DBFS refers to the dbfs_client installation directory in the middle tier. For 12.2.1.2 SOACS this exists under

/u01/app/oracle/middleware/dbclient


Since the ssh is set up for the opc user, securely copy (scp) first the file to the tmp folder in the middle tier node and then move it to the middle tier database client ORACLE_HOME

From the DbaaS node

[oracle@soacsddb ~]scp -i ssh_key_file.ppk $ORACLE_HOME/network/admin/tnsnames.ora

opc@maa-jcs-wls-1:/tmp/

From the middle tier node

[oracle@soacsdr-jcs-wls-1~] cp /tmp/tnsnames.ora $ORACLE_HOME_DBFS/network/admin/

Create a mount point in the SOACS administration node

As root in the SOACS node

[root@soacsdr-jcs-wls-1~] mkdir /u02

[root@soacsdr-jcs-wls-1~] chown oracle:oracle /u02

Update the oracle users shell environment for DBFS

As the oracle user in the Administration Server SOACS node:

[oracle@soacsdr-jcs-wls-1~] cat >> $HOME/.bash_profile <<EOF

export ORACLE_HOME=$ORACLE_HOME_DBFS

export LD_LIBRARY_PATH=$ORACLE_HOME_DBFS/lib:$LD_LIBRARY_PATH

export PATH=$PATH:$ORACLE_HOME/bin

EOF

[oracle@soacsdr-jcs-wls-1~]. $HOME/.bash_profile

By following the instructions above, the DBFS file system should be ready for mounting using the dbfs_client command. Additionally, the file system should be added to fstab for mounting and included in the node’s boot operations. To add the file system to fstab, follow these steps:

Update the system for DBFS

a) As root user, change the user and group of dbfs_client to user root and group fuse.

[root@soacsdr-jcs-wls-1~] chown root:fuse $ORACLE_HOME_DBFS/bin/dbfs_client

b) Set the setuid bit on dbfs_client and restrict execute privileges to the user and group only.

[root@soacsdr-jcs-wls-1~] chmod u+rwxs,g+rx-w,o-rwx $ORACLE_HOME_DBFS/bin/dbfs_client

c) Create a symbolic link to dbfs_client in /sbin as "mount.dbfs".

[root@soacsdr-jcs-wls-1~] ln -s $ORACLE_HOME_DBFS/bin/dbfs_client /sbin/mount.dbfs

d) Add the root user that is running the DBFS Client to the fuse group.

[root@soacsdr-jcs-wls-1~] usermod -a -G fuse root


e) Include the user_allow_other parameter in the fuse configuration

[root@soacsdr-jcs-wls-1~] echo “user_allow_other” >> /etc/fuse.conf

f) Add the required libraries to the user environment

[root@soacsdr-jcs-wls-1~] echo “$ORACLE_HOME_DBFS/lib” >>

/etc/ld.so.conf.d/usr_local_lib.conf

g) Add the required libs also to the root user’s shell

[root@soacsdr-jcs-wls-1~] cat >> /root/.bash_profile <<EOF

export ORACLE_HOME=$ORACLE_HOME_DBFS

export LD_LIBRARY_PATH=$ORACLE_HOME_DBFS/lib:$LD_LIBRARY_PATH

EOF

h) Reload the dynamic loadable library cache:

[root@soacsdr-jcs-wls-1~] ldconfig

i) Add the required DBFS line to /etc/fstab:

[root@soacsdr-jcs-wls-1~] cat >> /etc/fstab<<EOF

/sbin/mount.dbfs#/@PDB1 /u02 fuse rw,allow_other,noauto 0

EOF

NOTE: Notice the allow_other flag which will allow other users besides root to access the filesystem

j) The root user can mount the DBFS file system using the standard Linux mount command. For example:

[root@soacsdr-jcs-wls-1~] mount /u02 &

Add the required cron to mount DBFS when the node reboots

FUSE does not currently support automount. In order to mount the DBFS point when the middle tier node

reboots, follow these steps:

a) Create a cron job for reboot.

[root@soacsdr-jcs-wls-1~] cat >>dbfsmountcron.sh << EOF

echo "@reboot /root/mount_dbfs.sh" > mycron

crontab mycron

rm mycron

EOF

b) Enable the mount on reboot cron job


[root@soacsdr-jcs-wls-1~]chmod +x ./dbfsmountcron.sh

[root@soacsdr-jcs-wls-1~]./dbfsmountcron.sh

c) Create the mount script /root/mount_dbfs.sh with the following contents:

[root@soacsdr-jcs-wls-1~]cat >> /root/mount_dbfs.sh <<EOF

#!/bin/bash

export MOUNT_PATH=/u02

mkdir -p \$MOUNT_PATH

if mountpoint -q \$MOUNT_PATH ; then

echo "DBFS is already mounted"

exit

fi

if ! grep -q "^fuse\b.*\b\${USER}\b" /etc/group ; then

echo "\${USER} user not a member of the fuse group"

exit

fi

#Include a latency of 1 minute minimum for network and system to be ready for the DB

connnection

sleep 60

. /root/.bash_profile

mount \$MOUNT_PATH &

EOF

[root@soacsdr-jcs-wls-1~] chmod 755 /root/mount_dbfs.sh

With this, the script should execute every time the VM reboots and enable the DBFS mount. All the preceding steps, except those involving updates in the DB which are propagated with DataGuard, need to be equally performed on the standby Weblogic Administration Server node (the DBFS mount points will be available in read only mode by default using the steps in the DataGuard set up section)

Create domain configuration copy and replacement script. Optionally enable cron jobs

The domain configuration copy and replacement script is intended to copy (using rsync) the Weblogic domain configuration to the DBFS stage directory in the primary location and to copy (using rsync) the stage directory contents to the Weblogic domain directory in the secondary/standby system. It also “maps” (replaces) the Datasources’ connect strings and Cloud identity domain names in each site. The script is used on both sites so that either one can assume primary role and replicate configuration to the other. It can be executed manually in a controlled fashion or automatically at regular intervals using cron jobs. Whether the manual or “croned” approach applies better to each case depends on the frequency and size of the configuration changes applied. The frequency recommended will vary depending on each system’s lifecycle (how frequently new deployments and configuration changes are applied). When the configuration copy is “croned” and the changes do not involve large files, then rsync and Data Guard will provide a quick propagation that will transfer everything in a small lapse of time from the primary to the secondary location. However if the rsync copy takes a long time (because some files are very large or because many files are involved) it could occur that only a part of the configuration changes get “applied” to the primary DBFS stage directory at the point in time where the standby cron moves things to the secondary domain location. If the primary node crashed at that point in time, the standby domain config could be invalid/inconsistent. This scenario can be avoided with different approaches: 1)Using the --delay-updates will make the rsync copy operation more “atomic” (this is achieved by creating a temporary file for each updated file into a holding directory until the end of the transfer, at which time all the files are renamed into place in rapid succession) 2)Using rsync to copy to another intermediate folder from which a tar file is created, copying this tar file to the DBFS location and then extracting it in standby. Both approaches incur in additional space requirement. Their applicability will depend on the type of configuration changes and deployment frequency. The following script provides an example using the default approach for “regular” file copies. The delay-update option is commented out in the script and can be enabled in those scenarios where the copy operation will involve long periods of time due the domain’s size. The script can be extended to use also the tar approach with just a couple additions. The script requires a sudoers update (it uses the


oracle user to mount the dbfs point in case it is not up already). For this and as root in both primary and standby, the following line needs to be added to the /etc/sudoers file:

oracle ALL=NOPASSWD:/bin/mount,/bin/umount,/root/mount_dbfs.sh

Additionally, the script requires access from each middle tier site to the remote Database to retrieve status information. The following updates need to be applied in the security configuration of the DbaaS nodes. In the primary location, update the Security IP list with the standby middle tier’s public IP

Type Name Option Value

Security IP List soacsdbB IP List Add the secondary SOACS

Administration Server’s node

Public IP

Add the primary SOACS


Public IP

In the secondary location, update the Security IP list with the primary middle tier’s IP


Security IP List soacsdb IP List Add the primary SOACS


Public IP

Add the secondary SOACS


Public IP


The sample script can be downloaded from OTN also. Modify the variables in its initial section to adjust it to each environment/service

#!/bin/bash

#dbfscopy.sh

### Description: File to copy WLS configuration using DBFS and Dataguard across data centers

### Detects if middletier is pointing to primary or standby database.

### If pointing to primary, copies WLS domain configuration from the WLS domain directory to the

### dbfs mount point.

### If pointing to secondary, converts DB to snapshot, copies WLS domain configuration from dbfs

### mountpoint to WLS domain directory, replaces Cloud identity domain information for standby

### and reverts database to standby.

#The following provides an example of the environment variables settings in the primary location based

on the provisioned example used in previous sections:

#export WLS_DOMAIN_NAME=soacsdr-jcs_domain

#export ADMIN_SERVER_NAME=soacsdr-jcs__adminserver

#export REMOTE_IDENTITY_DOMAIN=domaina

#export LOCAL_IDENTITY_DOMAIN=domainb

#export LOCAL_DB_NODE=soacsDB

#export REMOTE_DB_NODE=soacsDBb

#export DBFS_MOUNT_PATH=/u02

#export DBFS_FILE_SYSTEM=/u02/soadbfs

#export SYS_USERNAME=sys

#export SYS_USER_PASSWORD=Acme1234#

#export TNS_IDENTIFIER=PDB1

#export DB_TNS_IDENTIFIER=orcl

#export REMOTE_DB_IDENTIFIER=orclb

#export ORACLE_HOME=/u01/app/oracle/tools/12.1.0/client_ADMIN/

#export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

#export PATH=$PATH:$ORACLE_HOME/bin

#The following provides an example of the environment variables settings in the secondary location

based on the provisioned example used in previous sections:

#export WLS_DOMAIN_NAME=soacsdr-jcs_domain

#export ADMIN_SERVER_NAME=soacsdr-jcs__adminserver

#export REMOTE_IDENTITY_DOMAIN=domainb

#export LOCAL_IDENTITY_DOMAIN=domaina

#export LOCAL_DB_NODE=soacsDBb

#export REMOTE_DB_NODE=soacsDB

#export DBFS_MOUNT_PATH=/u02

#export DBFS_FILE_SYSTEM=/u02/soadbfs

#export SYS_USERNAME=sys

#export SYS_USER_PASSWORD=Acme1234#

#export TNS_IDENTIFIER=PDB1

#export DB_TNS_IDENTIFIER=orclb

#export REMOTE_DB_IDENTIFIER=orcl

#export ORACLE_HOME=/u01/app/oracle/tools/12.1.0/client_ADMIN/

#export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

#export PATH=$PATH:$ORACLE_HOME/bin

export WLS_DOMAIN_NAME=your_weblogic_domain_domain

http://download.oracle.com/otn/other/dbfscopy.sh


export ADMIN_SERVER_NAME=your_weblogic_administration_server_name

export REMOTE_IDENTITY_DOMAIN=your_peer_cloud_identity_domain

export LOCAL_IDENTITY_DOMAIN=your_local_cloud_domain

export LOCAL_DB_NODE=your_local_db_node_name

export REMOTE_DB_NODE=your_peer_db_node_name

export DBFS_MOUNT_PATH=your_dbfs_mount_point

export DBFS_FILE_SYSTEM=path_to_your_dbfs_file_system

export SYS_USERNAME=your_DB_sys_user_name

export SYS_USER_PASSWORD=your_DB_sys_user_password

export TNS_IDENTIFIER=tns_names_for_dbfs_pdb

export DB_TNS_IDENTIFIER=tns_names_for_local_db

export REMOTE_TNS_IDENTIFIER=tns_names_for_remote_db

export ORACLE_HOME=your_client_db_installation_home

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

export PATH=$PATH:$ORACLE_HOME/bin

export db_type=$(

echo "set feed off

set pages 0

select database_role from v\$database;

exit

" | sqlplus -s $SYS_USERNAME/$SYS_USER_PASSWORD@$DB_TNS_IDENTIFIER "as sysdba"

)

convert_to_snapshot(){

echo "*********************************************CONVERSION TO

SNAPSHOT*********************************************"

echo "Converting to snapshot..."

#We connect dgmgr to remote DB since that would be primary

export conversion_result=$(

dgmgrl ${SYS_USERNAME}/${SYS_USER_PASSWORD}@${REMOTE_TNS_IDENTIFIER} "convert database

'${DB_TNS_IDENTIFIER}' to snapshot standby"

)

if [[ $conversion_result = *successful* ]]

then

echo "DB CONVERTED TO SNAPSHOT."

return 1

else

echo "DB CONVERSION FAILED. CHECK DATAGUARD STATUS."

return 0

fi

}

convert_to_standby(){

echo "Converting to standby..."

#We connect dgmgr to remote DB since that would be primar

export conversion_result=$(

dgmgrl ${SYS_USERNAME}/${SYS_USER_PASSWORD}@${REMOTE_TNS_IDENTIFIER} "convert database

'${DB_TNS_IDENTIFIER}' to physical standby"

)

if [[ $conversion_result = *successful* ]]


then

echo "DB CONVERTED TO STANDBY."

return 1

else

echo "DB CONVERSION FAILED. CHECK DATAGUARD STATUS."

return 0

fi

echo "*********************************************CONVERSION TO SNAPSHOT MODULE

COMPLETE*********************************************"

}

run_in_primary(){

echo "*********************************************SYNC IN

PRIMARY*********************************************"

echo "Rsyncing from domain dir to dbfs mount..."

#Optional backup of configuration

#cp -R $DBFS_FILE_SYSTEM/$WLS_DOMAIN $DBFS_FILE_SYSTEM/$WLS_DOMAIN_backup"\$(date '+%d-%m-%Y-

%H-%M-%S')"

#Sync to DBFS mount

#This approach skips the managed server domain directories. It reduces the footprint of the

remote sync

#but it also requires additional selected syncs (included belllow) and the recreation of the

security/encrupted password

#rsync -avz --exclude 'servers' --exclude 'tmp' /u01/data/domains/$WLS_DOMAIN_NAME/

$DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME

#This approach includes all contents (including logs) which is easier but causes more overhead

rsync -avz --exclude 'dbfs' --exclude 'servers/*/logs' --exclude

'servers/*/data/nodemanager/*.lck' --exclude 'servers/*/data/nodemanager/*.pid' --exclude

'servers/*/data/nodemanager/*.state' --exclude 'tmp' /u01/data/domains/$WLS_DOMAIN_NAME/

$DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME

echo "RSYNCING COMPLETE."

echo $(date '+%d-%m-%Y-%H-%M-%S') > $DBFS_FILE_SYSTEM/last_primary_update.log

#The following 2 steps are only required if the servers directory is excldued

#1.-We sync upload dir in Admin Server for deployments to be copied over.

#echo "ALSO COPYING DEPLOYMENTS..."

#mkdir -p $DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload

#rsync -avz --delete /u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload/

$DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload

#echo "DEPLOYMENTS COPY COMPLETE"

#2.-We sync Admin Server boot.properties to avoid having to alter the local NM and other local

encrypted files.

#echo "COPYING BOOT PROPERTIES FOR ADMIN..."

#rsync -avz /u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/security/

$DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/security/


#rsync -avz --delete /u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload/

$DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload

#echo "BOOT PROPERTIES COPY COMPLETE"

echo "*********************************************SYNC IN PRIMARY MODULE

COMPLETE*********************************************"

}

run_in_secondary(){

echo "*********************************************SYNC IN

SECONDARY*********************************************"

#Optional backup of configuration

#mkdir -p /u01/data/domains/backup

#cp -R /u01/data/domains/$WLS_DOMAIN /u01/data/domains/backup_"$(date '+%d-%m-%Y-%H-%M-%S')"

#Sync from DBFS mount

echo "Rsyncing from dbfs mount to domain dir..."

#This approach skips the managed server domain directories. It reduces the footprint of the

remote sync

#but it also requires additional selected syncs (included belllow) and the recreation of the

security/encrupted password

#rsync -avz --exclude 'servers' --exclude 'tmp' $DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/

/u01/data/domains/$WLS_DOMAIN_NAME/

#This approach includes all contents (including logs) which is easier but causes more overhead

rsync -avz --exclude 'tmp' $DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/

/u01/data/domains/$WLS_DOMAIN_NAME/

echo "RSYNCING COMPLETE."

echo $(date '+%d-%m-%Y-%H-%M-%S') >

/u01/data/domains/$WLS_DOMAIN_NAME/last_secondary_update.log

#The following 2 steps are only required if the servers directory is excldued

#1.-We sync upload dir in Admin Server for deployments to be copied over.

#echo "ALSO COPYING DEPLOYMENTS..."

#mkdir -p /u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload

#rsync -avz --delete $DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload/

/u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/upload/

#echo "DEPLOYMENTS COPY COMPLETE."

#2.-We sync Admin Server boot.properties to avoid having to alter the local NM and other

local encryptions. Alternatively we could copy the

#echo "COPYING BOOT PROPERTIES FOR ADMIN..."

#rsync -avz $DBFS_FILE_SYSTEM/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/security/

/u01/data/domains/$WLS_DOMAIN_NAME/servers/$ADMIN_SERVER_NAME/security/

#echo "BOOT PROPERTIES COPY COMPLETE"

#Replace Cloud Domain and Node

echo "Replacing cloud identity domain in DB connect strings..."

#Replacing the cloud identity domain in the logs is an overkil and it may be good to keep

domain references for roubleshooting

#The startup.properties for each managed server will also contain a refernece tot he current

domain, but it will be recreated when servers are started


cd /u01/data/domains/$WLS_DOMAIN_NAME/config/

find . -name '*.xml' | xargs sed -i "s/$REMOTE_DB_NODE/$LOCAL_DB_NODE/g"

find . -name '*.xml' | xargs sed -i "s/$REMOTE_IDENTITY_DOMAIN/$LOCAL_IDENTITY_DOMAIN/g"

rm /u01/data/domains/$WLS_DOMAIN_NAME/servers/*/data/nodemanager/startup.properties

echo "REPLACING COMPLETE."

#An optional restat of the Admin Server may be introduced but it will make sense ume the

changes. This will make sense or not

#depending on the frequency of the cron job and whether we wnat to also do snapshot ops or

other configuration tests while config is replicated

echo "*********************************************SYNC IN SECONDARY

COMPLETED*********************************************"

}

check_and_retry_mount(){

echo "*********************************************CHECK AND RETRY MOUNT POINT IF

NEEDED*********************************************"

#Check DBFS mount is available

if mountpoint -q $DBFS_MOUNT_PATH; then

echo "MOUNT AT $DBFS_MOUNT_PATH IS READY."

return 1

else

echo "DBFS Mount point not available. Will try to mount again..."

sudo umount $DBFS_MOUNT_PATH

sudo /root/mount_dbfs.sh

sleep 10

if mountpoint -q $DBFS_MOUNT_PATH; then

echo "MOUNT AT $DBFS_MOUNT_PATH IS READY."

return 1

else

echo "DBFS Mount point not available even after another try to mount. Check

your DBFS set up. If the DB does not allow read-only mode, this is expected"

return 0

fi

fi

echo "*********************************************CHECK AND RETRY MOUNT POINT IF NEEDED

COMPLETED*********************************************"

}

echo "********************************************STARTING SYCHRONIZATION OF WLS DOMAIN CONFIGURATION

TRHOUGH DBFS**********************************************"

if [[ $db_type = *PRIMARY* ]]

then

check_and_retry_mount

check_and_retry_mount_result=$?

if [ "$check_and_retry_mount_result" == 1 ]

then

echo "Copying data from domain directory to DBFS mount..."


run_in_primary

else

echo "Cannot copy domain configuration data to DBFS."

fi

elif [[ $db_type = *"PHYSICAL STANDBY"* ]]

then

#This step is not needed when using ADG since the standby DB would be in read-only mode.

convert_to_snapshot

snapshot_conversion_result=$?

if [ "$snapshot_conversion_result" == 1 ]

then


check_and_retry_mount_result=$?

if [ "$check_and_retry_mount_result" == 1 ]

then

echo "Copying data from DBFS mount to domain directory..."

run_in_secondary

else

echo "Cannot copy domain configuration data from DBFS."

fi

#We revert the DB to standby

#This step is not needed when using ADG since the standby DB would be in read-only mode.

convert_to_standby

standby_conversion_result=$?

if [ "$standby_conversion_result" == 1 ]

then

echo "Database back to standby mode. When not using ADG may want to umount $DBFS_MOUNT_PATH"

#Leave mount point in goodstatus if possible


else

echo "Failed to revert DB to standby. Check Dataguard Status."

fi

else

echo "Failed to convert DB to snapshot. Check Dataguard Status."

fi

else

echo "Either the Database is not a Physical Standby or we are unable to identify the DB's role."

echo $(date '+%d-%m-%Y-%H-%M-%S') > /u01/data/domains/$WLS_DOMAIN_NAME/last_failed_update.log

fi

It is recommended to execute the copy script manually first in the primary middle tier before creating a cron in both primary and standby to execute at regular intervals. The first copy of the domain may take some time. Once it completes, execute it also in the standby location to copy from DBFS to the WebLogic domain directory as an initial sync point. The script will synchronize the configuration between the Weblogic domain directory in the primary location’s Administration Server node and the WebLogic domain directory in the secondary location’s Administration Server node. The configuration under domain_home/config is automatically copied over to all other nodes that are part of the WebLogic domain when the servers residing in those nodes are started. Additionally, it is required to update the domain security information in those nodes with the one propagated from the primary. This update is needed only the first time the configuration is synchronized across sites. To copy this security information from the WebLogic Administration Server node to other WebLogic Managed server nodes that are part of the SOACS cluster follow these steps:


Once the dbfscopy script completes, create a tar in the secondary’s WebLogic Admin Server node and copy it over to the other managed server node(s):

[oracle@soacsdr-jcs-wls-1~] cd /u01/data/domains/domain_name/security

[oracle@soacsdr-jcs-wls-1~] tar -czvf security.gz *

[oracle@soacsdr-jcs-wls-1~] scp -i /ssh_key_file security.gz opc@soacsdr-jcs-wls-2:/tmp

In the managed server’s node, replace the security directory:

[oracle@soacsdr-jcs-wls-2~] cd /u01/data/domains/maa2_domain/security

[oracle@soacsdr-jcs-wls-2~] mv /u01/data/domains/maa2_domain/security/

/u01/data/domains/maa2_domain/securitybackup

[oracle@soacsdr-jcs-wls-2~] tar -xzvf /tmp/security.gz

Once this initial copy from primary to secondary is complete, the scripts can be added to the cron list in the system so that they get executed regularly. Notice that “croning” the copy script automates synchronization but also has the following implications:

1) Synchronization may incur in latency as high as the frequency of the cron jobs in both locations added up. i.e. if the cron jobs are set to execute every 30 minutes each, the changes may take 60 minutes to be available if the window in primary overlaps with the one on the secondary location. Before performing a switchover, make sure that this amount of time has passed by after the last configuration change. Otherwise, you could switchover before the change is present on standby and overwrite the changes originally applied with the role switch

2)The cron frequency should be set at minimum to the largest amount of time a deployment or configuration change may take to be copied from the domain directory to the dbfs stage directory. Otherwise, copy jobs may overlap

3)Oracle recommends running copy jobs manually after each configuration change and in a controlled fashion and verify that artifacts are properly transferred to standby to avoid inconsistencies and data loss scenarios.

To synchronize the configuration between two sites, execute first the dbfscopy.sh script in the primary middle tier (SOACS VM in the primary location where the Weblogic Administration Server is running). Once completed, execute the same script in the secondary middle tier (SOACS VM in the secondary location where the WebLogic Administration Server is running). Once completed, the configuration and deployments in both locations should be the same. To make the configuration changes effective, the WebLogic Administration Server in the standby location needs to be restarted. Refer to the script’s comments for customizations and further details.

Scaling the SOACSDR system

Scale operations may be applied in the SOACSDR service. To scale up simply apply the compute nodes’ shape change both in primary and standby. To scale out follow these steps:

1. Using the SOA Cloud Service console, scale out first in primary

2. Using the SOA Cloud Service console, scale out in secondary

3. Using the dbfs copy script, synchronize the configuration from primary to secondary

4. Copy the security information from the Administration server to the new Managed Server node:

a. Create a tar in the secondary’s WebLogic Admin Server node and copy it over to the other managed server node(s):

[oracle@soacsdr-jcs-wls-1~] cd /u01/data/domains/domain_name/security

[oracle@soacsdr-jcs-wls-1~] tar -czvf security.gz *

[oracle@soacsdr-jcs-wls-1~] scp -i /ssh_key_file security.gz opc@soacsdr-jcs-wls-2:/tmp

b. In the managed server’s node, replace the security directory:

https://docs.oracle.com/en/cloud/paas/soa-cloud/csbcs/scaling-oracle-soa-suite-cloud-service-cluster.html

https://docs.oracle.com/en/cloud/paas/soa-cloud/csbcs/scaling-oracle-soa-suite-cloud-service-cluster.html


[oracle@soacsdr-jcs-wls-2~] cd /u01/data/domains/maa2_domain/security

[oracle@soacsdr-jcs-wls-2~] mv /u01/data/domains/maa2_domain/security/

/u01/data/domains/maa2_domain/securitybackup

[oracle@soacsdr-jcs-wls-2~] tar -xzvf /tmp/security.gz

5. It is recommended to do a switchover and verify the correct behavior of the new managed server. To scale in a SOACSDR system, use the same steps as for scaling out except step #4:

1. Using the SOA Cloud Service console, scale-in in primary

2. Using the SOA Cloud Service console, scale-in in secondary

3. Using the dbfs copy script, synchronize the configuration from primary to secondary

https://docs.oracle.com/en/cloud/paas/soa-cloud/csbcs/scaling-oracle-soa-cloud-service-cluster1.html

https://docs.oracle.com/en/cloud/paas/soa-cloud/csbcs/scaling-oracle-soa-cloud-service-cluster1.html


Conclusion

Disaster recovery in a SOA Cloud Service configuration consists of a production database and a standby database

synchronized by Oracle Data Guard. Two separate middle tier configurations, each pointing to their local database,

are created to minimize the file synchronization needs across data centers. With this Disaster Recovery solution,

Oracle Cloud eliminates the costs and complexity of owning and managing a standby hardware, software, and

remote data center – while achieving the best Recovery Time Objective and Recovery Point Objective.

The use of Oracle Data Guard for disaster recovery provides better RTO and RPO than restoring a remote backup;

production is quickly failed over to an already running and synchronized copy of your production database on the

Oracle Cloud. The standby database in the cloud not only provides disaster recovery, it can also be used to seed

clone databases for development and test.

The use of middle tiers with a streamlined configuration replication facilitates maintenance and reduces the

overhead caused by continuous configuration approaches. However, an appropriate methodology and regular

standby verifications are need to guarantee a consistent recovery. Depending on each system’s lifecycle, different

synchronization approaches may be used for optimum behavior.


Appendix A - OPTION2

Pre-condition: Primary SOACS System does not exist

In this approach Data Guard is set up and verified first and then similar steps as in OPTION1 are used to configure

virtual hostnames, update t3 urls, copy and update the opss configuration and replace the schema prefix in the

JDBC configuration. The most relevant aspect is that in this option instead of converting the physical standby to

snapshot (as in OPTION 1) a switchover is performed (since the system is not production yet, there is no downtime

impact). Follow these steps for this approach:

1. Provision DBaaS in both locations

2. Set up and verify DataGuard

3. Provision SOACS in primary location

4. Create virtual hostname for front end OTD and update front end address in primary site


6. Switchover Database

7. Provision SOACS in secondary location

8. Create virtual hostname for front end OTD and update front end address in secondary site



11. Replace schema prefix in datasources


13. Switchback to original site.

A. Provision DBaaS in both locations

Follow the standard procedures as described in the SOACS documentation for this in both locations:

Create an SSH key pair

Create an Oracle Storage Cloud Service Container

Create an Oracle Database Cloud Service instance

B. Set up and verify Data Guard

Follow the steps in Section “Set up and verify Data Guard” in OPTION 1 above.

C. Provision SOACS in primary location

Follow the standard procedures as described in the SOACS documentation for this. Keep track of the

service/instance name used as this will have to be the same in the secondary location:







D. Create virtual hostname for front end OTD and update front end address in primary site

Follow the steps in Section “Create virtual hostname for front end OTD and update front end address in primary site”

in OPTION 1 above.

E. Update t3/rmi urls (if used)

Follow the steps in Section “Update t3/rmi urls (if used)” in OPTION 1 above

F. Switchover Database

From the primary database host as user oracle execute the following commands:











G. Provision SOACS in secondary location

Follow the standard procedures as described in the SOACS documentation for this. Use the same service/instance

name as in the primary location:


H. Create virtual hostname for front end OTD and update front end address in secondary site

Follow the same steps as in section “Create virtual hostname for front end OTD and update front end address in

primary site” in OPTION1 above using the public IP of the OTD instance in the secondary site to map the virtual

host.

I. Update t3/rmi urls in the secondary site (if used)

Follow the same steps as in section “Update t3/rmi urls in the primary site” in OPTION1 above using the same

cluster name as in primary.

J. Copy and update fmwconfig bootstrap artifacts to secondary location

Follow the steps in Section “Copy and update fmwconfig bootstrap artifacts to secondary location” in OPTION 1

above.

K. Replace schema prefix in datasources

Follow the steps in Section “Replace schema prefix in datasources” in OPTION 1 above.

L. Verify SOACS in secondary site

Follow the steps in Section “Verify SOACS in secondary site” in OPTION 1 above.

M. Switchback to original site.

Follow the steps in Section “Switchback to original site” in OPTION 1 above.



Appendix B - Setting up Data Guard

The primary’s site Database is already provisioned following the SOACS set up process described in the

documentation. The following steps were performed to configure Data Guard:

A. Install Grid Infrastructure on the primary database:

The Oracle Grid Infrastructure (GI) has become an integral part of the application failover features for Oracle

Data Guard. In the single instance case the GI provides Oracle Restart functionality.

The Grid Infrastructure is not installed with an Oracle Database Cloud Service in a standalone configuration

and so it should be installed manually following the appropriate documentation for 12.1, 12.2 or 11.2 Note: The

steps followed in the case study assumed that the GI had been installed and configured. The Grid Infrastructure

(GI) is not an absolute requirement for DR to function correctly, however, the steps in this document expect that

it is installed. Alternate commands are available, but are not provided. GI is required for Fast Application

Failover (FAN) functionality.

Download the oracle Grid Infrastructure software to the DBaaS machine in a readable-

writable directory by the oracle user (for example /u02/install/). The Oracle Grid

infrastructure software is available for download from Oracle eDelivery. It is possible to

download the software directly to the DBCS instance using the browser in the DBaaS

VM19.

Unzip the GI software:

[oracle@soacsdb ~]$ cd /u02/install

[oracle@soacsdb ~]$ unzip linuxamd64_12102_grid_1of2.zip

[oracle@soacsdb ~]$ unzip linuxamd64_12102_grid_2of2.zip

Start the installer:20

[oracle@soacsdb ~]$ cd grid/

[oracle@soacdb ~]$./runInstaller &

Select the following options during the installation:

1. Install option: Install Oracle Grid Infrastructure Software Only

2. Language: English*

3. Oracle ASM Operator: dba

4. Oracle Base: /u01/app/oracle

19 Some locations may require appropriate SSO hostname resolution. Adding 209.17.4.8 login.oracle.com to the compute nodes’ /etc/hosts

addresses this in most cases.

20 Alternatively, you can run a silent installation that does not require any graphical environment. You can use these sample response files:

Sample Grid Infrastructure 12.1.0.2 response file for SOACS DR

Sample Grid Infrastructure 12.2.0.1 response file for SOACS DR

Unzip the file to extract the response file. Each file includes instructions to customice it and steps to execute the Grid silent installation.

http://www.oracle.com/technetwork/database/availability/grid-12102-simplified-4406581.zip




5. Software Installation Location: /u01/app/12.1.0/grid

During the installation, for certain prerequisite check failures, Fix & Check Again was

clicked to generate a fixup script (runfixup.sh). This fixup script was executed as the root

user to complete the required pre-installation steps.

» Zeroconf check was fixed by the fixup script

Also during the installation the following three warnings were ignored:

» Insufficient swap space

» NTP (not required for single instance) (not coming up in latest DbaaS 12.2.1.2

installs)

» Task resolv.conf Integrity.

» Not recommended path for oracle base (only in latest DbaaS 12.2.1.2 installs)

B. Configure Oracle Restart on the primary database:

As user oracle on the primary database host :

[oracle@soacsdb ~]$ lsnrctl stop listener

[oracle@soacsdb ~]$ cp $ORACLE_HOME/network/admin/listener.ora \

/u01/app/12.1.0/grid/network/admin/listener.ora

[oracle@soacsdb ~]$ srvctl add listener -listener listener –oraclehome \

/u01/app/12.1.0/grid

[oracle@soacsdb ~]$ srvctl start listener

The steps followed in the case study assume that the database listeners on all database servers are listening on

port 1521.

Configure the Database in Oracle Restart: As user oracle on the Primary site’s

database host (appdba):

[oracle@soacsdb ~]$ srvctl add database -d ORCL -oraclehome $ORACLE_HOME -startoption

open

[oracle@soacsdb ~]$ srvctl start database -d ORCL

[oracle@soacsdb]~$ srvctl status database -d ORCL

Database is running.

C. Set TCP Socket Maximum Sizes recommended for DG:

NOTE: As documented in the Oracle Public Cloud documentation, you can gain root access to a provisioned

cloud instance by connecting to the opc user and then using the sudo command:

[opc@soacsdb ~]$ sudo -s

[root@soacsdb ~]$ sysctl -w net.core.rmem_max=10485760

[root@soacsdb ~]$ sysctl -w net.core.wmem_max=10485760


The /etc/sysctl.conf file was also edited to reflect the changes so that they would be preserved during an instance

reboot.

D. Set the MTU Size for OracleNet21 traffic

[root@soacsdb ~]$ ip link set eth0 mtu 1500

[root@soacsdb ~]$ echo "ip link set eth0 mtu 1500" >> /etc/rc.local

E. Size the Online Redo Logs appropriately for you application: The redo logs created by default in

the Database Cloud Service were 1GB in size and these were sufficient for our testing.

Online redo logs should be sized by this formula, but should not be less than 1GB in size:

peak redo rate per minute x 20

Redo rates can be extracted from AWR reports during peak workload periods such as batch processing, quarter or

year end processing. It is very important to use peak workload and not averages (averages can obscure peak redo

rates and lead to provisioning redo logs that are too small).

F. Create Standby Redo Logs

The following commands were executed as user sys on the primary site’s database (orcl):

alter database add standby logfile thread 1 group 4

'/u04/app/oracle/redo/stbyredo04.log' size 1073742336;







select * from v$logfile;

The MAA best practice for Standby Redo Logs (SRLs) is to create the same number as there are groups of Online

Redo Logs (ORLs) plus 1. On RAC this must be done for each thread. Use the following query to discover how

many redo log groups you have in each thread.

select thread#, count(group#) from v$log group by thread#;

SRLs should be created the same size as the largest of the ORLs. The group numbers are shared with the ORLs

and so SRLs are created with higher group numbers. To gather the size of the largest ORLs and the current highest

group number execute the following query:

select max(bytes), max(group#) from v$log;

The MAA best practice is that SRLs are not duplexed.

21 This may be required only in some Cloud Datacenters but it won’t have side effects in those that don’t need it so it is recommended to run this step


G. Encrypt any custom tablespaces in the DB: The SOACS database schemas are already

encrypted by default. If new/custom tablespaces are added to the DB, oracle recommends

encrypting them also.

The Data Guard primary and standby databases are physical replicas of each other. If a standby database in the

cloud is to be encrypted at rest, then the primary must be encrypted first. Oracle MAA Best Practices provide

guidance for converting to TDE with minimal downtime for Oracle Database 11.222 or for Oracle Database 12c23.

H. Enable Force Logging, Flashback Database, and Set Recommended MAA Database

Parameters:

The following command was execute as user sys the primary’s site DB:

SQL> alter database force logging;

SQL> alter database flashback on;

SQL> alter system set DB_FLASHBACK_RETENTION_TARGET=120 scope=both sid='*';

SQL> alter system set remote_login_passwordfile='exclusive' scope=spfile sid='*';

SQL> alter system set DB_BLOCK_CHECKSUM=FULL;

SQL> alter system set DB_BLOCK_CHECKING=MEDIUM;

SQL> alter system set DB_LOST_WRITE_PROTECT=TYPICAL;

SQL> alter system set LOG_BUFFER=256M scope=spfile sid='*';

NOTE: DB_BLOCK_CHECKSUM, DB_BLOCK_CHECKING and DB_LOST_WRITE_PROTECT are parameters

that assist in preventing block corruption. The listed settings are recommended for the protection benefits though

DB_BLOCK_CHECKING can have an impact on performance that should be considered when choosing a value. A

value of FALSE turns DB_BLOCK_CHECKING off.

NOTE: Setting LOG_BUFFER to 256MB will aid asynchronous redo transport in reading redo from memory instead

of having to do disk I/Os from the online redo logs.

I. Enable ARCHIVELOG:

SQL> alter system set log_archive_dest_1=

'location=USE_DB_RECOVERY_FILE_DEST valid_for=(ALL_LOGFILES,ALL_ROLES)

DB_UNIQUE_NAME=orcl24' scope=both sid='*';

SQL> shutdown immediate

SQL> startup mount

SQL> alter database archivelog;

SQL> alter database open;

SQL> alter system archive log current;

J. Provision a DBaaS instance in the secondary site

Follow the steps in the SOA CS documentation to provision the required Database for the standby datacenter

1. Create an SSH key pair

22 http://www.oracle.com/technetwork/database/availability/tde-conversion-11g-2531187.pdf

23 http://www.oracle.com/technetwork/database/availability/tde-conversion-12c-2537297.pdf

24 This should be your ORACLE_UNQNAME value in the primary location

http://www.oracle.com/technetwork/database/availability/tde-conversion-11g-2531187.pdf

http://www.oracle.com/technetwork/database/availability/tde-conversion-12c-2537297.pdf



2. Create an Oracle Storage Cloud Service Container

3. Create an Oracle Database Cloud Service instance. The following

parameters were used in the standby Database creation

System Secondary site

DBaaS Configuration

Oracle Cloud Domain domainb

Oracle Cloud User [email protected]

Oracle Cloud Password Acme1234#

DBCS Software Version 12c R1

DBCS Software Edition Enterprise Edition High Performance

DBCS Service Name soacsdbB

DBCS Shape OC4

Database Name orcl25

Pluggable Database PDB1

DB Instance Name orcl26

DB Backup Container orclbContainer

Database System Password (all databases) Acme1234#

Usable Database Storage (GB) 25

Administration Password **********

Character Set AL32UTF8

National Character Set AL16UTF16

Database Clustering with RAC Unchecked

Standby Database with Data Guard Unchecked

Enable Oracle Goldengate Unchecked

Include “Demos” PDB Unchecked

Backup Destination Both Cloud Storage and Local Storage

Cloud Storage Container Storage-domainb/orclbContainer

Cloud Storage User Name [email protected]

Cloud Storage Password Acme1234#

K. Delete or Repurpose the Default Database

The default database created in the application database instance on the secondary site can be deleted (if not used

for other purposes) as it cannot be used as a Data Guard standby database27. The database can be deleted using

the following command:

[oracle@soacsdbB ~]$ dbca -silent -deleteDatabase -sourceDB ORCL -sysDBAUserName sys -

sysDBAPassword Acme1234#

25 The DbaaS Service name needs to be the same as in primary to keep scripts’ consistency

26 This Db instance name is not relevant for the set up as it cannot be used as a Data Guard standby database.

27 The lifecylce of the system is sustained by a number of scripts based on the initial database name. Before removing this database, make sure to note

down the contents of the /u01/app/oracle/product/12.1.0/dbhome_1/dbs/opcDB_NAME.ora file. Wallet’s alias may change from one Data Center to

another and the files under /u01/app/oracle/admin/ will be overwritten in this procuedure.




[oracle@soacsdbB ~]$ cd /u03/app/oracle/fast_recovery_area

[oracle@soacsdbB ~]$ rm -rf ORCL

L. Configure Network Security

The following network configuration was created from the Oracle Compute Cloud Service – Service Console –

Network Tab on the primary site to provide access to the secondary one:


Security IP List soacsdbB IP List < soacsdbb’s public IP address>

Security Rule soacsdbB-to-soacsdb Status Enabled

Security Application soacsdb/db_1/ora_dblistener

Source Security IP List soacsdbB (created above)

Destination Security List soacsdb/db_1/ora_db

The following network configuration was performed from the Oracle Compute Cloud Service – Service Console –

Network Tab on the secondary site to grant access to the primary one:


Security IP List soacsdb IP List < soacsdb’s public IP address>

Security Rule soacsdb-to-soacsdb Status Enabled

Security Application soacsdbB/db_1/ora_dblistener

Source Security IP List soacsdb(created above)

Destination Security List soacsdbB/db_1/ora_db

To function correctly, Data Guard requires Oracle Net communication between the database servers on the primary

and standby site. The application servers and possibly other servers may also need access to the database through

Oracle Net. All other servers should not be allowed access. If OracleNet/JDBC access is required from external

clients to the cloud instances and internet access to the db listener has been granted, then the security requirements

above will be overridden by the latter.

M. Install Grid Infrastructure in the standby site DB

Oracle Grid Infrastructure software needs to be installed on the secondary database host (soacsdrB) in the same

way as in the primary site. Refer to section A above. Notice that since the initial DB instance has been removed the

last steps in section A (where the Db is started) do not need to be applied.

N. Configure the Database Listener in Oracle Restart in the secondary site:

As user oracle on the secondary site’s database host:

[oracle@soacsdbB ~]$ lsnrctl stop listener

[oracle@soacsdbB ~]$ cp $ORACLE_HOME/network/admin/listener.ora \

/u01/app/12.1.0/grid/network/admin/listener.ora

[oracle@soacsdbB ~]$ srvctl add listener -listener listener –oraclehome \

/u01/app/12.1.0/grid

[oracle@soacsdbB ~]$ srvctl start listener


O. Set TCP Socket Maximum Sizes recommended for DG in the secondary site:

NOTE: As documented in the Oracle Public Cloud documentation, you can gain root access to a provisioned

cloud instance by connecting to the opc user and then using the sudo command:

[opc@soacsdb ~]$ sudo -s

[root@soacsdbB ~]$ sysctl -w net.core.rmem_max=10485760

[root@soacsdbB ~]$ sysctl -w net.core.wmem_max=10485760

The /etc/sysctl.conf file was also edited to reflect the changes so that they would be preserved during an instance

reboot.

P. Set the MTU Size for OracleNet traffic in the secondary site:

As root execute the following

[root@soacsdbB ~]$ ip link set eth0 mtu 1500

[root@soacsdbB ~]$ echo "ip link set eth0 mtu 1500" >> /etc/rc.local

Q. Configure Oracle Net Encryption in the secondary site:

Copy the wallets from the primary site DB node to the secondary:

On soacsdr (assuming this was the primary’s site hostname) as user oracle:

[oracle@soacsdb ~]$ cd /u01/app/oracle/admin

[oracle@soacsdb ~]$ tar -czf /tmp/ORCL.tgz ORCL

The /tmp/orcl.tgz file was copied from soacsdr to soacsdrB.

On soacsdrB as user oracle:

[oracle@soacsdbB ~]$ cd /u01/app/oracle/admin

[oracle@soacsdbB ~]$ tar -xzf /tmp/ORCL.tgz

The following parameters where added to the $ORACLE_HOME/network/admin/sqlnet.ora file on both site’s

database hosts:

SQLNET.ENCRYPTION_CLIENT = requested

SQLNET.ENCRYPTION_TYPES_CLIENT = (AES256, AES192, AES128)

For the detailed procedure to create and disperse wallets, refer to Appendix C

R. Prepare an environment file for DataGuard configuration

A script will be used to configure and duplicate the database and to configure Data Guard broker. The file

~/script.env was created in the user oracle on the primary site’s database host as follows:

export passwd='<sys password of the database>'

export DB_NAME='ORCL'

export A_DBNM='<Primary Site’s database unique name>'

export A_PUB_IP='<Primary site’s database public IP address>'

export A_PRIV_IP='<Primary site’s database private IP address>'


export A_PORT='1521'

export A_DB_DOMAIN='<Primary site’s DB_DOMAIN>'

export B_DBNM='<Secondary Site’s database unique name>

export B_PUB_IP='<Secondary site’s database public IP address>'

export B_PRIV_IP='<Secondary site’s database private IP address>'

export B_PORT='1521'

export B_DB_DOMAIN='<Primary site’s DB_DOMAIN>'

export A_FILE_LOC='/u02/app/oracle/oradata'

export A_RECOV_LOC='/u03/app/oracle/fast_recovery_area'

export A_REDO_LOC='/u04/app/oracle/redo'

export B_FILE_LOC='/u02/app/oracle/oradata'

export B_RECOV_LOC='/u03/app/oracle/fast_recovery_area'

export B_REDO_LOC='/u04/app/oracle/redo'

export B_BASE='/u01/app/oracle'

For example:

export passwd='Acme1234#'

export DB_NAME='ORCL'

export A_DBNM='ORCL'

export A_PUB_IP='160.34.103.90'

export A_PRIV_IP='soacsdb.compute-us6z12opcsoa01.oraclecloud.internal'

export A_PORT='1521'

export A_DB_DOMAIN='us6z12opcsoa01.oraclecloud.internal'

export B_DBNM='ORCLB'

export B_PUB_IP='141.145.25.60'

export B_PRIV_IP='soacsdbB.compute-esoracle92991.oraclecloud.internal'

export B_PORT='1521'

export B_DB_DOMAIN='esoracle92991.oraclecloud.internal'

export A_FILE_LOC='/u02/app/oracle/oradata'

export A_RECOV_LOC='/u03/app/oracle/fast_recovery_area'

export A_REDO_LOC='/u04/app/oracle/redo'

export B_FILE_LOC='/u02/app/oracle/oradata'

export B_RECOV_LOC='/u03/app/oracle/fast_recovery_area'

export B_REDO_LOC='/u04/app/oracle/redo'

export B_BASE='/u01/app/oracle'

The file was copied to user oracle on Secondary site’s database host.

S. Set the appropriate value for DB_UNQNAME in the standby’s bash profile: edit the .bashrc file in

the oracle user’s home and and replace ORACLE_UNQNAME with B_DBNM above, for

example: ORACLE_UNQNAME=ORCLB;

Remember that all shell/terminals opened in the standby node before this update will contain the

incorrect ORACLE_UNQNAME value!

T. Configure TNS Entries for Redo Transport

The following entries were configured in the tnsnames.ora file in the database hosts of both sites. Executed as user

oracle on both database hosts:

. ~/script.env

cat >> $ORACLE_HOME/network/admin/tnsnames.ora <<EOF

${A_DBNM} =

(DESCRIPTION =


(SDU=65536)

(RECV_BUF_SIZE=10485760)

(SEND_BUF_SIZE=10485760)

(ADDRESS = (PROTOCOL = TCP)(HOST = ${A_PUB_IP})(PORT = 1521))

(CONNECT_DATA =


(SERVICE_NAME = ${A_DBNM}.${A_DB_DOMAIN})

)

)

${B_DBNM} =

(DESCRIPTION =

(SDU=65536)

(RECV_BUF_SIZE=10485760)

(SEND_BUF_SIZE=10485760)

(ADDRESS = (PROTOCOL = TCP)(HOST = ${B_PUB_IP})(PORT = 1521))

(CONNECT_DATA =


(SERVICE_NAME = ${B_DBNM}.${B_DB_DOMAIN})

)

)

EOF

The tsnnames.ora file was checked and the duplicate ORCL alias was removed in the primary site

Entries for each database are needed in both primary and standby tnsnames.ora files for proper redo transport.

NOTE: The primary database may already have a TNS entry in the Primary site DBCS instance tnsnames.ora with a

server name for the HOST. In this case simply change the server name in that entry to use the IP address for the

host instead.

NOTE: IP addresses are used since there is no DNS to resolve server names to IP addresses.

U. Configure Static Listeners

The following was executed as user oracle on Secondary site’s database host (soacsdbB):


[oracle@soacsdbB ~]$ cat >> /u01/app/12.1.0/grid/network/admin/listener.ora <<EOF

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = ${B_DBNM}.${B_DB_DOMAIN})

(ORACLE_HOME = /u01/app/oracle/product/12.1.0/dbhome_1)

(SID_NAME = ${DB_NAME})

)

)

EOF

The following was executed as the oracle user on Secondary site’s database host (soacsdbB):

[oracle@soacsdbB ~]$ /u01/app/12.1.0/grid/bin/lsnrctl reload


A static listener is needed for initial instantiation of a standby database. The static listener enables remote

connection to an instance while the database is down in order to start a given instance. The following steps will aid

in configuring the static listener on the cloud VM. See MOS 1387859.1 for additional details.

Static "_DGMGRL" entries are no longer needed as of Oracle Database 12.1.0.2 in Oracle Data Guard Broker

configurations that are managed by Oracle Restart, RAC One Node or RAC as the Broker will use the clusterware to

restart an instance.

For 11.2 configurations, a static listener is also required for Data Guard Broker.

Primary site:

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = <${A_DBNM}_DGMGRL)



)

)

Secondary site:

SID_LIST_LISTENER =

(SID_LIST =

(SID_DESC =

(GLOBAL_DBNAME = <${B_DBNM})



)

(SID_DESC =

(GLOBAL_DBNAME = <${B_DBNM}_DGMGRL)



)

)

V. Instantiate the Standby Database

Create the Auxiliary Database Password and Parameter Files. As user oracle on Secondary site’s database host

(soacsdbB):


[oracle@soacsdbB ~]$ cd $ORACLE_HOME/dbs

[oracle@soacsdbB ~]$ $ORACLE_HOME/bin/orapwd file='$ORACLE_HOME/dbs/orapw${DB_NAME}'

password=${passwd} force=y

[oracle@soacsdbB ~]$ cat > /tmp/aux.pfile << EOF

db_name=${DB_NAME}

db_unique_name=${B_DBNM}

db_domain=${B_DB_DOMAIN}

sga_target=800M

EOF


The size of sga size in this pfile does not need to match the primary; a new spfile will be generated during the

instantiation process with the correct values. This auxiliary pfile is only used to start the auxiliary instance.

Start the Auxiliary Instance: The auxiliary instance was started as user oracle on secondary site’s database host

with the following command:


[oracle@soacsdbB ~]$ export ORACLE_SID=${DB_NAME}

[oracle@soacsdbB ~]$ sqlplus "/ as sysdba" <<EOF

startup nomount pfile='/tmp/aux.pfile'

EOF

The auxiliary instance is a temporary instance (just memory structures) that enable communication between the

primary and standby sites. Think of it as a place-holder until the primary is able to copy its files making the auxiliary

instance a database.

Create Standby Database Audit and File Locations: The following command was executed as user oracle on

secondary site’s database host:


[oracle@soacsdbB ~]$ mkdir -p ${B_BASE}/admin/${B_DBNM}/adump

[oracle@soacsdbB ~]$ mkdir ${B_FILE_LOC}/${B_DBNM}

[oracle@soacsdbB ~]$ mkdir ${B_FILE_LOC}/${B_DBNM}/PDB1

[oracle@soacsdbB ~]$ mkdir ${B_FILE_LOC}/${B_DBNM}/pdbseed

[oracle@soacsdbB ~]$ mkdir ${B_FILE_LOC}/${B_DBNM}/any_specific_locations_with_data_files

[oracle@soacsdbB ~]$ mkdir ${B_RECOV_LOC}/${B_DBNM}

[oracle@soacsdbB ~]$ mkdir ${B_REDO_LOC}/${B_DBNM}

[oracle@soacsdbB ~]$ mkdir –p /u04/app/oracle/oradata/ORCLB/PDB1

Create the Standby Database: The standby database was instantiated by duplicating from the primary database.

The following script was run as user oracle from secondary site’s database host to create the RMAN duplicate script:


[oracle@soacsdbB ~]$cat >/tmp/${B_DBNM}.rman <<EOF

connect target sys/${passwd}@${A_DBNM}

connect auxiliary sys/${passwd}@${B_DBNM}

run {

allocate channel prmy1 type disk;

allocate auxiliary channel stby1 type disk;




duplicate target database for standby from active database

spfile

PARAMETER_VALUE_CONVERT= '${A_DBNM}','${B_DBNM}', '${A_DB_DOMAIN}','${B_DB_DOMAIN}'

set db_unique_name='${B_DBNM}'

set control_files='${B_FILE_LOC}/${B_DBNM}/control01.ctl',

'${B_RECOV_LOC}/${B_DBNM}/control02.ctl'

set audit_file_dest='${B_BASE}/admin/${B_DBNM}/adump'

set local_listener='(ADDRESS=(PROTOCOL=tcp)(HOST=${B_PRIV_IP})(PORT=${B_PORT}))'


set fal_server='${A_DBNM}'

set log_file_name_convert='${A_REDO_LOC}/${A_DBNM}','${B_REDO_LOC}/${B_DBNM}'

set db_file_name_convert='${A_FILE_LOC}/${A_DBNM}','${B_FILE_LOC}/${B_DBNM}',

'/u04/app/oracle/oradata/ORCL', '/u04/app/oracle/oradata/ORCLB'28

set db_create_file_dest='${B_FILE_LOC}'

set db_create_online_log_dest_1='${B_REDO_LOC}'

set db_recovery_file_dest='${B_RECOV_LOC}'

set db_recovery_file_dest_size='40G'

set diagnostic_dest='${B_BASE}/diag/rdbms'

set db_domain='$B_DB_DOMAIN'

set STANDBY_FILE_MANAGEMENT='AUTO'

NOFILENAMECHECK

section size 100M

dorecover

;

}

EOF

NOTE: It is assumed that no other directories have been used for datafiles or redo beyond those originally

employed by SOACS. The NOFILENAMCHECK settings is required because the RMAN duplication is creating a DB

in a different host that has the same disk configuration, directory structure, and file names as the host of the source

database.

The duplicate script was executed as user oracle from secondary site’s database host (soacsdbB):


[oracle@soacsdbB ~]$ rman << EOF

@/tmp/${B_DBNM}.rman

EOF

During the duplication process the following error may be observed and can be ignored:

ORA-38757: Database must be mounted and not open to FLASHBACK.

W. After the standby database is created in the cloud restart the databases

X. Set file conversion paths in the primary database:

The appropriate file name conversion parameters were set on the primary database by executing the following as

oracle:


[oracle@soacsdb ~]$sqlplus / as sysdba <<EOF

alter system set log_file_name_convert='${B_REDO_LOC}','${A_REDO_LOC}'

scope=spfile sid='*';

alter system set db_file_name_convert='${B_FILE_LOC}/${B_DBNM}','${A_FILE_LOC}/${A_DBNM}'

, '${B_FILE_LOC}/${B_DBNM}/PDB1','${A_FILE_LOC}/${A_DBNM}/PDB1',

'${B_FILE_LOC}/${B_DBNM}/pdbseed', '${A_FILE_LOC}/${A_DBNM}/pdbseed'

scope=spfile sid='*';

28 The /u04/app/oracle/oradata/ORCL location is used by SOACS temp datafiles


EOF

NOTE: This change needs a DB restrat to be effective)

NOTE: If additional custom DB files are spread across other mount points or subdirectories additional entries will be

required for db_file_name_convert and pdb_file_name_convert if applicable (datafiles in the scope of a pdb) to

revert the conversion established in the primary to secondary relationship. For each different path to the data files a

‘<secondary path>/${SECONDARY_DBNM}’ , '${primary path}/${PRIMARY_DBNM}' pair is required. Retrieve

the data file paths from the primary database with ‘select name from v$datafile;’ For example

db_file_name_convert = '/u02/app/oracle/oradata/ORCLB, /u02/app/oracle/oradata/ORCL,

/u02/app/oracle/oradata/ORCLB/PDB1, /u02/app/oracle/oradata/ORCL/PDB1,

/u02/app/oracle/oradata/ORCLB/PDB2,/u02/app/oracle/oradata/ORCL/PDB2'

NOTE: If log files are spread across multiple mount points additional entries will be required for

log_file_name_convert. For each different path to the log files a ‘<secondary path>/${SECONDARY_DBNM}’ ,

'${primary path}/${PRIMARY_DBNM}' pair is required. Retrieve the log file paths from the primary database with

‘select member from v$logfile;’

Y. Configure the Database in Oracle Restart

As user oracle on the secondary site’s database host (soacsdbB):


[oracle@soacsdbB ~]$ srvctl add database -db ${B_DBNM} -dbname ${DB_NAME} \

-oraclehome $ORACLE_HOME –startoption open -r PRIMARY –instance ${DB_NAME}

Z. Start the database using srvctl from here on:

srvctl start database -db ${B_DBNM}

AA. Enable Flashback on New Standby Database

Flashback was enabled on the standby by executing the following as oracle from the secondary’s site’s database

host:


[oracle@soacsdbB ~]$ sqlplus -s sys/${passwd}@${B_DBNM} as sysdba <<EOF

alter database flashback on;

EOF

BB. Configure Data Guard Broker

The broker was configured by executing the following as oracle from Primary site’s database:



alter system set dg_broker_start=FALSE;

alter system set dg_broker_config_file1='${A_FILE_LOC}/${A_DBNM}/dr1.dat';

alter system set dg_broker_config_file2='${A_RECOV_LOC}/${A_DBNM}/dr2.dat';

alter system set dg_broker_start=TRUE;

EOF

[oracle@soacsdb ~]$ sqlplus -s sys/${passwd}@${B_DBNM} as sysdba <<EOF


alter system set dg_broker_start=FALSE;

alter system set dg_broker_config_file1='${B_FILE_LOC}/${B_DBNM}/dr1.dat';

alter system set dg_broker_config_file2='${B_RECOV_LOC}/${B_DBNM}/dr2.dat';

alter system set dg_broker_start=TRUE;

EOF

sleep 30

dgmgrl sys/${passwd}@${A_DBNM} <<EOF

create configuration 'DGconfig' as primary database is ${A_DBNM}

connect identifier is ${A_DBNM};

add database ${B_DBNM} as connect identifier is ${B_DBNM};

rem !!The following 2 commands must be uncommented for 11g databases!!

rem edit database ${A_DBNM} set property RedoRoutes='(LOCAL:${B_DBNM} ASYNC)';

rem edit database ${B_DBNM} set property RedoRoutes='(LOCAL:${A_DBNM} ASYNC)';

EDIT CONFIGURATION SET PROTECTION MODE AS MaxPerformance;

enable configuration;

exit

EOF

The steps followed in the case study assumed that the primary database was not part of an existing Data Guard

broker configuration. If there is an existing broker configuration for the database the administrator will need to

remove it or know how to add the new standby database to the configuration.

A return value other than ‘NOCONFIG’ for the following query implies an existing broker configuration.

select decode(count(1),0,'NOCONFIG') from v$DG_BROKER_CONFIG;

CC. Verify Data Guard configuration

The Data Guard Broker VALIDATE DATABASE command is highly recommended for the most comprehensive Data

Guard specific health check. VALIDATE DATABASE performs an extensive configuration check and validates the

configuration’s readiness for switchover or failover.

Example:



DGMGRL> validate database orclb;

Database Role: Physical standby database

Primary Database: orcl

Ready for Switchover: No

Ready for Failover: Yes (Primary Running)

Temporary Tablespace File Information:

orcl TEMP Files: 6

orclb TEMP Files: 4

Standby Apply-Related Information:

Apply State: Running

Apply Lag: 2 seconds (computed 4 seconds ago)


Apply Delay: 0 minutes

The show configuration command will also show the status of both databases and will report any errors in the

DataGuard Broker configuration:

DGMGRL> show configuration verbose

Configuration - DGconfig

Protection Mode: MaxPerformance

Members:

orcl - Primary database

orclb - Physical standby database

Properties:

FastStartFailoverThreshold = '30'

OperationTimeout = '30'

TraceLevel = 'USER'

FastStartFailoverLagLimit = '30'

CommunicationTimeout = '180'

ObserverReconnect = '0'

FastStartFailoverAutoReinstate = 'TRUE'

FastStartFailoverPmyShutdown = 'TRUE'

BystandersFollowRoleChange = 'ALL'

ObserverOverride = 'FALSE'

ExternalDestination1 = ''

ExternalDestination2 = ''

PrimaryLostWriteAction = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

Oracle provides several automated health check tools that can be downloaded from My Oracle Support specific for

each type of hardware platform. Refer to the MAA oracle documentation for additional details on how to ensure that

Data Guard databases are compliant with Oracle MAA best practices and for additional data Guard run-time

monitoring queries.

DD. Update RMAN Repository

In lack of a central RMAN catalog database, Oracle recommends to run a crosscheck right after the initial set up and

after every switchover/failover to avoid inconsistencies between the file structure in primary and standby. As the

oracle user on the secondary host, run the following steps:


[oracle@soacsdbB ~]$ cat >/tmp/${B_DBNM}.crosscheck.rman <<EOF

connect target /

run {

crosscheck backup;

delete expired backup;

}

EOF

[oracle@soacsdbB ~]$ rman << EOF


@/tmp/${B_DBNM}.crosscheck.rman

EOF

NOTE: The crosscheck operation may take a long time, be patience for the command to complete

It is important that backups complete cleanly at least in primary. Please check the /home/oracle/bkup/<sid>/log files

for any possible errors (these should also be reported in the Database Alert)

NOTE: The automated backup script will fail to perform backups or to clean archive log in the standby database

while it is in mounted state (an Active Data Guard DB Edition is required in order to open the standby in read only

mode). Customers must perform the pertaining archive log clean up and storage allocation tasks in the standby

database to prevent possible storage capacity issues.


Appendix C - Creating and dispersing wallets in 12c

1. Create encryption wallet

Set the wallet location in the sqlnet.ora on all nodes of primary and standby.

ENCRYPTION_WALLET_LOCATION =

(SOURCE = (METHOD = FILE)

(METHOD_DATA =

(DIRECTORY = /u01/app/oracle/admin/TDE/$ORACLE_SID)

)

)

NOTE: Using ORACLE_SID in the directory path ensures that all databases do not share the wallet. If there is just

one database on the system the ORACLE_SID is not necessary.

2. Create the corresponding directory on all nodes with the proper ORACLE_SID.

$mkdir -p /u01/app/oracle/admin/TDE/$ORACLE_SID

3. Initiate a new SQL*Plus session. This causes the changes to sqlnet.ora to be picked up.

4. Create the password-based keystore

SQL>ADMINISTER KEY MANAGEMENT CREATE KEYSTORE

'/u01/app/oracle/admin/TDE/<ORACLE_SID>' IDENTIFIED BY "AbCdEfGh!";

NOTE: Ensure the password string in double quotation marks (" ").

5. Open the wallet

SQL>ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "AbCdEfGh!";

6. Set the Encryption Key

SQL>ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "AbCdEfGh!" WITH BACKUP USING 'TDE';

7. Create Auto-login wallet

An Auto-login wallet removes the requirement of manually opening the wallet when the database is started.

SQL>ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE

'/u01/app/oracle/admin/TDE/$ORACLE_SID' IDENTIFIED BY “AbCdEfGh!”;

8. Copy the files generated in the keystore directory to all nodes of the primary and standby.

Copy files to each node:

$scp /u01/app/oracle/admin/TDE/$ORACLE_SID/*

oracle@<host>:/u01/app/oracle/admin/TDE/<SID_NAME>/

9. Ensure the wallet is open on all nodes

SQL> select * from gv$encryption_wallet;

INST_ID WRL_TYPE WRL_PARAMETER STATUS

1 file /u01/app/oracle/admin/TDE/primary1 OPEN

Creating and dispersing wallets in 11gR2

1. Create encryption wallet

Set the wallet location in the sqlnet.ora on all nodes of primary and standby.

ENCRYPTION_WALLET_LOCATION =

(SOURCE = (METHOD = FILE)


(METHOD_DATA =

(DIRECTORY = /u01/app/oracle/admin/TDE/$ORACLE_SID)

)

)

NOTE: Using ORACLE_SID in the directory path ensures that all databases do not share the wallet. If there is just one database on the system the ORACLE_SID is not necessary.

2. Create the corresponding directory on all nodes with the proper ORACLE_SID.

$mkdir -p /u01/app/oracle/admin/TDE/$ORACLE_SID

3. Initiate a new SQL*Plus session. This causes the changes to sqlnet.ora to be picked up.

4. Set the Master Encryption Key

SQL>ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY "AbCdEfGh!";

NOTE: Ensure the password string in double quotation marks (" ").

5. Create Auto-login wallet

An Auto-login wallet removes the requirement of manually opening the wallet when the database is started.

$ orapki wallet create -wallet /u01/app/oracle/admin/TDE/$ORACLE_SID -auto_login

6. Copy the files generated in the keystore directory to all nodes of the primary and standby.

Copy files to each node:

$ scp /u01/app/oracle/admin/TDE/$ORACLE_SID/*

oracle@<host>:/u01/app/oracle/admin/TDE/<SID_NAME>/

7. Ensure the wallet is open on all nodes

SQL> select * from gv$encryption_wallet;

INST_ID WRL_TYPE WRL_PARAMETER STATUS

1 file /u01/app/oracle/admin/TDE/primary1 OPEN


Oracle Corporation, World Headquarters Worldwide Inquiries

500 Oracle Parkway Phone: +1.650.506.7000

Redwood Shores, CA 94065, USA Fax: +1.650.506.7200

Copyright © 2017 Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the

contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0615 White Paper Title: SOA Cloud Service Disaster Recovery - Production and DR in the Cloud August 2018

C O N N E C T W I T H U S

blogs.oracle.com/oracle

facebook.com/oracle

twitter.com/oracle

oracle.com

Date post:	04-Jan-2019
Category:	Documents
Upload:	haliem
View:	218 times
Download:	0 times

SOA Cloud Services Disaster Recovery - oracle.com · SOA CLOUD SERVICES DISASTER RECOVERY 12....

Documents