SRDF Solutions

Copyright © 2007 EMC Corporation. Do not Copy - All Rights Reserved.

SRDF Introduction - 1

© 2007 EMC Corporation. All rights reserved.

Symmetrix Business Continuity:

SRDF Solutions

Symmetrix Business Continuity:

SRDF Solutions

Welcome to Symmetrix Business Continuity: SRDF Solutions.

Copyright © 2007 EMC Corporation. All rights reserved.

These materials may not be copied without EMC's written consent.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC and SRDF are registered trademarks of EMC Corporation.

All other trademarks used herein are the property of their respective owners.



© 2007 EMC Corporation. All rights reserved. SRDF Introduction - 2

Revision History

Updated SE – 6.3December 20063.2a

CompleteAugust 20063.1

November 20063.2

CompleteSeptember 20053.0a

Updated for SE 6.4/72June 20073.3

CompleteJuly 20051.2a

CompleteMarch 20051.0a

RevisionsCourse DateRev Number




Program Administration

� Attendance roster

� Restrooms

� Telephones / Etiquette

� Attendance Rules

� Fire / Evacuation Procedures

� Cafeteria

� Labs

� Local Sites of Interest

� Class Evaluations




Course Objectives

Upon completion of this course, you will be able to:

� Describe the relevancy of SRDF solutions with different (RPO) Recovery Point Objective needs

� Identify SRDF concepts, terminology and functionality

� Discus EMC’s Symmetrix Management Console (SMC)

� Describe SRDF host considerations and configurations within Sun Solaris, HP-UX, IBM AIX, and Windows LVM environments

� Describe SRDF/A theory of operation, and execute SRDF/A operations

� Identify the architectural components of SRDF/A

� Describe and execute SRDF/AR operations

� Describe EMC Consistency Technology

The objectives for this program are shown here. Please take a moment and review them.




SRDF Introduction

Upon completion of this module, you will be able to:

� Discuss the Concept of Business Continuity

� Identify SRDF (Symmetrix Remote Data Facility)

solutions

� Identify SRDF solutions to satisfy different RPO (Recovery Point Objective) needs

� Describe EMC’s Symmetrix Management Console (SMC)

The objectives for this module are shown here. Please take a moment and review them.




“All Material is for Training Purposes Only”

Documentation

Student Resource Guide

• Module 1: BCR - Business Continuance Remote Introduction

• Module 2: SRDF/S (Synchronous)

• Module 3: SRDF Operations

• Module 4: SRDF/A (Asynchronous)

• Module 5: SRDF/AR (Automated Replication)

Labs Guide

• SRDF - BCR Labs 1-7

Appendix

• BCR (Business Continuance Remote) Command Lookup Guide

The following modules will be presented over the next three days. All training materials presented in

this program are for training purposes only.




Training Schedule: Day 1 & 2

Training Schedule for SRDF Solutions, day one and day two.




Training Schedule: Day 3

Open – Lab - Activity

Training Schedule for SRDF Solutions, day three.




Software Environment

The following Software Environment has been established for each student:• Sun Solaris 5.8 OS

�OS Recommended Patches

�J2SE Cluster Patch

�Veritas Volume Manager (VxVM – Ver– 4.0)

• HPux 11.00 with 2004 patch bundle

�HP-LVM

• AIX – 5.2 / ML03 (Maintenance Level)

�AIX-LVM

• W2K

• EMC Solutions Enabler 6.4

• Enginuity Code 5672

• Custom Shell Scripts

The above software environment supports the BCL (Business Continuance Remote) program.

Appendix I is a Command Quick Reference Guide, created to support this training program.

All students attending this program should have a good understanding of the following:

� Unix file systems.

� The Unix Vi editor

� A general understanding of a Unix Volume Manager

� EMC Solutions Enabler

� Symm 5 / Symm 6 overview




What is Business Continuity?

� Business Continuity is the preparation for, response to,

and recovery from an application outage that adversely affects business operations

� Business Continuity Solutions addresses systems

unavailability, degraded application performance, or

unacceptable recovery strategies

Business Continuity remains at the top of every executive’s priority list. Yet executives find

themselves in a financial tug-of-war between business continuity solutions and other projects

competing for the limited resources. Fundamental to business continuity is the need to understand an

organization’s practices relative to the protection, availability, and usability of data.




Loss Revenue

Know the downtime costs per hour, day, two days...

Number of employees

impacted X hours out X

burdened hourly rate

Damaged Reputation

CustomersSuppliers

Financial markets

Banks

Business partners

Financial Performance

Revenue recognition

Cash flow

Lost discounts (A/P)

Payment guarantees

Credit rating

Stock price

Other Expenses

Temporary employees, equipment rental, overtime costs, extra shipping

costs, travel expenses...

Why Business Continuity?

Direct loss

Compensatory payments

Lost future revenue

Billing losses

Investment losses

Loss Revenue

Failures happen - hardware, software, natural disasters etc. Downtime has a significant impact, the cost

is more than just financial loss. What can we do to avoid downtime or minimize the length of time we

are down? EMC offers Business Continuity Solutions that help address common failures or outages.

Host to Storage failures and Performance bottleneck of a Host Bus Adapter: PowerPath

Local Storage Protection with local mirroring: TimeFinder Family of Products

Remote Storage Protection and Site Protection: SRDF Family of Products



© 2007 EMC Corporation. All rights reserved. SRDF Introduction - 12 12

Business Needs Drive the Technology Choice

Synchronous

Replication

Wk

s

Days

Hrs Mins

Secs

Sec

s

Mins

Hrs Days

Wks

Recovery Point Recovery Time

Asynchronous

Replication

Periodic

Replication

Tape

Backup

SRDF Recovery Point Objectives (RPO)

Recovery Point Objective (RPO): Maximum amount of data loss an application can tolerate as measured in time. In other

words, the amount of data loss that can be tolerated (cost of transaction versus risk).

Individual customer business needs drive the technology chosen to meet specific recovery point objectives. This is also

known at RPO (Recovery Point Objective).

Data Characteristics that Influence Data Storage Decisions:

Several factors affect the value of data including: Legislation, which can mandate how long the data must be accessible,

and by whom; Business processes that are tied to points in time (book closing, quarterly reports, tax deadlines, billing

cycles, etc.)

Business processes that are tied to customer satisfaction service levels associated with the data as its purpose changes. For

example, data can start out as transactional, then migrate to billing, then reporting and customer service, then to scoring

data for a marketing system, and finally to archival.

The usefulness of data to the business will vary over time, and hence the necessity to have immediate access to the data

changes. The decisions about where data is placed in the storage infrastructure and the methods used to protect that data

are fundamentally driven by three factors: The time required to access the data relative to the cost of the access (that is,

usefulness to the business versus cost).

Recovery Time Objective (RTO): This refers to the maximum time a company budgets to bring an application back online

in the event of a disaster. In other words, the time it takes to recover the data once a disaster or other recovery event is

declared (risk versus cost). Each change in data placement and protection criteria represents a stage in the life cycle of the

data and is directly related to the usefulness or importance of the data to keep the business functioning.




What are SRDF Solution Sets?

� Symmetrix Logical Volumes are mirrored between geographically dispersed locations

� Maintain real-time physically separate mirrors of data with SRDF

� Maintains near real-timephysically separate mirrors of selected volumes with SRDF/A (Asynchronous) or SRDF/AR (Automated Replication)

� Continue running through events such as individual drive/link failures

� Mirror copy can be split and used for disaster recovery or business continuance applications

SRDF/S

SRDF/A

ReportGeneration

DecisionSupport

Tape Backup

ERP

E-mail

ERP

E-mail

EMC Snap

TimeFinderBCV

SRDF/AR

The SRDF family of software is the most powerful suite of remote storage replication solutions

available for disaster recovery and business continuity. It leverages high-end Symmetrix storage

architecture to offer unmatched deployment flexibility and massive scalability so you can meet mixed

service level requirements with minimal operational impact. The SRDF family is the most widely

deployed set of high-end remote replication solutions, and is installed in tens of thousands of

demanding environments worldwide. The SRDF family is the only product that provides cross volume

and storage system consistency, tight integration with industry-leading applications, and automated

management for simplified usage.




SRDF Solutions

SRDF/AR

• Hours of data exposure

• No performance impact

• Unlimited distance

• Requires BCVs

Source

Unlimited Distance

Target

Prod

SRDF/S

• No data exposure

• Some performance impact

• Limited distance

Source

Limited Distance

Target

SRDF/A

• Seconds of data exposure

• No performance impact

• Extended distance

Source

ExtendedDistance

Target

EMC has several remote replication offerings for various service level requirements. For zero data

exposure, EMC offers the industry leader for synchronous mirroring: SRDF. However, as with any

synchronous solution, there are characteristics that must be understood. Distance is limited by

application time-outs and bandwidth must be sized for peak workload at all times.

SRDF/Asynchronous is a solution for service level requirements that need Recovery Point Objectives

(RPO) in the seconds-to-minutes area. SRDF/AR delivers solutions that combine SRDF with

TimeFinder to create single-hop and multi-hop environments for specialized needs.

These solutions offer different RPOs and have different requirements for bandwidth, supported

distances, etc. No matter what your requirements are, EMC can help deliver the right Remote

Replication Solution.




PrimarySite

Long Distance

Site

SourceTarget’

BunkerSite (Local)

Target

X

Synchronous Target Site

Workload Site

Asynchronous Site

SRDF/Star (5671)

� SRDF/STAR (Symmetrix Triangular Automated Replication)

– Start with the scenario of a Primary site with multi-site protection

� An SRDF/S link to a nearby bunker site (zero time lag RPO)

� An SRDF/A link to a long distance site (Seconds - Minutes time lag RPO)

– "Disaster" strikes and the primary site is lost

� Bunker site and long distance site are incrementally synchronized to most recent

data

� Third SRDF/A link between the 2 remaining sites maintains continuous protection

SRDF/Star is a three-site1 disaster recovery solution that uses concurrent RDF technology to replicate

data from a primary production site (referred to as the workload site) to a nearby remote site and a

distant remote site. Data is transferred in SRDF/Synchronous (SRDF/S) mode to the nearby remote site

(referred to as the synchronous target site) and in SRDF/Asynchronous (SRDF/A) mode to the distant

remote site (referred to as the asynchronous target site).

SRDF/Star provides consistent data protection and incremental data recovery between target sites in

the event of a workload site failure




SRDF Management Tools

� Solutions Enabler

� EMC Control Center • Easy, point-and-click access

• Excellent for ad hoc TimeFinder operations

� SMC: Symmetrix Management Console• Web-based application for managing the Symmetrix

• Works the same as the CLI/API – the same rules apply

• Makes it easier to perform complex CLI tasks

� EMC Replication Manager• Discovers replication environments

• Automates replication process

• Integrates replication technologies at the application level

In addition to SYMCLI (Solutions Enabler), SRDF can be managed by a variety of management tools

such as EMC Control Center and Symmetrix Management Console. Through its graphical user

interface, EMC ControlCenter and SMC software will organize related devices into device groups.

SRDF operations may be performed on all devices in this device group by using a single command.

The group information is maintained in the SYMAPI database. Another management tool is EMC

Replication Manager. It is an application that automates, simplifies, and manages disk-based

replications by using SRDF operations.




Symmetrix Management Console (SMC)

SRM Planning and Provisioning

Serv

ice

s

Sto

rag

e M

an

ag

em

en

t

SRM Monitoring and Reporting

Device Management

� Independent, light-weight, web-based application• Simple and easy to use browser interface

• Hosted on small Windows/Linux server

• Enables remote access and management from nearly any client

� Enables access, configuration, and basic operation of Symmetrix arrays• Supports all configuration capabilities of

Solutions Enabler/CLI

� Supports multiple generations of Symmetrix• Enginuity version 5x68 and newer

� Provides day-one support of new Symmetrix features when released

� Adding full-feature ControlCenter does not require management data to be migrated

EMC offers a Graphical User Interface (GUI) product called Symmetrix Management Console, or

SMC. SMC enables the user to deploy web-based device management within a Symmetrix

environment. The user can now choose the right management product or products to meet a set of

specific requirements.

Almost anything you can do using the Solutions Enabler command line (CLI) can now be done using

the SMC GUI. SMC manages all Symmetrix systems running Enginuity version 5568 and up, and

supports new hardware and software features and functionality at the time of product release.

Addressing customer demands for enhanced platform interoperability, SMC is an independent

application which runs using its own lightweight Windows/Linux server. The client runs in a browser

window, supporting nearly any client with remote access to the server.

The SMC GUI features closely match Solutions Enabler CLI features to include all basic monitoring,

configuration, and control of Symmetrix arrays. SMC has no Symmetrix-related database other than

the Solutions Enabler database, so all data automatically transfers to the full-feature ControlCenter

when discovered by the Symmetrix agent. Due to the combination of being light-weight yet feature

rich, early response to SMC shows it to be a product leader in this space.




SMC Functionality

� Access Management

• Manage users, permissions/roles

• Symmetrix Access Controls

� Configuration Management

• Create devices, map and mask devices, create device groups, set Symmetrix attributes

� Replication Management

• TF/Clone, TF/Mirror, TF/Snap, SRDF/S, SRDF/A, SRDF/DM, Open Replicator, Optimizer

� Alerts and Monitoring

• Monitor Device status, device attributes, operations status

• Monitor array alerts

Primary SMC features include providing the facility to manage SMC access and permission levels,

discover, and configure, Symmetrix arrays. Also SMC enables monitoring of replication operations

within Symmetrix arrays.

Configuration activities include Create devices, map and mask devices, create device groups, set

Symmetrix attributes; create Symmetrix Logical Volumes from un-configured storage, create meta

volume devices, both concatenated and striped. Additionally, the ability to modify existing configured

storage after un-mapping it from the hosts; map and un-map one or more logical volumes to a port or

ports; delete devices to convert configured storage into un-configured space; manage virtual and save

devices, and manage dynamic spares.

SMC enables you to monitor device status, device attributes, and operations status, perform and

monitor TimeFinder/Mirror, TimeFinder/Clone, and TimeFinder/Snap operations, SRDF operations,

Open Replicator sessions, Optimizer and Quality of Service features, as well as monitor and report

alerts.




Components of the SMC Interface

(Properties) View (Details)(Properties) View (Details)

Navigation Tree

Menu BarMenu Bar

View BarView Bar

The major components of the SMC Interface are highlighted above. To monitor and control

operations, you can select a single object or a single folder of multiple objects in the SMC Navigation

Tree above.

The View Bar is used to switch between five different views: 1. Properties, 2. Configuration Session,

3. Alerts, 4. Command History and 5. Replication Monitor.

The current “View” selected will display details in the “view area”. The View button and

corresponding view display are color coded to match.

Note the “Alert” counter in the top right, which also selects the Alert View. Tree Selection, View

Selection, and Object Selection within the View area determine the current display. The View may be

split horizontally into two or three areas depending on the detail associated with the selection.




� SRDF• Synchronous

• Asynchronous

• Adaptive Copy

• SRDF Configuration

� TimeFinder• Mirror

• Clone

• Snap

• SAVE Device Pool management

� Open Replicator

� Quality of Service

Replication Operations Supported by SMC

Almost all replication technologies available on the Symmetrix Arrays can be monitored and managed

via the Symmetrix Management Console (SMC) application.

Note:

At the present time SRDF/Star cannot be managed via SMC.




Creating Device Groups (Replication / SRDF Control)

� Device Group Management � Create Device Group

SRDF Action

- Control- Settings- Configuration

SRDF Action

- Control- Settings- Configuration

SMC Device Group creation is a multi-step process, similar to creating SYMCLI device groups using

symdg, symld, and symbcv. The wizard is launched by right clicking the Device Group Folder and

choosing the Device Group Management, and then the Create Device Group option.

Once a Device Group has been created selecting Replication / SRDF Controls, Settings, or Config

enables SMC / RDF control functionality on the selected Device Group.

The above SMC01 Device Group has two R1 devices (02EF, and 02F0) paired with two RDF / R2

devices (02EF, and 02F0). The above device group is currently in synchronized, with no Alerts to

report.




SRDF Settings: Setting SRDF Mode

� Replication � SRDF Settings

Drop Down to select “Mode Setting”

SRDF Mode operations are shown in this slide. Right Click an RDF device group and choose

Replication � SRDF Settings.

The dialog box shows the current mode and pair states. The mode can be changed by using the Set

Mode pull down. When the Set Mode is pulled down, the user seea Synchronous, Asynchronous, Semi

Synchronous, to mention just a few.

In this example, the mode can be changed from Synchronous to Asynchronous.




Module Summary

Key points covered in this module:

� Business Continuity concepts

� SRDF (Symmetrix Remote Data Facility) solutions

� SRDF solutions used to satisfy different RPO (Recovery

Point Objective) needs

� EMC’s Symmetrix Management Console (SMC)

These are the key points covered in this module. Please take a moment to review them


SRDF/S (Synchronous) - 1

© 2007 EMC Corporation. All rights reserved. SRDF/S (Synchronous) - 1

SRDF Overview


� Describe EMC SRDF functionality and its uses

� Describe SRDF Link configurations

� Describe the concept of SRDF Group

� Describe SRDF swap

� List the characteristics of:

• Concurrent SRDF

• Dynamic SRDF

The objectives for this module are shown here. Please take a moment to read them.




Symmetrix Remote Data Facility (SRDF)

� Facility for maintaining real-time or near-real-time physically separate mirrors of selected volumes

� Uses no host CPU resources• Mirroring done at the storage

level

� Operating system independent• Open Systems

• Mainframe

R1 R2

SRDF

Open Systems / Mainframe

Symmetrix Remote Data Facility (SRDF) is a Symmetrix system based business continuance, disaster

recovery, restart, and data mobility solution. In the simplest terms, SRDF is a configuration of multiple

Symmetrix units that maintains real time copies of logical volume data in more than one location. The

Symmetrix units can be in the same room, in different buildings within the same campus, or hundreds

and even thousands of miles apart.




SRDF Source and Target Volumes

� Symmetrix Logical Volume types:

• SRDF Source or R1 Volumes: Primary Volume with R/W access to local host

• SRDF Target or R2 Volumes: Backup Volume used for DS or DR Applications

� The attached host is unaware of SRDF protection

WDRW

M1 M4M3M4M2 M3M1

M2

TargetSource

This slide displays the representation of the mirror positions when both the Source and the Target

SRDF Logical Volumes have local protection (RAID-1).

In this diagram, the Target-R2 volume is also represented with 4 mirror positions and has local

protection implemented. Three of the mirror positions are used. The first two mirror positions represent

local mirrors and the third mirror is occupied by SRDF. If a BCV is established with the R2 volume,

then it will occupy the next available mirror position.

Under normal circumstances, the R1 volume presents a Read-Write (RW) status to the host which

access it, and the R2 presents Write-Disabled (WD) to its host.




Remote Link Director (RLD)

Remote

Link

Director

Remote

Link

Director

TargetRemote

LinkDirector

Remote

Link

Director

Source

A Remote Link Director is a hardware that provides communication and data paths between local and

remote Symmetrix units. The Symmetrix can be configured with the following RLDs:

•Fibre Channel directors (RF)

•ESCON directors (RA)

•Multiprotocol Channel Directors (MPCD) available with these channel connections:

− FICON

− iSCSI for host

− GigE (RE) for SRDF




SRDF Groups

Remote

Link

Director

RemoteLink

Director

Remote

Link

Director

Remote

Link

Director

R1R1

R1 R1 R2R2

R2 R2

R2R2

R2R1

R1R1

RDF Group 1

RDF Group 1

RDF Group 2RDF Group 2

RDF Group 1,2,3….

RDF Group 1,2,3….

An SRDF group, also known as RDF group or RA group, logically defines relationships between

Symmetrix systems. An SRDF group is a set of SRDF director port connections configured to

communicate with a another set of SRDF director ports in another Symmetrix system. Logical volumes

(devices) are assigned to SRDF groups.

Many SRDF groups can share a physical link between the Remote Link Directors. There are two ways

to create an RDF group - static and dynamic. Both share the same features and functionality, the

difference between the two types is how they are created. Static RDF groups are created during the

Symmetrix configuration, and almost always by EMC personnel. Dynamic RDF groups are created and

deleted by users through a set of Symmetrix command line interface (SYMCLI) commands.

Prior to SE 6.3, the Symmetrix DMX supported up to 64 total RDF groups. With SE 6.4 and 5772, 1

to 250 RDF groups are supported.




Uni-DirectionalSymmetrix A

Source

Symmetrix B

Target

Bi-DirectionalSymmetrix A Symmetrix B

Source TargetTarget Source

Dual Configuration

Symmetrix A Symmetrix B

Source TargetSource Target

Target SourceTarget Source

SRDF Link Configuration

RA

Group

RA

Group

RA

Group

RA

Group 1

RA

Group 1

RA

Group 2

RA

Group 2

RA

Group

SRDF offers three types of link configurations between source (local) and target (remote) Symmetrix

systems: Uni-Directional, Bidirectional and Dual Configuration.

SRDF Unidirectional Link Configuration

If all primary (source or R1) volumes reside in one Symmetrix system and all secondary (target or R2)

volumes reside in another Symmetrix

system, write operations move in one direction, from primary to secondary. Data moves in the same

direction over every link in the SRDF group.

SRDF Bidirectional Link Configuration

If an SRDF group contains both primary and secondary volumes, write operations move data in both

directions over the SRDF links for that group.

SRDF Dual-Directional Link Configuration

With a dual-directional configuration, multiple SRDF groups are used; some groups send data in one

direction, while other groups send data in the opposite direction.




SRDF Modes of Operations

� Primary and Secondary Modes

• Two Primary SRDF Modes

• Synchronous

• Semi-synchronous

• Secondary SRDF Mode

• Adaptive Copy

• Write Pending

• Disk Mode

• Operational Modes are set on Symmetrix Logical Volume level Using GUI or CLI and can be changed dynamically

� SRDF/A – Asynchronous

� Domino Mode (or attribute)

Listed are the operational modes for SRDF operations: Synchronous mode, Semi-Synchronous mode,

Adaptive Copy-Write Pending mode, Adaptive Copy-Disk Copy mode, and Asynchronous mode.

These operational modes are selectable based on many requirements such as RPO, bandwidth, and

performance. One of the two primary SRDF modes of operations is set at the source (R1) volume

during Symmetrix configuration. All source (R1) volumes are configured for either the Synchronous or

Semi-Synchronous mode. These two modes are considered to be pre-determined SRDF modes, which

may be altered using SymCli. Adaptive copy is the secondary mode that facilitates data sharing and

migration. Asynchronous mode continually collects and sends data to the remote Symmetrix.

Asynchronous mode must be set for the entire RA group. Users can set SRDF to function in a

secondary or Asynchronous mode. SRDF will revert to the pre-determined primary mode if it cannot

maintain the criteria to remain in the secondary mode.

Domino Mode could be classified as an SRDF attribute. Not necessarily a “Mode”. This attribute is

set or used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write

operations to both source and target volumes if the target volume become unavailable, or if all SRDF

links become unavailable.




Source Target

SRDF links

Write I/O received from host/server at the source

I/O is transmitted to the target

An acknowledgment is provided by target back to the source

I/O is serviced to the host

Synchronous Mode

SRDF Synchronous Mode is used primarily in SRDF campus environments. In this mode of operation,

Symmetrix maintains a real-time mirror image of the data of the remotely mirrored volumes.

Data on the source (R1) volumes and target (R2) volumes are always fully synchronized at the

completion of an I/O sequence.

The sequence of operations is:

� A write is received from the host/server at the source.

� The write is transmitted to the target.

� An acknowledgment is provided by the target back to the source.

� The write is acknowledged to the Host.

If step 3 never happens, the source SRDF services the I/O after a pre-determined timeout to keep the

production machine running.




Semi-Synchronous Mode

Source Target

SRDF links



I/O is transmitted to target

An acknowledgment provided by target back to source

SRDF Semi-Synchronous Mode is used primarily in extended distance environments. Semi-

synchronous mode allows the primary and secondary volumes to be out of synchronization by one

write I/O operation. Data must be successfully stored in the Symmetrix system containing the primary

volume before an acknowledgement is sent to the local host.

Semi-synchronous mode will not allow the next write operation to a primary device until a positive

acknowledgement is received from the target Symmetrix system that the first write operation was

received in the target Symmetrix global memory. However, any number of read operations can be

performed to the primary device while awaiting acknowledgement of the first write operation. Semi-

synchronous mode writes data to the primary device in the source Symmetrix system, completes the

I/O, and then synchronizes

The data with the secondary device in the target Symmetrix.


� An I/O write is received from the host/server at the source.

� The I/O is serviced to the host/server.

� The I/O is transmitted to the cache of the target.





Adaptive Copy Mode

Source Target

SRDF links



I/O accumulates in/on:- Symmetrix cache � Write Pending Mode

- R1 volumes � Disk Mode

I/O is transmitted to the target

An acknowledgment is provided by target back to the source

SRDF Adaptive Copy Mode is used primarily for data migrations and data center moves. This operational mode is not

recommended for use when mirroring for disaster recovery/restart purposes unless used with TimeFinder. This mode is

very useful for initial synchronization, especially over long distances. (Used within a SRDF/Star configuration).

SRDF Adaptive Copy Mode allows the source (R1) volumes and target (R2) volumes to be a out of synchronization by a

number of I/O’s that users can define, a skew value. There are two types of adaptive copy: Write Pending Mode and Disk

Mode. Adaptive Copy data movement is handled at the track level. The target data is only usable after a full

synchronization.


� An I/O write is received from the host/server at the source.

� I/O is accumulating.

� I/O is serviced.

� The I/O is transmitted to the target.


In Write Pending Mode, the unit of transfer across the SRDF link is the updated blocks rather than an entire track, resulting

in more efficient use of SRDF link bandwidth. Data is read from global memory than from disk, thus improving overall

system performance. However, the global memory is temporarily consumed by the data until it is transferred across the

link.

In Disk Mode, while less global memory is consumed it is typically slower to read data from disk than from global memory,

additionally, more bandwidth is used because the unit of transfer is the entire track. Additionally, because it is slower to

read data from disk than global memory, device resynchronization time increases.

Adaptive copy disk mode should not be used if the primary volumes are not RAID protected.




Asynchronous Mode

Source Target

SRDF links


I/O accumulates in Source Symmetrix cache


I/O is continually transmitted to the target

I/O accumulates in Target Symmetrix cache

SRDF/A provides a long-distance replication solution with minimal impact on performance. This

protection level is intended for customers requiring minimal host application impact, who need to

maintain a restartable copy of data at the target site at all time.

SRDF/A continually process Write I/O’s in batches. The interval between batches is referred to as a

cycle.


� An I/O write is received from the host/server into the cache of the source.

� I/O is accumulating.

� I/O is serviced.

� The I/O is transmitted to the target.




Domino Mode (attribute) with SRDF/Synchronous

Source Target

SRDF links


I/O fails to transmit to the target

Both Source and Target become unavailable

Domino Mode is used in conjunction with other SRDF modes except SRDF/A. It effectively stops all

write operations to both source and target volumes if target volume become unavailable, or if all SRDF

links become unavailable. User will need to manually re-enable the source volumes. While such a

shutdown temporarily halts production processing, domino modes can prevent data integrity exposure

that causes the inconsistent image on the target volume.




SRDF Level of Synchronization

� Synchronous Mode�Source = Target

� Semi Synchronous Mode

Source ≅ Target

�At most, Source is 1 I/O ahead of Target, per volume

� Adaptive Copy

Source ≠ Target

�Source may be up to 65535 tracks per volume ahead of Target

�Skew value set per logical volume

� Asynchronous�SRDF/A - Source is minutes ahead of Target

�SRDF/AR - Source is hours ahead of Target

SRDF offers considerable flexibility for various levels of synchronization. To determine the level of

synchronization, one must understand the required Recovery Point Objective. This is the amount of

data that can be lost in the event of a site outage. There are other factors like distance, bandwidth, and

response time latency that must be considered before determining a synchronization level.




SRDF Serialization

� Writes to Target volumes must happen in the same order

as they are written to the Source in order to have an instance in time, consistent and recoverable copy

� In Synchronous, Semi-synchronous and Asynchronous

modes, writes are sent to the remote Symmetrix in the

order received

• If the remote Symmetrix is not accessible, writes are accumulated as

invalid tracks

• When the remote Symmetrix becomes available, invalid tracks are

sent without regard to serialization

� Serialization is not maintained in Adaptive Copy mode

• Typically used for data migrations

Serialization maintains the order in which writes are received at the remote (target) Symmetrix. SRDF

serialization must be maintained in order to have a recoverable/restartable copy of data at a target site.

Through serialization, write fidelity is guaranteed. In normal operations, SRDF maintains order writes

with Synchronous, Semi-synchronous, and Asynchronous modes. But when the link becomes

unavailable for any reason, writes accumulate as invalid tracks which the application continues to

function on the host. When the link is restored, the Adaptive Copy mode is used to propagate changes

across the link. This introduces risk, since serialization is not maintained with Adaptive Copy.




T1 T2 T3 T4

32k 32k 32k 32k

T5 T6 T7 T8

32k 32k 32k 32k

T1 T3 T4

32k 32k 32k 32k

T5 T6 T7

32k 32k 32k 32k

SRDF / DM

Source – R1 Target – R2

Tracks

Adaptive Copy: Disk Mode

The slide shows Adaptive Copy – Disk Mode during in operation. SRDF does not guarantee

serialization of the tracks being transferred in this mode. In this example, track 2 and track 8 may not

be present on the target volume at the time of disaster rendering the target volume useless. Therefore,

the target volume will not serve as a disaster protection mechanism. The consistency of the target

volume is not maintained during the replication process in Adaptive Copy –Write Pending or Disk

Mode. The target is consistent only after the replication has completed.




Dynamic SRDF

� Enables user to dynamically define relationships

between R1 and R2 volumes

� Provides flexibility for user to tailor SRDF configuration to

their changing application requirements001

STD

001

R1

054

R2

054

STD

001

R1

054

R2

001

STD

054

STD

Connectrix(s)

Create pair

Establish

Delete pair

Prior to Dynamic SRDF, the R1 and R2 pairings were static and defined in the configuration file (BIN

File) on the Symmetrix. Any changes to SRDF device pairing required a new BIN file to be defined

and loaded into the Source and Target Symmetrix.

Dynamic SRDF available with 5x68 Enginuity code will provide the capability to change device

pairings on the fly without requiring a BIN file configuration change to be performed by an EMC

Customer Engineers.




R1/R2 Swap

001

R1

001

R2

054

R2

054

R1Connectrix(s)

An R1/R2 personality swap (or R1/R2 swap) refers to when the RDF personality of the RDF device designations of a

specified device group are swapped so that source R1 device(s) become target R2 device(s) and target R2 device(s) become

source R1 device(s). Dynamic RDF swaps are available with Enginuity™ version 5567 or later. To perform an R1/R2

swap, you must have an SRDF license with Symmetrix 5567 microcode or higher and Dynamic RDF must be enabled in

your Symmetrix configuration.

Sample scenarios for R1/R2 Swap

- Symmetrix Load Balancing

In today’s rapidly changing computing environments, it is often necessary to deploy applications and storage on a different

Symmetrix without having to give up disaster protection. R1/R2 swap can enable this redeployment with minimal

disruption, while offering the benefit of load balancing across two Symmetrix storage arrays.

- Primary Data Center Relocation

Sometimes a primary data center needs to be relocated to accommodate business practices. For example, several financial

institutions in New York City routinely relocate their primary data center across the Hudson River to New Jersey as part of

their disaster drills. R1/R2 swaps allow these customers to run their primary applications in their New Jersey data centers.

The Manhattan data centers now act as the disaster protection site.

- Post-Failover Temporary Protection Measure

If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of production computing to the

target site without giving up the security of remote data protection. When all problems have been solved on the local

Symmetrix, you have to failover again and swap the personality of the devices to go back to the original configuration.




Concurrent SRDF

� One R1 can be paired with two R2 devices, concurrently

� Remote BCVs can be associated with only one of the R2 mirrors

Source

M1 M4M2 M3

Target “B”

M1 M4M2 M3

Target “A”

M1 M4M2 M3

Connectrix(s)

Concurrent SRDF allows two remote SRDF mirrors of a single R1 device, e.g. use one remote copy

for disaster recovery, and another for decision support or backup.

Each Remote Link Director is assigned to an RA Group. With ESCON, only one RA group per RLD is

allowed, but Fibre Channel SRDF RA Groups can be defined to the same RLD.

Any mixture of SRDF modes is allowed, except for Sync and Semi-sync configuration and Async and

Async configuration.

A write IO from the host at the primary device side cannot be returned as completed until both remote

Symmetrix’ signal the local Symmetrix that the SRDF IO is in cache at the remote side.

1 Sync and 1 Adaptive Copy remote mirror:

The SRDF IO from the secondary device operating in Synchronous mode must present ending status to

the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the

secondary device operating in Adaptive Copy mode.




Concurrent SRDF

� One R1 can be paired with two R2 devices, one in each Symmetrix, concurrently

� All combinations of Primary/Secondary modes for the R1-R2 pairs are allowed - except one pair in Sync and the other in semi-sync, both cannot be “Async”

� Cannot restore from both R2 mirrors to the R1 simultaneously

� SRDF swap is not allowed. For example if the R1 is changed to an R2 one will be left with R2->R1, R2->R2@#!

� Remote BCVs can be associated with only one of the R2 mirrors

A BCV can only be established with one of the Target volumes, not both. In case the source is locally

protected, the BCV device cannot be established with it’s source, because all four(4) mirror positions

will be occupied

2 Synchronous remote mirrors :

� A write IO from the host at the primary device side cannot be returned as completed until both

remote Symmetrix’ signal the local Symmetrix that the SRDF IO is in cache at the remote side.

1 Sync and 1 Adaptive Copy remote mirror:

� The SRDF IO from the secondary device operating in Synchronous mode must present ending

status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not

wait for the secondary device operating in Adaptive Copy mode.

The same general principle applies when both remote mirrors are operating in Semi-Sync mode.




Module Summary


� Overview of SRDF solutions

� SRDF functionality and its uses

� SRDF Link configurations

� SRDF Groups

� SRDF swap functionality

� Characteristics of Concurrent and Dynamic SRDF



SRDF Operations - 1

© 2007 EMC Corporation. All rights reserved. SRDF Operations - 1

SRDF Operations


� Identify RDF volumes with SYMCLI

� Configure and display properties SRDF Device Groups

� Display and monitor the status of a Device Group

� Perform the following operations using SYMCLI:

• SRDF Disaster Recovery

• SRDF Link Control

• SRDF Decision Support

� Create and delete Dynamic RDF pairs using SYMCLI

� Describe Consistency Technology and its applications



SRDF Operations - 2


Identify Accessible SRDF Volumes

� symrdf list pd displays summary information about all SRDF volumes accessible to the host

� symrdf list dev displays summary information about all SRDF volumes in the Symmetrix

# symrdf list pd

Symmetrix ID: 000187940398

Local Device View

-------------------------------------------------------------------------

STATUS MODES RDF S T A T E S

Sym RDF --------- ----- R1 Inv R2 Inv ----------------------

Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair

---- ---- ------ --------- ----- ------- ------- --- ---- -------------

0190 0190 R1:11 RW RW RW S.. 0 0 RW WD Synchronized

0191 0191 R1:11 RW RW RW S.. 0 0 RW WD Synchronized

0192 0192 R1:11 RW RW RW S.. 0 0 RW WD Synchronized...

Total -------- --------

Track(s) 0 2

MB(s) 0.0 0.1

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy

D(omino) : X = Enabled, . = Disabled

A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off


SRDF Operations - 3


# symdg create -type RDF1 srcdg

# symld -g srcdg add pd /dev/rdsk/c8t0d0




Create Device group

Add Physical Devices to

device group

Configuring SYMCLI SRDF Device Groups

Related devices are grouped into device groups

� All devices in a disk group must be in the same Symmetrix ICDA

� All devices must be the same type (RDF1, RDF2, Regular)

� A device can only belong to a single Device Group per SYMAPI database

A device group is a logical grouping of Symmetrix volumes. There are three types of device groups:

regular, rdf1, and rdf2. A device group with type regular cannot contain RDF volumes. Therefore,

users must create a device group with rdf1 or rdf2 for SRDF operations. The device group definition is

stored in the SYMAPI database on the host where the symdg create command was executed.


SRDF Operations - 4


Displaying SYMCLI Device Groups: Part 1

� symdg show displays detailed information about a device group

# symdg show srcdg

Group Name: srcdg

Group Type : RDF1 (RDFA)

Device Group in GNS : No

Valid : Yes

Symmetrix ID : 000187940398

Group Creation Time : Tue Mar 29 16:42:10 2005

Vendor ID : EMC Corp

Application ID : SYMCLI

Number of STD Devices in Group : 1

Number of Associated GK's : 0

Number of Locally-associated BCV's : 0

Number of Locally-associated VDEV's : 0

Number of Remotely-associated BCV's (STD RDF): 0

Number of Remotely-associated BCV's (BCV RDF): 0

Number of Remotely-assoc'd RBCV's (RBCV RDF) : 0


SRDF Operations - 5



� symdg show displays detailed information about a device groupStandard (STD) Devices (1):

--------------------------------------------------------------------

Sym Cap

LdevName PdevName Dev Att. Sts (MB)--------------------------------------------------------------------

DEV001 /dev/rdsk/emcpower182c 01EC RW 449

Device Group RDF Information

RDF Type : R1

RDF (RA) Group Number : 3 (02)

Remote Symmetrix ID : 000187940371

R2 Device Is Larger Than The R1 Device : False

RDF Mode : Synchronous

RDF Adaptive Copy : Disabled

RDF Adaptive Copy Write Pending State : N/A

RDF Adaptive Copy Skew (Tracks) : 65535

RDF Device Domino : Disabled

RDF Link Configuration : Fibre

RDF Link Domino : Disabled

Prevent Automatic RDF Link Recovery : Enabled

Prevent RAs Online Upon Power ON : Enabled

Device RDF Status : Ready (RW)


SRDF Operations - 6



Device Suspend State : N/A

Device Consistency State : Disabled

RDF R2 Not Ready If Invalid : Disabled

Device RDF State : Ready (RW)

Remote Device RDF State : Not Ready (NR)

RDF Pair State ( R1 <===> R2 ) : Synchronized

Number of R1 Invalid Tracks : 0

Number of R2 Invalid Tracks : 0

RDFA Information:

Session Number : 2

Cycle Number : 0

Number of Devices in the Session : 1

Session Status : Inactive

Session Consistency State : N/A

Minimum Cycle Time : 00:00:30

Average Cycle Time : 00:00:00

Duration of Last cycle : 00:00:00

Session Priority : 33

Tracks not Committed to the R2 Side: 0

Time that R2 is behind R1 : 00:00:00

R1 Side Percent Cache In Use : 0


� symdg show displays detailed information about a device group


SRDF Operations - 7


SRDF SYMCLI Commands

Syntax :

symrdf -g <group> <action> [options]

Where action can be one of the following:• Establish

• Restore

• Split

• Failover

• Failback

• Update

• Suspend

• Resume

Users can perform a number of Symmetrix SRDF operations using host-based SYMCLI commands.

Major SRDF operations include: ping, control, or modify operations on a device group; composite

group, device file, or on a device within a device or composite group; performs Dynamic RDF group

controls to add, modify, and remove a dynamic RDF group.

Please refer to EMC Solutions Enabler Symmetrix CLI Version 6.3 COMMAND REFERENCE P/N

300-000-877 REV A08 for the complete list of capability of the symrdf command.


SRDF Operations - 8


symrdf ping

symcfg -RA all list

Verifying SRDF Link Status

# symrdf ping

Successfully pinged (Remotely) Symmetrix ID: 000187940371

Successfully pinged (Remotely) Symmetrix ID: 000187940399

# symcfg -RA all list

Symmetrix ID: 000187940398 (Local)

S Y M M E T R I X R D F D I R E C T O R S

Remote Local Remote

Ident Symb Num Slot Type Attr SymmID RA Grp RA Grp Status

RF-1D 01D 49 1 RDF-BI-DIR - 000187940399 11 (0A) 11 (0A) Online

RF-2D 02D 50 2 RDF-BI-DIR - 000187940371 52 (33) 52 (33) Online

- 000187940371 62 (3D) 62 (3D)

- 000187940371 3 (02) 4 (03)

- 000187940371 12 (0B) 12 (0B)

RF-15D 15D 63 15 RDF-BI-DIR - - - - Online

RF-16D 16D 64 16 RDF-BI-DIR - - - - Online

# export SYMCLI_DG=srcdg

# symrdf ping

The symrdf ping command checks if RDF link communication is operational. The symcfg command

lists configuration information of all remote directors.


SRDF Operations - 9


Suspend SRDF Link

• Logically suspends mirror relationship between source and target volumes


# symrdf suspend DEV001

# symrdf suspend DEV001 -nop

An RDF 'Suspend' operation execution is in progress for device

'DEV001' in group 'srcdg'. Please wait...

Suspend RDF link(s).......................................Done.

The RDF 'Suspend' operation successfully executed for device

'DEV001' in group 'srcdg'.

# symrdf query

Device Group (DG) Name : srcdg

DG's Type : RDF1

DG's Symmetrix ID : 000187940398

Source (R1) View Target (R2) View MODES

-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A

Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair

Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE

-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC RW 0 0 NR 032C WD 0 0 E.. Suspended

Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

The symrdf suspend command stops data transfer between specified pairs. New writes to source

volume accumulate as invalid tracks.


SRDF Operations - 10


Resume SRDF Link

� Logically resumes mirror relationship between source and target volumes

# symrdf resume DEV001 -nop

An RDF 'Resume' operation execution is in progress for device

'DEV001' in group 'srcdg'. Please wait...

Resume RDF link(s)........................................Started.

Resume RDF link(s)........................................Done.

The RDF 'Resume' operation successfully executed for device

'DEV001' in group 'srcdg'.

# symrdf query DEV001


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC RW 0 0 RW 032C WD 0 0 S.. Synchronized

Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0


# symrdf suspend DEV001

The symrdf resume command resumes data transfer between pair. The Invalid tracks will start to

synchronize. However, remember that the serialization is not maintained during the synchronization.

Note: The #export SYMCLI_DG=srcdg command at the top of the slide. This command is setting the

SYMCLI_DG variable, to equal a specific device group name. Setting this enables the user to perform

Symrdf commands on a default device group.

Example:

Performing a query on the device group “srcdg” without the SYMCLI_DG variable set.

# symrdf –g srcdg query

Performing a query on the device group “srcdg” with the SYMCLI_DG variable set.

# symrdf query

Because the SYMCLI_DG variable has been set the command “symrdf query DEV001” is

performing a query on device “DEV001” within the device group “srcdg”.

Note – For this module, from this point on, the SYMCLI_DG variable has been set.




symrdf set mode

Changing SRDF Operational Mode

# symrdf set mode semi –nop

An RDF Set 'Semi-synchronous Mode' operation execution is in

progress for device group 'srcdg'. Please wait...

The RDF Set 'Semi-synchronous Mode' operation successfully executed

for device group 'srcdg'.

# symrdf query


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC RW 0 0 RW 032C WD 0 0 E.. Synchronized

Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0


# symrdf set mode semi

The symrdf set mode command changes SRDF operation mode. The –noprompt (-nop) is used to

bypass a confirmation question from the command line. This switch is applicable to most operations

with the symrdf command.




SRDF Volume Operational Status

symrdf query

# symrdf query


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------


Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

Legend for MODES:





# symrdf query

The name of the device group can be exported as a SYMCLI environment variable so that you have to

type it in each time.

The “query” shows the SRDF status of the device group.




Failover: symrdf failover

• Make copy of data on target Symmetrix volumes (R2) available to

attached hosts

• Disaster: host channel, Symmetrix or site failure

• Maintenance operation: provide data availability during host,

Symmetrix, or site maintenance

Update: symrdf update

• Begins synchronization prior to resuming operations on source volumes

Failback: symrdf failback

• Resumes operation using primary host and copy of data on source volumes (R1): Saves all changes made during failover

SRDF Disaster Recovery Operations

The disaster recovery operations for SRDF devices are:

•Failover from the source side to the target side, switching data processing to the target side.

•Failback from the target side to the source side by switching data processing to the source side.

•Update the source side after a failover while the target side may still be operational to its local host.




SRDF Failover

symrdf failover

# symrdf failover -nop

An RDF 'Failover' operation execution is

in progress for device group 'srcdg'. Please wait...

Write Disable device(s) on SA at source (R1)..............Done.


Read/Write Enable device(s) on RA at target (R2)..........Done.

The RDF 'Failover' operation successfully executed for

device group 'srcdg'.

# symrdf query

Device Group (DG) Name : srcdgDG's Type : RDF1DG's Symmetrix ID : 000187940398

Source (R1) View Target (R2) View MODES-------------------------------- ------------------------ ----- ------------

ST LI STStandard A N ALogical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF PairDevice Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC WD 0 0 NR 032C RW 0 0 S.. Failed Over

Total -------- -------- -------- --------MB(s) 0.0 0.0 0.0 0.0


# symrdf failover

In a period of scheduled downtime for maintenance, or after a serious system problem which has

rendered either the host or Symmetrix unit containing the source (R1) devices unreachable, no

read/write operations can occur on the source (R1) device. In this situation, the failover operation

should be initiated to make the target (R2) devices read/write enabled to their local host(s).




Updating Source Volumes symrdf update


# symrdf update –until 1000

# symrdf update -until 1000

Execute an RDF 'Update R1' operation for devic

group 'srcdg' (y/[n]) ?

An RDF 'Update R1' operation execution is



Merge device track tables between source and target.......Started.

Device: 01EC ............................................ Merged.

Merge device track tables between source and target.......Done.



The RDF 'Update R1' operation successfully initiated for


# symrdf queryDevice Group (DG) Name : srcdg

DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC WD 0 0 RW 032C RW 0 0 S.. R1 Updated

Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

While the target (R2) device is still operational (Write Enabled to its local host(s)), an incremental data

copy from the target (R2) device to the source (R1) device can be initiated in order to update the R1

mirror with changed tracks from the target (R2) device.




# symrdf failback -nop

An RDF 'Failback' operation execution is


Write Disable device(s) on RA at target (R2)..............Done.



Device: 01EC ............................................ Merged.




Read/Write Enable device(s) on SA at source (R1)..........Done.

The RDF 'Failback' operation successfully executed for


# symrdf queryDevice Group (DG) Name : srcdg

DG's Type : RDF1



-------------------------------- ------------------------ ----- -----------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- -----------


Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

SRDF Failback symrdf failback

A failback, or source (R1) device takeover, is performed when you are ready to resume normal SRDF

operations by initiating read/write operations on the source (R1) devices, and stopping read/write

operations on the target (R2) devices. The target (R2) devices become read-only to their local host(s)

while the source (R1) devices are read/write enabled to their local host(s).

Host activity should be stopped prior to execution:

� Stop Application

� Unmount file systems and deactivate volume groups

May be executed on either source or target Symmetrix.

Will abort if data integrity cannot be guaranteed.




� Split: symrdf split

• Places the Symmetrix units in a state for concurrent access

�Suspends link between source (R1) and target (R2) volumes

�Enables read and write operations on both source and target volumes

� To re-integrate SRDF Volumes:

• Save source data: symrdf establish

�Resume Normal SRDF operations

�Preserves data on the source (R1) volumes, discarding changes to the target (R2) volumes

• Save target data: symrdf restore

�Resume SRDF operations

�Preserves data on the target (R2) volumes, discarding changes to the source (R1) volumes

SRDF Decision Support/Concurrent Operations

The decision support operations for SRDF devices are:

•Establish an SRDF pair by initiating a data copy from the source side to target side. The operation can

be full or incremental.

•Restore remote mirroring. Initiates a data copy from the target side to the source side. The operation

can be full or incremental.

•Split an SRDF pair which stops mirroring for the SRDF pairs in a device group.




Concurrent Operations symrdf split


# symrdf split# symrdf split -nop

An RDF 'Split' operation execution is



Read/Write Enable device(s) on RA at target (R2)..........Done.

The RDF 'Split' operation successfully executed for


# symrdf query


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 01EC RW 0 0 NR 032C RW 0 0 S.. Split

Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

The command suspends the link between source (R1) and Target (R2) volumes. It also enables read

and write operations on both source and target volumes. Changes to source are kept track of as R2

invalids. Changes to target are kept track of as R1 invalids.




# symrdf establish -nop

An RDF 'Incremental Establish' operation execution is






Device: 01EC ............................................ Merged.



The RDF 'Incremental Establish' operation successfully initiated for


# symrdf query


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------


Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

Concurrent Operations: Saving Source Data

symrdf establish

Resume SRDF operation retaining data from source and overwriting data on target.




Concurrent Operations: Saving Target Data # symrdf restore –nop

An RDF 'Incremental Restore' operation execution is


Write Disable device(s) on SA at source (R1)..............Done.




Device: 01EC ............................................ Merged.




Read/Write Enable device(s) on SA at source (R1)..........Done.

The RDF 'Incremental Restore' operation successfully initiated for


# symrdf query -i 5 -c 5Device Group (DG) Name : srcdg

DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------


Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.

symrdf restore

Resumes SRDF operation retaining data on target and overwriting data on source.




Continuous Monitoring

symrdf query -i 5 -c 5


# symrdf query -i 5 -c 5


DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------


Total -------- -------- -------- --------

MB(s) 0.0 0.0 0.0 0.0

Status of RDF Volume is a specific device group:

[-i] interval

[-c] count

Provides summary of synchronization rate and estimated completion time.




# symcfg list -ra 4D -sid 11



Remote Local Remote


RF-4D 04D 52 4 RDF-R1 - 000187400093 27 (1A) 23 (16) Online

- 000187400093 28 (1B) 24 (17)

- 000187400093 29 (1C) 25 (18)

- 000187400093 30 (1D) 26 (19)

- 000187400093 31 (1E) 27 (1A)

- 000187400093 58 (39) 58 (39)

Add / Remove Dynamic RDF Groups

# symrdf -v addgrp -label grp58 -rdfg 58 -sid 187400011 -dir 4D \

-remote_rdfg 58 -remote_sid 000187400093 -remote_dir 3C -noprompt

# symrdf -v removegrp -label grp58 -noprompt

The symrdf addgrp command creates an empty dynamic RDF group that represents another RDF link

between Symmetrix 000187400011 and Symmetrix 000187400093. It adds dynamic RDF group 58 on

the local Symmetrix, and RDF group 58 on the remote Symmetrix. You must specify a group label

(grp58 in this case) that can be used when modifying or deleting the group. Creation of the dynamic

RDF group includes director 4D from the local Symmetrix and 3C from the remote Symmetrix as the

director end points of this connection. The symrdf removegrp command deletes the dynamic group.

The group must be empty to be deleted.

It is important to be aware of your network topology when creating dynamic RDF groups between two

Symmetrix arrays. To create a dynamic RDF link (a connection) between RA directors, the director

end points must be able to see each other through the Fibre Channel fabric. For example, a dynamic

RDF link can be created between local and remote directors only if the Fibre Channel zoning is set up

so that the two directors can see each other through the fabric.




Dynamic RDF Pair Requirement

� Make sure both Symmetrix arrays have the dynamic

setting enabled# symconfigure –sid 296 list -v


Configuration Server Version : 5671.32.36

Configuration Server Protocol : 0x509

Configuration Server Date : 03.29.2005

(TRUNCATED……)

Dynamic RDF Configuration : Enabled

RAID-S support : Enabled

(TRUNCATED……)

# symcfg –sid 296 list -v# symcfg –sid 296 list -v

Switched RDF Configuration State : Enabled

Concurrent RDF Configuration State : Enabled

Dynamic RDF Configuration State : Enabled

Concurrent Dynamic RDF Configuration : Enabled

RDF Data Mobility Configuration State: Disabled

Since Enginuity 5568, devices can be configured to be Dynamic RDF-capable devices. Dynamic RDF

functionality enables you to create, delete, and swap SRDF pairs while the Symmetrix array is in

operation. Using Dynamic RDF technology, you can establish SRDF device pairs from non-configured

SRDF devices, then synchronize and manage them in the same way as configured SRDF pairs.

Note:

Running the “symcfg list” command to check for concurrent RDF configuration.




Pairing Dynamic Devices

� Determine source and target dynamic devices for pairing

symdev list –dynamic both

� Create a device file and list source and target pairs

09C 054

09D 055

09E 056

� Issue symrdf createpair against this file

� Upon execution of this command, pairing information

will be added to the SYMAPI database file on the host

Using Dynamic RDF technology, you can establish SRDF device pairs from non-SRDF devices using

the symrdf createpair command. Once established, the new SRDF pairs can be synchronized and

managed in the same way as configured SRDF pairs.

Prior to Enginuity version 5568, SRDF device pairing was limited to the static SRDF pairs set at

Symmetrix configuration time. Dynamic RDF enables the creation and deletion of SRDF pairs while

the Symmetrix array is in operation.

The symrdf deletepair command is used to cancel the Dynamic SRDF pairing.




Dynamic SRDF Options

� Establish option:• Invalidates R2s, merges track tables, brings up RDF links, starts copy process from

R1 to R2

symrdf createpair –file file –sid 01 –type rdf1 -rdfg 2 –establish

� Restore option:• Invalidates R1s, merges track tables, brings up RDF links, starts copy process from

R2 to R1

symrdf createpair –file file –sid 01 –type rdf1 -rdfg 2 –restore

� Invalidate option:• Allows creation of dynamic SRDF pairs, but does not bring up the RDF links and

initiate data copy� To perform an establish, use -invalidate r2

� To perform a restore, use -invalidate r1

symrdf createpair –file file –sid 01 –type rdf1 -rdfg 2 –invalidate r2




Deleting Device Pairings

� Removes the pairing information from the Symmetrix

� Must suspend RDF links using before issuing symrdf deletepaircommand (link state must be NR and pair state is Suspended, Split, or FailedOver):

symrdf -f <devicefile> suspend -sid 01 -rdfg 2

symrdf deletepair -f <devicefile> -sid 01 -rdfg 2

� Dynamic SRDF pairs can also be cancelled within the context of adevice group

symrdf deletepair –g sargrp

� Canceling dynamic SRDF pairings changes the type of the device group from RDFx to Regular

� Devices in the device group are changed from DRx devices to RDF-capable standard devices and the SYMAPI database is updated




What is Consistency Technology?

� Consistent Split Ensures

consistency of applications in the same

Symmetrix

� Composite Groups

Ensures consistency of multiple databases and

platforms across

Symmetrix over

distances

SRDF

Oracle

SRDF

UDB

UDBUDB

UDB

BCV

BCV

Oracle

BCV

Composite Group

Consistent split allows you to avoid inconsistencies and restart problems that can occur if you split a

database-related BCV without first quiescing the database. TimeFinder provides the capabilities to

split off a consistent, restartable copy of a database with negligible impact on the production

environment.

A Composite Group is a user-defined group of SRDF devices that act in unison to maintain the

integrity of a database distributed across multiple Symmetrix units or multiple RDF groups within a

single Symmetrix. If a source R1 device in the Composite Group cannot propagate data to the target

R2 device, data propagation from all R1 devices in the Composite Group is halted. This suspension is

called tripping the Composite Group.




What Are Composite Groups?

� A Composite Group is a user-defined group of devices that act in unison to maintain the integrity of a database distributed across multiple Symmetrix units or multiple RDF groups within a single Symmetrix

� It allows you to remotely mirror your largest databases and automatically split off a consistent, DBMS restartable copy of the database in seconds, with no interruption to online service

� Disaster restart solutions using Composite Groups provide remote restart with a short restart time and zero data loss

� A composite group with remote consistency protection enabled is known a Consistency Group

Composite group is another logical grouping of Symmetrix devices. There are three types of composite

groups: REGULAR, RDF1, RDF2. Remote consistency protection is applicable only to the RDF type

composite group. Consistency groups protect the consistency of one or more database management

system (DBMS) that span RDF groups during a disaster. A Consistency Group provides synchronous

disaster restart with zero data loss, even when databases span multiple hosts and Symmetrix.




Tripping the Consistency Group

� If a source R1 device in the Consistency Group cannot

propagate data to its corresponding target R2 device, the data propagation from all R1 devices in the Composite

Group is suspended

• Automatic Tripping: Occurs when one or more R1 source devices in

an enabled Composite Group cannot propagate data to their

corresponding R2 target devices. For example:

�All RDF links between the R1 and R2 might go down

�R2 device might fail

�RDF directors on the R1 side or R2 side might fail

• Manual Tripping: Occurs when you invoke the symrdf –cg

suspend or split command

Suspending or splitting the Consistency Group creates an on-demand, DBMS restartable copy of the

database on the R2 target side.

− symrdf –cg suspend

The R2 devices are in the write-disabled state at the end of the trip and cannot be accessed by target-

side hosts. (This maintains the consistency of the R2 database copy with the production copy on the

R1 side.)

− symrdf –cg split

The R2 devices are enabled for both reads and writes by the target-side hosts.




Problem: Rolling Disasters

R2(B)

R2(D)

HOST

R2(A)

R2(C)

R1(D)

R1(C)

R1(B)

R1(A)

DBMS

1

2

3

3

Data

ahead

of Log4

1. Rolling disaster begins1. Rolling disaster begins2. Log write1. Rolling disaster begins2. Log write3. Dependent data write

1. Rolling disaster begins2. Log write3. Dependent data write4. Inconsistent data

Log write

Data writeData write

The diagram shows that the log write cannot reach the remote Symmetrix because of a problem with

the SRDF link. Meanwhile, the data write has reached the remote Symmetrix via another SRDF link.

This is known as “data ahead of log” condition. Almost all database management systems restart from

this condition, without an error or any correction. The integrity of the data in the DBMS is

compromised and the data is inconsistent.




Solution: Enterprise SRDF Consistency Groups

1. Consistency protection enabled

2. Rolling disaster begins

3. Log write

4. Consistency “trip”

5. Suspend R1/R2

6. Data write

7. Restartable copy

R2(B)

R2(D)

HOST

R2(A)

R2(C)

R1(D)

R1(C)

R1(B)

R1(A)

DBMS2

3

6

4Suspend

R1/R2

Relationship

5

7DBMS

Restartable

CopyConGroup Started

Task

with Host Component

ConGroup definition

ECA

PowerPath

1

Log write

Data write

Now that the composite group is defined and consistency protection enabled, our rolling disaster

begins with the loss of the SRDF links from bottom source Symmetrix to the bottom target Symmetrix.

A sense code is sent back stating that the data from volume A could not be propagated to it’s target

side. The composite group started task on the mainframe or the SYMCLI/SYMAPI detects the sense

code and works with ECA or PowerPath to hold the I/O.

While the I/O is held, two I/Os are sent per Symmetrix. The first request sets the volumes in the

Composite Group in a suspend pending state for all volumes in the composite group, and the second

request suspends the relationship between the source and target volumes (R1 and R2s). The I/O is

released within milliseconds.

I/O continues to occur on the source host until the complete disaster happens. The DBMS or

application are not aware that we held the I/O and created a Dependent Write Consistent copy or

DBMS restartable copy of the data on the target side. We have simulated a local power failure at the

target side at the point of the beginning of the rolling disaster. After complete failure on the source

side, the target side can be restarted and the DBMS can be restarted, which provides transactional

consistency.




What Happens to the Host-Based Groups

� Moved into the inactive list

� To display the inactive SYMAPI database groups on the host:symdg list –inactive

symcg list -inactive

� To import specific inactive groups into GNS:symdg activate MyDG

symcg activate MyCG

� To import all inactive groups:symdg activateall

symcg activateall




SRDF Control – RDF Split Action

� Replication � SRDF Control

Select the Devices to “Split”Select the Devices to “Split”

Split option “Flags”Split option “Flags”

SRDF Control operations are performed by right clicking the RDF Device Group and choosing

Replication � SRDF Control.

The user selected a Split operation in the above slide. The same dialog box can be used for the other

actions like Failback, Failover, Establish etc.

The SRDF Control window has 2 pages. On page 1 the action (“Split” for example) and the device

pairs are chosen.

Page 2 allows the user to choose options that relate to a specific action and to execute the action via the

Finish button.




Devices GroupsDevices Groups

Devices Group SMC01 – In a “Split” StateDevices Group SMC01 – In a “Split” State

SMC SRDF – RDF Split Action (Cont.)

The above slide shows the RDF Device Group “SMC01” in a split state. Selecting the SMC01 device

group within the SMC Navigation Tree automatically presents the current status in the View Area

window.

Remember, the SMC GUI closely match Solutions Enabler CLI command, to include all basic

monitoring, configuration, and control for any attached Symmetrix arrays.




Module Summary


� RDF volumes with SYMCLI

� SRDF Device Groups configuration and display properties

� Displayed and monitor the status of a Device Group

� SRDF Disaster Recovery operations using SYMCLI

� SRDF Link Control operations using SYMCLI

� SRDF Decision Support operations using SYMCLI

� Created and deleted Dynamic RDF pairs using SYMCLI

� Described Consistency Technology and its applications



SRDF/A (Asynchronous) - 1

© 2007 EMC Corporation. All rights reserved. SRDF/A (Asynchronous) - 1

SRDF/A (Asynchronous)


� List the technical requirements of a successful SRDF/A implementation

� Identify all supported SRDF/A hardware platforms

� Describe factors that affects RPO (Recovery Point Objective) in an SRDF/A implementation

� Discus Transmit Idle and DSE (Delta Set Extension)

� List and describe Cycles within SRDF/A operations

� Describe SRDF/A Consistent Deactivation

� Describe Multi Session Consistency

� Describe configuration parameters that affect SRDF/A behavior





What is SRDF/A?

� Asynchronous remote mirroring• Minimal impact to production

applications

• Extended distance

• Always consistent image on R2

� Efficient bandwidth usage

� Supports Mainframe and Open Systems

� Complements existing SRDF solutions • Meet a wide range of RPO and RTO

service level requirements

� Mixed SRDF and SRDF/A • Share links and directors

SRRD/A’s unique architecture delivers a remote mirroring solution that has no impact on production

applications over extended distance because I/O requests from the host are acknowledged locally.

Changes made to the same data blocks are periodically sent only once to the remote Symmetrix. This

enables significant operational savings through reduced bandwidth requirements. Moreover, SRDF/A

provides an alternative Disaster Recovery solution in addition to SRDF/S by maintaining a consistent

image of RDBMS on the R2 at all timess.

SRDF/A is a single solution supporting both Mainframe and Open Systems attaches. It also

compliments SRDF solutions to meet mixed service level requirements. In fact, it can also share the

same communication links as SRDF.




� All existing SRDF topologies• ESCON with Farpoint

• Fibre Channel

• IP

� Any host• Mainframe

• Open System

� All emulation types supported by Symmetrix, including:• FBA (Fixed Block Architecture)

• CKD (Count Key Data)

� Key operating systems, including:• UNIX (Sun, HP, IBM)

• Windows (NT, 2000, 2003)

• IBM Mainframe (OS/390, z/OS)

Note:

Support begins with Symm 5670 microcode and carries forward to future generations of Symmetrix

SRDF/A Supported Environments




Industry’s Traditional Write Ordering

Target

Source Location Target Location

1. Host I/O

Written To

Cache

Cache Cache

ALL WRITES

Write #1:

Write #2

Write #3

Write #4

Write #5

Write #6

Write #7

Write #3

Write #5

Write #2

Write #6

Write #4

Write #1

Write #7

3 3. Writes

Must Be Re-ordered

Before

Destaging

Write #1

Write #2

Write #3

Write #5

Write #7

Write #6

Write #4

2. EVERY

WRITE must be

time-

stamped /

ordered and

sent to the

Target side

Files

ALL WRITES

1

2

4

Traditional approaches to asynchronous mirroring have their architectural shortfalls.

� Uses a combination of cache and files to perform mirroring.

� Timestamps or sequence numbers are applied to each and every incoming write.

� Each write must have a timestamp applied to it before sending it to the remote side.

That means every single write MUST be sent to the remote side, and because they do not necessarily

arrive in order, they must be re-sequenced before being applied to disk. If you have writes number

100-200 pending at the remote side, all waiting for write number 99 to arrive, the system must re-

sequence and wait for number 99 to arrive before committing writes 100-200.

This creates a significant amount of overhead and data management activity in both the source and

target systems as they scramble to time-stamp, send, re-order, wait for a dependant time-stamp, and

then eventually commit the writes to disk at the target side.




Target Sym

SRDF/A Architecture and Ordered Cycles

Source HostSide A

Target HostSide B

Source(R1)

Device

Target(R2)

Device

CycleN

CycleN-1

CycleN-1

CycleN-2

Capture New WriteCycle -(N)

Transmit to R2Cycle -(N-1)

Receive Writes on R2Cycle -(N-1)

Write Applied to R2Cycle -(N-2)

SRDF/A Delta Set Begins

SRDF/A Delta Set

SRDF/A Delta SetSource Sym

When the SRDF/A cycle (N) is active on the source Symmetrix, it collects any new writes in the R1

Symmetrix cache, overwriting any duplicate tracks intended for data transfer over the link. The cycle is

active for a pre-determined amount of time that can be configured on the Symmetrix at the time of the

initial configuration of the SRDF/A environment; the default time is 30 seconds. After the set time has

been reached, the delta set data inherits the next cycle position (N-1) and begins transferring the delta

set over the link to the R2. Then, a new cycle N begins collecting new writes again for the next delta

set transfer.

In cycle (N-1), the delta set is temporarily collected on the R2 side for destaging. When the (N-1) cycle

has finished transferring data into the R2 and the minimum cycle time has elapsed, the delta set

inherits the next cycle position (N-2) and begins destaging the data to the R2 storage devices. The delta

set data is considered committed to the R2 in cycle (N-2).

Thus, it takes two(2) cycles for the changes from R1 to get to the R2 which make the shortest RPO of

an SRDF/A environment to be twice the cycle. That is, if the cycle time is 30 seconds, the RPO is at

least 60 seconds.




Dependent Write Consistency

� Dependent write logic:

• If ‘A’ is a predecessor and ‘B’ is a dependent write

• Any I/O ‘B’ that arrives after I/O ‘A’ has been acknowledged to the

host, must be dependent on ‘A’

• Implemented by the application, such as RDBMS’s

� SRDF/A ensures that:

• ‘A’ and ‘B’ are in the same Delta Set or

• ‘B’ is in later Delta Set

All commonly used database management systems are inherently dependent write consistent. For

instance, a DBMS will not perform a log write, indicating that a transaction is complete, until it has

received an acknowledgement from the storage subsystem that the log data pertaining to the

transaction itself was completely written to disk. Symmetrix honors this logic in SRDF/A by treating

any successor I/O , which arrives after a predecessor I/O as a dependent I/O.




Verifying the Environment

� List the configuration of the Symmetrix to verify that

relevant System Attributes have been set

• symcfg list -v –sid xxx

� List the Remote Adapters configured on the Symmetrix

and verify their status

• symcfg list –ra all

� List the RDF Groups that have been created on the

Symmetrix

• symcfg list –rdfg all

• symrdf list –rdfg RDF_Group_Number




System Attributes

DMX8HP1[ksh] # symcfg list -v -sid 35|more


-Output Truncated-

Cache Size : 16384 (MB)

# of Available Cache Slots : 475943

Max # of System Write Pending Slots : 110000

Max # of DA Write Pending Slots : 55000

Max # of Device Write Pending Slots : 10340

-Output Truncated-

SDDF Configuration State : Enabled

Configuration Change State : Enabled

-Output Truncated-

Switched RDF Configuration State : Enabled

Concurrent RDF Configuration State : Enabled

Dynamic RDF Configuration State : Enabled

Concurrent Dynamic RDF Configuration : Enabled

-Output Truncated-

SRDF/A Maximum Host Throttle (Secs) : 0

SRDF/A Maximum Cache Usage (Percent) : 94

As noted earlier, SRDF/A is supported under all topologies. However, if the attributes indicated in bold

are also set, then the end user can create Dynamic RDF Groups, create dynamic pairs of SRDF

devices, and perform Concurrent RDF operations. The switched RDF Configuration State must be

enabled from the service processor. The other attributes can be set using Symmetrix Configuration

Manager command – symconfigure. The use of Host Throttle, and Maximum Cache Usage attributes

are explained later in this module.




Configured Remote Adapters

DMX8HP1[ksh] # symcfg list -ra all|more

Symmetrix ID: 000187910035 (Local)


Remote Local Remote


RF-1D 01D 49 1 RDF-R1 - 000187910156 1 (00) 1 (00) Online

RF-2D 02D 50 2 RDF-R1 - 000187910156 1 (00) 1 (00) Online

RF-15D 15D 63 15 RDF-R1 - 000187910156 1 (00) 1 (00) Online

RF-16D 16D 64 16 RDF-R1 - 000187910156 1 (00) 1 (00) Online

Symmetrix ID: 000187910156 (Remote)


Remote Local Remote


RF-1D 01D 49 1 RDF-R2 - 000187910035 1 (00) 1 (00) Online

RF-2D 02D 50 2 RDF-R2 - 000187910035 1 (00) 1 (00) Online

RF-15D 15D 63 15 RDF-R2 - 000187910035 1 (00) 1 (00) Online

RF-16D 16D 64 16 RDF-R2 - 000187910035 1 (00) 1 (00) Online

The output has been truncated and formatted to show just the relevant information. As displayed, the

pair of Symmetrix units have four Remote Adapters each – 1D, 2D, 15D, 16D. Currently all four are

Online (Status).




List of All RDF Groups

DMX8HP1[ksh] # symcfg list -rdfg all


S Y M M E T R I X R D F G R O U P S

Local Remote Group RDFA Info

------------- -------------------- ----------------------- -----------------

LL Flags Dir Flags Cycle

RA-Grp (sec) RA-Grp SymmID T Name LPD Cfg CSR time Pri

------------- -------------------- ----------------------- ----- ----- ---

1 ( 0) 10 1 ( 0) 000187910156 S RDFDVGR00 .X. F-S -IS 30 33

2 ( 1) 10 2 ( 1) 000187910156 S RDFDVGR01 .X. F-S -IS 30 33

3 ( 2) 10 3 ( 2) 000187910156 D vsrdfag3 XX. F-S -IS 30 33

4 ( 3) 10 4 ( 3) 000187910156 D vsrdfag4 XX. F-S XAS 30 33

-Remote Symmetrix Output Truncated-

The output shows the pairings between the local and remote RA Groups (RDF Groups). These pairings provide the logical

connection between the two Symmetrix. For example, SRDF devices in RA Group 1 in sid 35 have their remote mirrors in

RA Group 1 in sid 56. RA Groups 1 and 2 are of the Type Static. These were created by the CE from the service processor.

RA Groups 3 and 4 are of the Type Dynamic. These were created by the user from the command line (SYMCLI). The

Directors have been configured in Fibre Channel Switched mode (F-S). RA Group 4 is in SRDF/A Active state (XAS), and

consistency has been enabled for this group.

Legend:

Group (T)ype : S = Static, D = Dynamic

Group Flags :

Prevent Auto (L)ink Recovery : X = Enabled, . = Disabled

Prevent RAs Online Upon (P)ower On: X = Enabled, . = Disabled

Link (D)omino : X = Enabled, . = Disabled

Director (C)onfig : F-S = Fibre-Switched, F-H = Fibre-Hub

G = GIGE, E = ESCON, T = T3, - = N/A

RDFA Flags :

(C)onsistency : X = Enabled, . = Disabled, - = N/A

(S)tatus : A = Active, I = Inactive, - = N/A

(R)DFA Mode : S = Single-session, M = MSC, - = N/A




List of an Individual RDF Group

DMX8HP1[ksh] # symrdf list -rdfg 3


Local Device View

-------------------------------------------------------------------------

STATUS MODES RDF S T A T E S

Sym RDF --------- ----- R1 Inv R2 Inv ----------------------

Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair

---- ---- ------ --------- ----- ------- ------- --- ---- -------------

000B 000B R1:3 RW RW RW S.. 0 0 RW WD Synchronized

000C 000C R1:3 RW RW RW S.. 0 0 RW WD Synchronized

Total -------- --------

Track(s) 0 0

MB(s) 0.0 0.0

Legend for MODES:




The list of devices in RDF Group 3 are displayed here. The two devices 00B and 00C are members of

the RDF Group. They are in Synchronous mode of SRDF operations.




Transition to SRDF/A

� From Synchronous

• If the devices are in Synchronized state, then by definition the R2

devices have a consistent image. Enabling SRDF/A immediately

provides consistent data on the R2

� From Adaptive Copy Disk

• Any invalid tracks owed to the R2 are synchronized. Two cycle

switches after Synchronization, SRDF/A provides consistent data on

the R2

� From Adaptive Copy Write Pending

• Write pending slots are merged into the Active SRDF/A cycles. When there are no more write pending slots, it takes an additional two cycle switches before R2 data is consistent

SRDF/A can be enabled when the device pairs are operating in any of the listed modes. In the case of

Adaptive Copy to SRDF/A transitions, it takes two additional cycle switches after resynchronization of

data for the R2 devices to be consistent.




Example: Synchronous to SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query

Device Group (DG) Name : vsrdfadg3

DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 0 RW 000B WD 0 0 S.. Synchronized

DEV002 000C RW 0 0 RW 000C WD 0 0 S.. Synchronized

Total -------- -------- -------- --------

Track(s) 0 0 0 0

MB(s) 0.0 0.0 0.0 0.0

The device pairs are operating in SRDF Synchronous mode (S..) and the pair states are Synchronized,

prior to enabling SRDF/A.

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C =

Adaptive Copy






Enabling SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 set mode async

DMX8HP1[ksh] # symrdf –g vsrdfadg3 enable

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa

RDFA Session Number : 2

RDFA Cycle Number : 2

RDFA Session Status : Active



RDFA R1 Side Percent Cache In Use : 0



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A


Device Dev E Tracks Tracks S Dev E Tracks Tracks MDAC STATE

-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 0 RW 000B WD 0 0 A..X Consistent

DEV002 000C RW 0 0 RW 000C WD 0 0 A..X Consistent

Transition to SRDF/A is immediate (A..X) and the pair state is immediately Consistent.

Legend for MODES:

M(ode of Operation): A = Async, S = Sync, E = Semi-sync,

C = Adaptive Copy



C(onsistency State): X = Enabled, . = Disabled, - = N/A




Example: ACP_Disk to SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query

Device Group (DG) Name : vsrdfadg3

DG's Type : RDF1



-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 151 RW 000B WD 0 0 C.D SyncInProg

DEV002 000C RW 0 103 RW 000C WD 0 0 C.D SyncInProg

Total -------- -------- -------- --------

Track(s) 0 254 0 0

MB(s) 0.0 7.9 0.0 0.0

In this example, the device pairs are operating in SRDF Adaptive Copy Disk Mode (C.D). There are a

number of invalid tracks owed to the R2 devices. This is governed by the skew value that has been set.

Legend for MODES:


C = Adaptive Copy






Enabling SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 set mode async

DMX8HP1[ksh] # symrdf -g vsrdfadg3 enable










-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 0 RW 000B WD 0 0 A..X SyncInProg

DEV002 000C RW 0 0 RW 000C WD 0 0 A..X SyncInProg

Note that the transition into SRDF/A is immediate (A..X), and the group has been enabled for

consistency. However, the pair state is SyncInProg. R2 device does not have consistent data until the

invalid tracks owed have been resynchronized, and a further two cycle switches have occurred.

Legend for MODES:








Consistent R2 DataDMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa









-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------



In this example, all invalid tracks were synchronized within the first cycle. Consequently, by the third

cycle the RDF Pair STATE has become Consistent. Tracks not Committed to the R2 Side is a measure

of data in Capture, Transmit, and Receive cycles at the time of the query.

Legend for MODES:








Display Devices with SRDF/A Enabled

DMX8HP1[ksh] # symdev show 0b|more

RDF Information:

RDF Mode : Asynchronous

RDF Pair State ( R1 <===> R2 ) : Consistent

RDFA Information:

Session Number : 2

Cycle Number : 8

Number of Devices in the Session : 2

Session Status : Active

Session Consistency State : Enabled

Minimum Cycle Time : 00:00:30

Average Cycle Time : 00:00:29

Duration of Last cycle : 00:00:29

Session Priority : 33





When devices are in SRDF/A enabled state, the display includes their RDFA information. Session

Number represents the RDF group number.




SRDF/A Consistent Deactivation from Async to Sync

R1R1

T

C

R2R2

A

R

C – Capture

T – Transmit

R – Receive

A - Apply

5671 Environment

R1R1

T

C

R2R2

A

R

Host

Host

Phase 2

R1R1

T

C

R2R2

A

R

Host

Phase 3

CacheCache

Asyn

Syn

Transition request

Phase 1

Switch

Switch

The Consistent Deactivation process works in three phases. In the first phase, SRDF/A operates

normally. When the request to transition from SRDF/A to synchronous is received, at the next cycle

switch, which guarantees an empty active cycle at the R1 side, a transition to the second phase occurs

In the second phase, new writes at the R1 side are sent directly in synchronous mode to the R2 side,

with one key exception. As these write arrive at the R2, they are kept in the inactive (receive) cycle at

the R2 side. And, the inactive (transmit) cycle at the R1 side continues to send data to the inactive

(receive) cycle at the R2. At the next cycle switch (two switches into the process), a transition to the

third phase occurs

From phase 2, when the next cycle switch is received (two cycle switches into the process), the

inactive cycle at the R2 side becomes the active cycle, and the SRDF/A restore process begins and

ends. At the end of the restore, when all tracks are marked write pending to the R2 devices, the

Consistent Deactivation is complete.




SRDF/A Consistent Deactivation: Limitations

1. The transition is not immediate

• It takes two cycle switches from request to completion

2. There will be some performance loss during the transition

• To host writes done during the transition

• Similar to host writes done during re-sync copy

3. Cache requirements

• During transition, all new writes are written to an existing inactive (receive) SRDF/A cycle at the R2 side

• For consistency, the data must be committed (restored) to the R2 all at once

• This may require an inactive (receive) cycle at the R2 which is ~2x the size as normal

4. Both Symmetrix systems must be running Enginuity 5x71

The limitations to this function are described above. Please note the additional cache which may be

required at the R2 side.

Because Enginuity changes are required at both R1 and R2 side Symmetrix systems, this is a 5x71 only

feature. This function fails if attempted on a 5x71 to 5670 SRDF connected Symmetrix pair.




SRDF/A: Loss of Link

� Temporary Loss of Link

• If all links are lost for a period less than 10 seconds, SRDF/A

remains in an Active state. Data continues to accumulate in cache

and consequently, the cycle time is elongated. However, R2 stays

consistent, and the relationship between R1-R2 is not suspended.

� Permanent Loss of Link

• In this event, all data in the Capture and Transmit cycles on the R1

side are changed from write-pending for the remote mirror to invalid

on the remote mirror. Any new data on the R1s is also be marked as invalid on the remote mirror. Likewise, all data in the Receive cycle on the R2 side are changed to invalid on the remote mirror. The

Apply cycle on the R2 side completes the commit to the local devices. The devices themselves are set to Not Ready on the link.

Data on the R2 devices is always consistent in SRDF/A, even with the loss of link. However, during

resynchronization, data is temporarily inconsistent on the R2 until all the invalid tracks have been sent

over to R2. For this reason, it is preferable to have a point-in-time copy on a BCV, for example, prior

to starting the resynchronization process. Resynchronization can be initiated by issuing symrdf

establish command.

The logical connection between R1 and R2 can be lost under several conditions. Some of them are

listed below:

•Network problems leading to loss of physical connection between source and target

•Symmetrix dropping the links due to link saturation

•User issued commands such as symrdf suspend/split




SRDF/A Reserve Capacity – Transmit Idle

� 5772 and later versions of 5x71 support Transmit Idle

� After links fail

– Data transmission from source to target will stop

– SRDF/A will remain active and the capture cycle will grow in cache

– Session will be suspended if cache fills up

� After links are revived, cycle switching will continue

� Not supported with ESCON RAs

� Should be enabled on both Source and Target side

� Should be used if using Delta Set Extension (DSE - 5772)

SRDF/A Transmit Idle is a Reserve Capacity enhancement to EMC’s SRDF/A feature that provides

SRDF/A with the capability of dynamically and transparently extending the Capture, Transmit, and

Receive phases of the SRDF/A cycle while masking the effects of an “all SRDF links lost” event.

Without the SRDF/A Transmit Idle enhancement, an “all SRDF links lost” event would normally

result in the abnormal termination of SRDF/A with either a CACA.20 or CACA.40 error. The SRDF/A

Transmit Idle enhancement has been specifically designed to prevent this event from occurring.

The versions of 5x71 which support Transmit Idle are 5771 Minimum release level of 92.99 with Epack# 1016 and

5671 Minimum release level 59.64 with Epack# 1017.




SRDF/A Reserve Capacity - Delta Set Extension

� Allows offloading of SRDF/A delta sets from cache to

specially configured device pools – Delta Set Extension Pools

� Intended to make SRDF/A resilient to temporary

increases in write workloads or link loss

• Should be used in conjunction with Transmit Idle

SRDF/A Delta Set Extension (DSE) provides a mechanism for augmenting the cache-based Delta Set

buffering mechanism of SRDF/A with a disk-based buffering ability. This extended Delta Set

buffering ability may allow SRDF/A to ride through larger and/or longer SRDF/A throughput

imbalances than would be possible with cache-based Delta Set buffering alone.




DSE Pools

� Save Pools are designated as DSE pools at creation

• Contains SAVE devices of a single emulation

�CKD3390, CKD3380, FBA or AS400

� Pools can be associated (shared) with multiple SRDF/A RDF groups

� Each SRDF/A session can be associated with zero or

one DSE pool of each type (e.g. FBA, 3390)

• Must have at least one DSE pool configured with a type that matches one of the device types in the SRDF/A group in order to activate DSE

� Pools can optionally start automatically when SRDF/A is

enabled

� SRDF/A DSE Pools and Save devices are managed in the same way as TimeFinder/Snap pools

� A RDF group can have at most one pool of each emulation

� A single rdfa_dse pool can be associated with more than one RDF group, similar to snap pools

shared by multiple snap sessions

� SRDF/A DSE Threshold sets the percentage of cache used for SRDF/A that will start offloading

cache to disk

� DSE must be enabled on both the source and target arrays. Extension on only one side of a link

would lead to failure of the SRDF/A recovery with SRDF/A dropping because the R2 side would

fail to have enough cache to hold the large and extended Transmit cycle




How DSE Fits Into SRDF/A Data Flow

� Traditional SRDF/A data flows

continue to interact directly with the cache buffer

� Separate DSE Task monitors Delta

Set cache utilization and transfers Delta Set data between cache and

disk

� Must be configured on R1 and R2

sides to be of value …

• and in all sessions tied together by MSC

Cache Buffer

Disk Buffer

DSE Task

SRDF/A has always buffered Delta Set data in cache. However a buffer full condition causes SRDF/A

to drop. With SRDF/A DSE delta set data is offloaded to disk buffers (DSE Pools) by the DSE task.

We will look at this in more details in the next few slides.




When can DSE help you?

� DSE is not designed to solve any permanent and persistent problems:• Mis-configurations:

�Unbalanced � cache or backend on target side is smaller/weaker than source side

�Not enough cache

�Not enough bandwidth

• Host writes consistently exceeding RDF link bandwidth

• Prolonged link outages

� SRDF/A DSE solves abnormal and temporary problems• Unexpected host load

• Link bandwidth issues

• Temporary link loss (use with Transmit Idle)

� Increase resilience of SRDF/A

SRDF/A DSE should be used with the Transmit Idle feature. Thus SRDF/A can ride through a temporary

link loss, once the DSE threshold is reached, data will be paged out to the DSE Pools.




Recovering after Loss of Links

� It is recommended to split a BCV copy of the R2 prior to

starting resynchronization

� In the event of an extended loss of link, a large number of

R2 invalid tracks can build up on the R1 side

� It is advisable to enable SRDF/A after the two sides are

synchronized

� Resynchronization prior to enabling SRDF/A can be

performed by:

• Setting SRDF mode to Adaptive Copy Write Pending

• Setting SRDF mode to Synchronous

As noted earlier, during resynchronization R2 does not have consistent data. A BCV copy of the

consistent R2 data prior to resynchronization can safeguard against unexpected failures during the

resynchronization process. When the link is resumed, if there are a large number of invalid tracks owed

by the R1 to its R2, it is recommended that SRDF/A not be enabled right away. Enabling SRDF/A

right after link resumption causes a surge of traffic on the link due to (a) shipping of accumulated

invalid tracks, and (b) the new data added to the SRDF/A cycles. This would lead to SRDF/A

consuming more cache and reaching the System Write Pending limit. If this happens, SRDF/A would

drop again. Like with SRDF/A, resynchronization should be performed during periods of relatively

low production activity.

Resynchronization in Adaptive Copy Write Pending mode minimizes the impact on the production

host. New writes are buffered and these, along with the R2 invalids, are sent across the link. The time

it takes to resynchronize is elongated.

Resynchronization in Synchronous mode impacts the production host. New writes have to be sent

preferentially across the link while the R2 invalids are also shipped. Switching to Synchronous is

possible only if the distances and other factors permit. For instance, if the norm is to run in SRDF/S

and toggle into SRDF/A for batch processing (due to higher bandwidth requirement). In this case, if a

loss of links occurs during the batch processing, it might be possible to resynchronize in SRDF/S.

In either case, R2 data is inconsistent until all the invalid tracks are sent over. Therefore, it is advisable

to enable SRDF/A after the two sides are completely synchronized.




Recovery Example




RDFA Session Status : Inactive


-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 407 NR 000B NA NA NA A..X Partitioned

DEV002 000C RW 0 406 NR 000C NA NA NA A..X Partitioned

In this example there is a workload on the devices in SRDF/A enabled state. A permanent loss of link

place the devices in a Partitioned state. Production work continues on the R1 devices and the new

writes arriving for the R1 devices are marked as invalid or owed to the R2.

Legend for MODES:








Recovery Example (Cont.)






-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 2431 NR 000B WD 285 0 A..X Suspended

DEV002 000C RW 0 2431 NR 000C WD 288 0 A..X Suspended

When the links are active again, note that the pair state has moved to Suspended. Also note there are

R2 Invalid Tracks on the R1 side AND R1 Invalid Tracks on the R2 side. In Synchronous SRDF, one

would have this condition only if changes are made to both the R1s and the R2s. However, in SRDF/A

data in Receive cycle during loss of link are marked as R1 invalids. The data in the Capture and

Transmit cycles, and new writes after link loss, are marked as R2 invalid on the R1 side. This is one of

the behavioral differences between SRDF/S and SRDF/A.

Legend for MODES:








Recovery Example (Cont.)






-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000B RW 0 3767 RW 000B WD 0 285 C.W. SyncInProg

DEV002 000C RW 0 0 RW 000C WD 0 0 C.W. Synchronized

As per recommendation, we place the devices in SRDF Adaptive Copy Write Pending mode (C.W.)

and wait for the pair states to become Synchronized. Prior to changing the mode to ACP_WP, we

have to disable consistency protection via symrdf –g vsrdfadg3 disable. When consistency is enabled,

one cannot switch out of SRDF/A without first disabling it. Once the synchronization is complete we

can then enable SRDF/A and enable consistency.

Legend for MODES:








SRDF Session Recovery Tool – symrecover

� EMC SRDF session recovery utility is initiated by the symrecover command

� Runs in the background and monitors the state of SRDF/A or SRDF/S sessions

� If failure is detected, automatic recovery and restart is attempted based on symrecover options file (pre-configured)

� The symrecover command can be run from either the R1 or the R2 side• As long as all the devices making up the group being monitored are fully

viewable from the host

• When concurrent RDF is used, this command must be run from the R1 side

• Solutions Enabler 6.4 or higher must be installed

The symrecover command was available through EMC Services prior to SE 6.4.

symrecover offers the ability to detect and automate restarting SRDF links in a safe and secure manner.




Failover/Failback with SRDF/A

� If the primary site fails, data on R2 is consistent up to the

last Apply cycle (N-2)

• Partial data in the Receive cycle is discarded

� SRDF failover procedure can then be executed and the

workload can be started on the R2 devices

• Consistency protection should be disabled prior to issuing symrdf

failover without the –force option

� Failback procedure after the primary site has been restored is identical to Synchronous SRDF

• After symrdf failback command completion, workload can be restarted on the R1 devices. SRDF/A can be enabled

Again, it is advisable to split off a BCV copy of the R2 prior to executing a failback operation. When

workload is resumed on the R1 devices immediately after a failback, accumulated invalid tracks have

to be synchronized from the R2 to the R1, and new writes must be shipped from the R1 to R2. If there

is an interruption now, data on the R2 is not consistent. Even though SRDF/A can be enabled right

after a failback, for reasons stated earlier, it should be enabled after the SRDF pairs entered the

Synchronized state.




Example: Multiple Independent SRDF/A Groups

DMX8HP1[ksh] # symcfg list -ra all -sid 35|more



Remote Local Remote


RF-1D 01D 49 1 RDF-R1 - 000187910156 1 (00) 1 (00) Online

- 000187910156 3 (02) 3 (02)

RF-2D 02D 50 2 RDF-R1 - 000187910156 1 (00) 1 (00) Online

- 000187910156 3 (02) 3 (02)

RF-15D 15D 63 15 RDF-R1 - 000187910156 1 (00) 1 (00) Online

- 000187910156 3 (02) 3 (02)

- 000187910156 4 (03) 4 (03)

RF-16D 16D 64 16 RDF-R1 - 000187910156 1 (00) 1 (00) Online

- 000187910156 3 (02) 3 (02)

- 000187910156 4 (03) 4 (03)

RA Group 3 uses all four RDF Directors – 1D, 2D, 15D, and 16D. RA Group 4 uses two of the RDF

Directors – 15D, and 16D




Example: Multiple Independent SRDF/A Groups(Cont.)

DMX8HP1[ksh] # symcfg list -rdfg all


S Y M M E T R I X R D F G R O U P S

Local Remote Group RDFA Info

------------- -------------------- ----------------------- -----------------

LL Flags Dir Flags Cycle

RA-Grp (sec) RA-Grp SymmID T Name LPD Cfg CSR time Pri

------------- -------------------- ----------------------- ----- ----- ---

1 ( 0) 10 1 ( 0) 000187910156 S RDFDVGR00 .X. F-S -IS 30 33

2 ( 1) 10 2 ( 1) 000187910156 S RDFDVGR01 .X. F-S -IS 30 33



Both the RA Groups 3 and 4 are in SRDF/A mode (XAS), as shown in the output above.









-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------



The devices in RA Group 3 are consistent, independently of the devices in RA Group 4 (shown in the

next slide).

Legend for MODES:


C = Adaptive Copy












-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000D RW 0 0 RW 000D WD 0 0 A..X Consistent

DEV002 000E RW 0 0 RW 000E WD 0 0 A..X Consistent

Devices in RA Group 4 are in SRDF/A Active state and are consistent. Note that RA Group 4 has a

different Cycle Number (62), than RA Group 3 (2972).

Legend for MODES:


C = Adaptive Copy












-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 000D RW 0 107 NR 000D NA NA NA A..X Partitioned

DEV002 000E RW 0 80 NR 000E NA NA NA A..X Partitioned

Loss of links on directors 15D, and 16D causes SRDF/A to drop for RA Group 4.

Legend for MODES:


C = Adaptive Copy












-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------



However, RA Group 3 still has 2 links available (1D, and 2D), and continues to be in SRDF/A Active

state and the device pairs are Consistent.

Legend for MODES:


C = Adaptive Copy







What is SRDF/A Multi Session Consistency?

� Manage multiple SRDF/A

sessions logically as if they were a single session

• RDF Daemon for Open System

support

• Sessions can be within or across

Symmetrix arrays

• Ensures a complete, restartable

point-in-time copy on the remote side

Delta Set

Delta

Set

Delta

Set

Since Enginuity 5671, consistency protection for SRDF/Asynchronous devices is provided using Multi

Session Consistency (MSC). If one or more source (R1) devices in an SRDF/A MSC enabled RDF

consistency group cannot propagate data to their corresponding target (R2) devices. The MSC process

suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2

targets SRDF/A with MSC supported by an RDF process daemon that performs cycle-switching and

cache recovery operations across all SRDF/A sessions in the group. This ensures that a consistent R2

data copy of the database exists at the point-in-time any interruption occurs.

A composite group must be created using the RDF consistency protection option (-rdf_consistency)

and must be enabled using the symcg enable command before the RDF daemon begins monitoring and

managing the MSC consistency group.




What is SRDF/A Multi Session Consistency? (Cont.)

� RDF Daemon coordinates cycle

switching of the SRDF/A MSC group sessions as a single

entity

• Responsible for detecting ‘failure’

conditions that would cause data on

the R2 side to become inconsistent

• When a ‘failure’ condition is detected,

the cycle switching for all SRDF/A sessions in the group are stopped in a manner that leaves the R2 side

with a consistent data image

Delta Set

Delta

Set

Delta

Set

The RDF process daemon maintains consistency for enabled composite groups across multiple

Symmetrix arrays for SRDF/A with MSC. For the MSC option (-rdf_consistency) to work in an RDF

consistency-enabled environment, each locally-attached host performing management operations must

run an instance of the RDF daemon (storrdfd). Each host running storrdfd must also run an

instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel

application syscalls. Optionally, if the Group Naming Services (GNS) daemon is also running, it

communicates the composite group definitions back to the RDF daemon. If the GNS daemon is not

running, the composite group must be defined on each host individually.




How Does MSC Cycle Switch Work?

� In single session, SRDF/A cycle switch occurs when the

Transmit cycle on the R1 side AND the Apply cycle on the R2 side are both empty. The switch is controlled by

Enginuity

� In MSC, the Transmit cycles on the R1 side of all

participating sessions must be empty, and also the corresponding Apply cycles on the R2 side. The switch is

coordinated and controlled by the RDF Daemon

� All host writes are held for the duration of the cycle

switch. This ensures dependent write consistency

If one or more sessions in MSC complete their Transmit and Apply cycles ahead of other sessions,

they have to wait for all sessions to complete, prior to a cycle switch.




MSC Cycle SwitchSymmetrix 1

Session 1

C(1)

T(1)

Symmetrix 2

Session 2

C(2)

T(2)

Symmetrix 3

Session 1

R(1)

A(1)

Symmetrix 4

Session 2

R(2)

A(2)

MSC/CG

RDF Daemon

Host

RDF Links

(N)

(N-1)

(N-1)

(N-2)

(M)

(M-1)

(M-1)

(M-2)

Tag a

Tag a-1

Tag a-1

Tag a-2

Tag a

Tag a-1

Tag a-1

Tag a-2

In this illustration we have two SRDF/A sessions, 1 and 2. Each have their own cycle numbers N and

M. When they are placed in MSC, the RDF Daemon assigns a Tag number to the capture cycle. This is

so the cycle numbers themselves do not have to be synchronized. The Tag number is incremented at

every cycle switch. It is this Tag number that is compared for recovery purposes.




SRDF/A MSC Operations

� Set SYMAPI_USE_RDFD = ENABLE in options configuration file

� Create a Composite Group (CG) with the -rdf_consistency option

• Group definition is passed to the RDF Daemon as a ‘candidate group’

• If the Daemon is not already running, it is started automatically

� Add all of the devices in the multiple SRDF/A sessions to the CG

� Put all CG devices into Async modesymrdf -cg <CGname> set mode async

� Enable CG devices for consistency protectionsymcg -cg <CGname> enable

• The RDF Daemon is notified that the group should now be monitored

• Enable command must be done after the devices are put into Async mode

� When the devices become RW on the link, the RDF Daemon:

• Starts performing cycle switching

• Actively monitors the health of the group to maintain R2 data consistency

There are three ways the RDF daemon can be started. If the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache.

Note:

Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the information stored in it to establish contact with your Symmetrix arrays. Alternatively, the daemon can be started manually via the stordaemon command line utility as follows: - stordaemon start storrdfd [-wait Seconds] Note: The stordaemon command requires a path of /usr/storapi/storbin.

By default, the stordaemon command waits 30 seconds to verify that the daemon is running. To override this, use the -wait option. Additionally, the daemon can be set to start automatically every time the local host is booted using the following command line: - stordaemon install storrdfd –autostart. Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load. If the daemon is stopped for some reason, it can optionally be restarted automatically by an internal Solutions Enabler watchdog mechanism. A combination of the watchdog mechanism and the auto-start option described above can be used to ensure that the daemon is always running. To stop the RDF daemon, use the following command: - stordaemon shutdown storrdfd|all [-wait Seconds] Applying the “all” option stops all of the daemons currently running, such as storapid, storgnsd, and storrdfd.




RDF Daemon for MSC

The RDF process daemon maintains consistency for enabled composite groups across multiple

Symmetrix arrays for SRDF/A with MSC.(Multi Session Control) For the MSC option (-

rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host

performing management operations must run an instance of the RDF daemon (storrdfd). Each host

must also be running an instance of the base daemon (storapid), which coordinates all Symmetrix locks

and parallel application syscalls.

Additional data about the current state of a composite group is communicated to the RDF daemon via

files written to the Symmetrix file system. MSC requires that the RDF daemon exist and every attempt

is made to start or restart the daemon to perform cycle switching for SRDF/A. Failure to switch

SRDF/A cycles may cause all SRDF/A sessions to be dropped due to a full cache slot. If SRDF/A

sessions have been dropped, the SYMAPI and RDF daemon logic determines whether to commit or

discard the data accumulated in cache memory.

For redundant consistency protection of RDF composite groups, multiple instances of the RDF daemon

can be running at the same time on separate hosts. Each host must have a common view of the

composite group being monitored. All redundant daemons run simultaneously, monitoring and

switching independently of each other. If one of the redundant daemons fails, the other existing

daemon(s) completes the task.




Managing RDF Daemon

� Modify the SYMAPI options file

• SYMAPI_USE_RDFD=ENABLE

� Start Daemon• stordaemon start storrdfd

� Stop Daemon• stordaemon shutdown storrdfd

Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since

storrdfd uses the stored information to establish contact with your Symmetrix arrays.

There are three ways the RDF daemon can be started. First, if the RDF daemon is enabled, the daemon

is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it,

which can cause a slight delay in performance on that initial connection while the daemon starts and

builds its cache.

Second, the daemon can be started manually via the stordaemon command line utility as follows:

stordaemon start storrdfd [-wait Seconds]

By default, the stordaemon command waits 30 seconds to verify the daemon is running. To override

this, use the -wait option.

Third, the daemon can be set to start automatically every time the local host is booted using the

following command line:

stordaemon install storrdfd -autostart

Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon

may take a while to initially construct its cache - depending on the number of groups and Symmetrix

arrays it has to load.




MSC: Example

DMX8HP1[ksh] # symcg show vsmscdg34|more

RDF Consistency Enabled : Yes

Number of RDF (RA) Groups : 2

Number of STD Devices : 4

Number of Symmetrix Units (1):

Number of RDF (RA) Groups (2):

----------------------------------------------------------------------------

Sym Device Flags Cap

LdevName PdevName Dev Config Sts CSR (MB)

----------------------------------------------------------------------------

DEV001 /dev/rdsk/c14t1d3 000B RDF1+Mir RW XAM 449

DEV002 /dev/rdsk/c14t1d4 000C RDF1+Mir RW XAM 449

----------------------------------------------------------------------------

Sym Device Flags Cap

LdevName PdevName Dev Config Sts CSR (MB)

----------------------------------------------------------------------------

DEV003 /dev/rdsk/c14t1d5 000D RDF1+Mir RW XAM 449

DEV004 /dev/rdsk/c14t1d6 000E RDF1+Mir RW XAM 449

We wish to control RA Groups 3 and 4 as one entity. A composite group has been created and enabled

using the following commands:

− symdg dg2cg vsrdfadg3 vsmscdg34 –rdf_consistency

− symdg dg2cg vsrdfadg4 vsmscdg34 –rdf_consistency –rename

− symcg –cg vsmscdg34 enable

Note the CSR flags, indicating that Multi-session Consistency has been enabled for these two RA

Groups.

Legend:

RDFA Flags:

C(onsistency) : X = Enabled, . = Disabled, - = N/A

(RDFA) S(tatus) : A = Active, I = Inactive, - = N/A

R(DFA Mode) : S = Single-session mode, M = MSC mode, - = N/A




MSC: Loss of Links for One RA Group

DMX8HP1[ksh] # symrdf -cg vsmscdg34 query -rdfa|more

MSC Session Status : Inactive

Consistency State : N/A


-------------------------------- ------------------------- ----- ------------

ST LI ST

Standard A N A

Logical Sym T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair


-------------------------------- -- ----------------------- ----- ------------

DEV001 000B RW 0 76 NR 000B WD 0 0 A..X Suspended

DEV002 000C RW 0 61 NR 000C WD 0 0 A..X Suspended

DEV003 000D RW 0 79 NR 000D NA NA NA A..X Partitioned

DEV004 000E RW 0 63 NR 000E NA NA NA A..X Partitioned

Similar to the previous example (independent), we have a total loss of links for RA Group 4. The

devices in this group go to a Partitioned state as before. However, note that the devices in RA Group 3

have been placed in Suspended state. Loss of links for one RA Group trips the Multi-session

Consistency Group. This prevents the propagation of data for the other RA Groups in the MSC-CG,

thus preserving a consistent image of data on ALL R2s at the time of link loss. As stated earlier, the

storrdfd daemon is responsible for coordinating cycle switches between the RA groups, to trip the

MSC-CG and perform any recovery/cache cleanup that might be necessary when the links are

resumed. Recovering from this state can be accomplished as usual:

symrdf –cg vsmscdg34 establish

Once the invalid tracks are marked, merged, and synchronized, MSC protection is automatically re-

instated. i.e User does not have to issue symcg –cg vsmscdg34 enable again.

Legend for MODES:








MSC Recovery and Cleanup

� There are three possible recovery scenarios in case of a

failure in MSC:

• All Receive cycles are marked as complete. In this case, the Receive

cycles are committed – i.e. promoted to Apply

• Some Receive cycles are marked as complete and others are

marked as incomplete. In this case, ALL Receive cycles are

discarded

• Some Receive cycles have been promoted to Apply, where as some

of them have not. In this case, the promoted Receive cycles and

those not yet promoted are committed

The first scenario is easy to understand. In this instance, the Receive cycles contain the most recent and

consistent data.

The second situation arises if there is a failure when some Receive cycles are complete while the

others are in transit. In this case, clearly it is only the Apply cycles of all sessions that contain the

consistent data. Therefore, ALL Receive cycles are discarded.

To understand the third scenario, keep in mind the following: For a cycle switch to be initiated, ALL

Transmits must be empty and all Apply’s must be empty. This means that the failure has occurred

DURING the cycle switch process in this case. Receive can only be promoted to Apply on a cycle

switch. In MSC, cycle switch is sent to all sessions at once. So each of them is in the process of

executing a cycle switch. For example, failure occurred prior to promoting some Receives to Apply.

Therefore, the Receives not yet promoted should be committed along with those that have already been

promoted.




MSC Recovery and Cleanup

� Recovery is automatically performed by the RDF Daemon

on the R1 side, if the link to the R2 side is available

� If the link is unavailable (total site failure on the R1 side),

then invocation of any SRDF command, such as symrdf

failover or split, from the R2 side performs the automatic

cleanup and recovery




SRDF/A Configuration Parameters

� 2 SRDF/A Symmetrix Array-wide parameters• Maximum SRDF/A Cache Usage

• Maximum Host Throttle Time

� 2 RDF Group (aka SRDF/A Session) level settings• Minimum Cycle Time

• Session Priority

� Settable via the SYMCLI symconfigure command

• Set Symmetrix Metrics for Symmetrix level attributes

�set symmetrix rdfa_cache_percent = 94;

�set symmetrix rdfa_host_throttle_time = 0;

• Set RDF Group Metrics for Group level attributes�set rdf group 3, session_priority = 33;

�set rdf group 3, minimum_cycle_time = 30;




SRDF/A System Configuration Parameters

� rdfa_cache_percent

• Defaults to 94, with a range of valid values from 0 to 100 percent

• It is the percentage of the Max# of System Write Pending Slots available to SRDF/A. The purpose is to ensure that other applications can utilize some of the WP limit

• When SRDF/A hits its WP cache limit it will be forced to drop SRDF/A sessions to free up cache

• Setting it lower reserves WP limit for non-SRDF/A cache usage. Setting it higher, allows SRDF/A to potentially use more of the cache WP limit, potentially creating performance problems for other applications

� rdfa_host_throttle_time

• Defaults to 0, with a range of valid values from 0 to 65535

• If >0, this value overrides the rdfa_cache_percent and session_prioritysettings

• When the System WP Limit is reached, throttling will delay a write from the host until a cache slot becomes free

• The value is the number of seconds to throttle host writes before dropping SRDF/A sessions. A value of 65535 means wait forever

� Each Symmetrix has an array-wide Max # of System Write Pending Slots limit (generally

calculated as 80% of available cache slots).

� The purpose of this limit is to ensure that cache is not filled with Write Pending (WP) tracks,

potentially preventing fast writes from hosts, because there is no place to put the I/O in cache.

� SRDF/A creates WP tracks as part of each cycle.




SRDF/A Group/Session Configuration Parameters

� SRDF/A group minimum_cycle_time

• Defaults to 30, with a potential range of 5 to 59 seconds

� SRDF/A group session_priority

• Defaults to 33, with a range of valid values from 1 to 64.

• The number ranks the SRDF/A session relative to other sessions to

determine the order for dropping sessions should the cache WP limit

be reached

• When SRDF/A needs to drop sessions when the cache WP limit is

reached. The sessions are dropped, starting with priority values of 64. Those with a setting of 1 are last to be dropped




Monitoring SRDF/A

� Using the symstat command options• symstat –type cycle –reptype rdfa –rdfg all

• symstat –type cache –reptype rdfa –rdfg all

• symstat –type request –reptype rdfa –rdfg all

� Using the symevent command• symevent list -error

Examples are displayed in the next several slides.




Monitoring SRDF/ADMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5

SRDF/A Session Cache Summary Information

Symmetrix Id : 000187910035

Timestamp : 15:17:23

System Write Pending Limit : 110000 (3.36 GB)

Cache Slots available for all SRDF/A sessions : 103400 (3.16 GB)

Total Local Write Pending Count : 30690

Total System Write Pending Count : 57066

Session

RA ----------------------------- Cache Slots %Available

Device Group Name Grp Type Number Priority Status In Use Cache Used

----------------- --- ---- ------ -------- -------- ----------- ----------

RaGrpNum_01 1 - 0 33 Inactive 0 0.0

RaGrpNum_02 2 - 1 33 Inactive 0 0.0

RaGrpNum_03 3 RDF1 2 33 Active 42 0.0



Total ----------- ----------

Slots 57007 55.1

GB 1.74

Note that the Cache Slots available for all SRDF/A sessions is 94% of the System Write Pending Limit

(3.16/3.6). In this example, RA Group 5 is utilizing 55% of the available cache. All SRDF/A sessions

have the default Priority value of 33.




Monitoring SRDF/A (Cont.)

DMX8HP1[ksh] # symstat -type cycle -reptype rdfa -rdfg all -i 5

SRDF/A Session Cycle Summary Information



Session Cycle Time (sec) Cycle Size

------------------ ------------------- ---------------

RA Active Last

Device Group Name Grp Type Number Cycle# Min Avg Last Switch Active Inactive

----------------- --- ---- ------ ------ --- --- ---- ------ ------ --------

RaGrpNum_01 1 - 0 0 - 0 0 - 0 0

RaGrpNum_02 2 - 1 0 - 0 0 - 0 0

RaGrpNum_03 3 RDF1 2 286 30 30 30 27 48 0

RaGrpNum_05 5 RDF1 4 34 30 29 29 3 6935 32589

RaGrpNum_04 4 RDF1 3 286 30 30 30 27 51 0

Legend for the Attribute of Cycle Size:

RDF1: Active = Capture Inactive = Transmit

RDF2: Active = Receive Inactive = Apply

This output was captured from the R1 side. As displayed, the Minimum cycle time for each of the

SRDF/A session is at the default of 30 seconds. The Active Cycle is the Capture and the Inactive is the

Transmit, as this output is from the R1 (source) perspective.




When Max Cache is used up…

DMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5






Total Local Write Pending Count : 37725

Total System Write Pending Count : 55153

Session



----------------- --- ---- ------ -------- -------- ----------- ----------




-Next iteration of the symstat command-


RaGrpNum_05 5 RDF1 4 33 Inactive 44308 80.6


For this example, the maximum available cache slots for SRDF/A has been reduced to 50%

(1.68/3.36). Now, cache utilization reaches a 100% of available cache, and RDFG 5 is dropped

(Inactive).




When the Session is dropped…

DMX800SUN1[ksh] # symrdf -g vsrdfadg query -rdfa





-------------------------------- ------------------------ ----- ------------

ST LI ST

Standard A N A



-------------------------------- -- ------------------------ ----- ------------

DEV001 0041 RW 0 0 NR 0041 WD 0 0 A... Suspended





As seen earlier, the Capture and Transmit data on the R1 side are marked as R2 invalids. The Receive

data on the R2 side is marked as R1 invalid. The pair state is suspended.

Legend for MODES:


C = Adaptive Copy







Monitoring SRDF/A – symevent

DMX8HP1[ksh] # symevent list -error


Time Zone : EDT

Detection time Dir Src Category Severity Error Num

------------------------ ------ ---- ------------ ------------ ----------

Fri Aug 19 11:55:24 2005 RF-16D Symm RDF Error 0x004e

SRDF/A Session dropped, no RDF links operational.

Fri Aug 19 15:39:15 2005 RF-15D Symm RDF Error 0x004e

SRDF/A Session dropped, no RDF links operational.

Fri Aug 26 16:29:48 2005 RF-2D Symm RDF Error 0x004a

SRDF/A Session dropped, write pending limit reached. Host throttling disabled.

The dropping of the SRDF/A session can also be displayed via the symevent command.




Giving Preference to Sessions via Priority Setting

DMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5





Session



----------------- --- ---- ------ -------- -------- ----------- ----------




Total ----------- ----------

Slots 90816 103.2

-Next iteration of the symstat command-



RaGrpNum_04 4 RDF1 3 33 Inactive 18010 20.5

In this instance, the Cache available for SRDF/A has been set to 80% (2.69/3.36). When 100% (103.2)

of this is used, we see that the session with the least priority is dropped first – RA Group 4 with a

priority of 33.




Factors that Affect RPO

Symmetrix Cache

SRDF BandwidthRPO

Workload

There are three factors that affects RPO of an SRDF/A implementation: SRDF bandwidth, Symmetrix

Cache, and Workload. During a SRDF/A cycle, new changes are captured in the local Symmetrix

cache before being sent via the SRDF link to the remote Symmetrix. The Symmetrix should have

enough cache to accommodate these changes occurring before the cycle switch time has elapsed. At

the same time, there should also be a sufficient bandwidth for SRDF link to push these changes to the

remote site. If there is not enough cache or bandwidth, SRDF/A may not be able to maintain the RPO

at twice the cycle time. The process to determine these three factors should involve EMC personnel

and is essential to a successful SRDF/A implementation.




Industry Bandwidth Estimates

Type of Link Price / Month

T1

T3

OC-3

OC-48

600

4,500

10,000

80,000

Dollars

. .

Industry Estimates of Bandwidth Pricing.




BANDWIDTH USAGE

0

5

10

15

20

25

30

35

23:45:00

7:45:00

8:45:00

9:45:00

10:45:00

11:45:00

12:45:00

13:45:00

14:45:00

15:45:00

16:45:00

17:44:59

18:45:00

19:45:00

20:45:00

21:45:00

22:45:00

TIME INTERVAL

Mbyte

per Second

COMPRESSED BANDWIDTH BANDWIDTH LIMIT

Example of Bandwidth, Cache, and Workload

� Interval Length is 15 minutes

� Peak Workload is for 6 intervals (or 90 minutes)

� Non-Peak Workload results in 30 second SRDF/A Cycles

Peak WorkloadNon-Peak Workload

In this example, there is 1 OC-3 Link (15.5 MB/s) with 2:1 Compression between the local and remote

site. The workload produces 6 peak writes above the bandwidth.




Cache Requirements for One OC-3 Link

� Cache Requirements are significant in order to cover the peak times

• Base Cache covers SRDF/A during normal operations (below b/w)

• Peak Cache is the significant contributing value

SRDF/A Cache Requirements

0.00

50.00

100.00

150.00

TIM

E

7:4

5:0

0

9:0

0:0

0

10

:14

:59

11

:30

:00

12

:45

:00

14

:00

:00

15

:14

:59

16

:30

:00

17

:44

:59

19

:00

:00

20

:15

:00

21

:30

:00

22

:45

:00

Time

Ca

ch

e (

GB

)

PEAK CACHE (GB/interval) BASE CACHE (GB/30 sec cycle)

The single OC-3 bandwidth is not sufficient to push the data surge during peak writes. As a result, the

data is accumulating the cache on the local Symmetrix. This drives the cache requirement up to around

80GB.




The RPO Impact with One OC-3 link

� Providing you can configure enough cache, SRDF/A works perfectlyfine with less than Synchronous bandwidth

� This, as depicted below, directly affects the RPO

SRDF/A Recovery Point Objective (RPO)

0.00

200.00

400.00

600.00

800.00

1000.00

TIM

E

7:4

5:0

0

9:0

0:0

0

10:1

4:5

9

11:3

0:0

0

12:4

5:0

0

14:0

0:0

0

15:1

4:5

9

16:3

0:0

0

17:4

4:5

9

19:0

0:0

0

20:1

5:0

0

21:3

0:0

0

22:4

5:0

0

Time of day

RPO

(Seconds)

RPO TIME (SECS)

If the additional cache was added, SRDF/A RPO would be almost 800 seconds with a single OC-3

link. Even though the SRDF/A cycle may have been set at 30 seconds, SRDF/A would not be able to

maintain it.




The Effect of Adding an Extra OC-3 Link

� Increasing the bandwidth to 31MB/s reduces the cache requirements

BANDWIDTH USAGE

0

5

10

15

20

25

30

35

23:45:00

7:45:00

8:45:00

9:45:00

10:45:00

11:45:00

12:45:00

13:45:00

14:45:00

15:45:00

16:45:00

17:44:59

18:45:00

19:45:00

20:45:00

21:45:00

22:45:00

TIME INTERVAL

Mbyte

per

Second


This example explores the option if the second OC-3 link was added to the configuration. The

bandwidth would be 31 MB/sec. The workload would not produce any peak writes that are above the

bandwidth.




Matching Cache Requirements for 2 ATM OC-3

� Cache Requirements are significant in order to cover the peak times

• Base Cache adequately covers SRDF/A during normal operations

• Peak Cache is no longer needed


0.00

0.50

1.00

1.50

2.00

2.50

TIME

7:45

:00

9:00

:00

10:1

4:59

11:3

0:00

12:4

5:00

14:0

0:00

15:1

4:59

16:3

0:00

17:4

4:59

19:0

0:00

20:1

5:00

21:3

0:00

22:4

5:00

Time

Ca

ch

e (

GB

)


This shows that the implementation does not need additional cache to accommodate peak writes. We

have eliminated the 80 GB peak cache with an extra OC-3 link.




The RPO Impact with 2 OC-3 Links

� We have a reduced requirement for cache and no requirement for peak cache

� This, as depicted below, allows us to continuously maintain a 60 second RPO


0.0010.0020.0030.0040.0050.0060.0070.00

TIME

7:45:00

9:00:00

10:14:59

11:30:00

12:45:00

14:00:00

15:14:59

16:30:00

17:44:59

19:00:00

20:15:00

21:30:00

22:45:00

Time of day

RP

O (S

econds)

RPO TIME (SECS)

Therefore, SRDF/A would be able to maintain the 30-second cycle resulting in the RPO of 60 seconds.




The Bandwidth Change

� If our sample customer is flexible on the RPO, but is only willing to provide limited bandwidth, say 5 T3 (4.5 MB/s each), then we can see the results

� We can still observe peak requirements. These directly affect the RPO and cache

BANDWIDTH USAGE

0

5

10

15

20

25

30

35

23:45:00

7:45:00

8:45:00

9:45:00

10:45:00

11:45:00

12:45:00

13:45:00

14:45:00

15:45:00

16:45:00

17:44:59

18:45:00

19:45:00

20:45:00

21:45:00

22:45:00

TIME INTERVAL

Mbyte

per S

econd


We have examined the implementation of SRDF/A with one and two OC-3 links. Now, let’s look at an

alternative in terms of link bandwidth with 5 T3 links.




Cache Requirements for Five T3 Links

� Cache Requirements are still high in order to cover the peak times

• Base Cache adequately covers SRDF/A during normal operations

• Again, Peak Cache is the major factor


0.00

10.00

20.00

30.00

40.00

TIME

7:45

:00

9:00

:00

10:1

4:59

11:3

0:00

12:4

5:00

14:0

0:00

15:1

4:59

16:3

0:00

17:4

4:59

19:0

0:00

20:1

5:00

21:3

0:00

22:4

5:00

Time

Ca

ch

e (

GB

)


With five T-3 links, SRDF/A would need 30 GB of cache to accommodate the sample workload.




RPO Impact with Five T-3 Links


0.0050.00

100.00150.00200.00250.00300.00350.00

TIM

E

7:4

5:0

0

9:0

0:0

0

10:1

4:5

9

11:3

0:0

0

12:4

5:0

0

14:0

0:0

0

15:1

4:5

9

16:3

0:0

0

17:4

4:5

9

19:0

0:0

0

20:1

5:0

0

21:3

0:0

0

22:4

5:0

0

Time of day

RP

O (

Seco

nd

s)

RPO TIME (SECS)

This implementation yields the RPO of 300 seconds. The balance of bandwidth and cache to achieve a

desirable RPO is specific to a workload pattern as shown by the three examples. A methodical and

thorough planning is crucial to a successful SRDF/A implementation.




Concurrent SRDF with SRDF/A

One source, two targets

� One target sync

• short distance, zero data lag

� One target async

• longer distance, variable data lag, no

performance impact

� SRDF: Remote Re-synch

• Continued protection upon source failure

SRDF/A

(async)

SRDF/S

(sync)

Host

In an SRDF configuration, a single source (R1) device can concurrently be remotely mirrored to two

target (R2) devices. This allows you to have two identical remote copies available at any point in time.

It is valuable for duplicate restarts or disaster recovery, or for increased flexibility in data mobility and

migrating applications. Concurrent RDF technology can use two different RA adapters (RAs, RAFs, or

RFs) in the interface link to achieve the connection between the R1 device and its two concurrent R2

mirrors. Each of the two concurrent mirrors must belong to a different RDF (RA) group

Enginuity 5671 supports Concurrent SRDF with SRDF/A. Only one of the SRDF mirrors is allowed

to be in Asynchronous mode, regardless if SRDF/A is active or not.




SRDF/A Restrictions

� Each SRDF/A group must be unidirectional

� Concurrent RDF with both RDF mirrors in SRDF/A mode is not allowed

� A personality swap with SRDF/A running is not allowed

� Cannot activate empty SRDF/A group

� SRDF/A-capable devices that are enabled for consistency group protection must be disabled before attempting to change the mode from asynchronous

� Symmetrix RDF Automated Replication (SRDF/AR) control operations are not supported for SRDF/A-capable devices running in asynchronous mode

The list is not conclusive. Please refer to EMC Solutions Enabler Symmetrix SRDF Family CLI

Version 6.0 PRODUCT GUIDE P/N 300-000-877 for more information.




Module Summary


� Technical requirements of a successful SRDF/A implementation

� Supported SRDF/A hardware platforms

� Transmit Idle and DSE (Delta Set Extension)

� Factors that affects RPO (Recovery Point Objective) in an SRDF/Aimplementation

� Cycles within SRDF/A operations

� SRDF/A Consistent Deactivation

� Multi Session Consistency

� Configuration parameters that affect SRDF/A behavior



SRDF/AR (Automated Replication) - 1

© 2007 EMC Corporation. All rights reserved. SRDF/AR (Automated Replication) - 1

SRDF/AR (Automated Replication)


� Describe the benefits of integrating EMC’s SRDF and TimeFinder applications

� List business needs and requirements using an SRDF

single or multi-Hop Symmetrix configuration

� Describe a Clustered SRDF/AR environment

� List SRDF/AR configuration requirements

� Describe the purpose of the symreplicate options file

� List the replication cycle steps for a single and multi-hop

environment





Symmetrix Remote Data Facility/Automated Replication

� Allows business restart

site to be any distance away from source

� Collaboration of SRDF and

TimeFinder commands

� Minimizes network costs R1 or

STDR2

SRDFAdaptive CopyR1/BCV

BCV

SRDF/AR allows users to automate the sequence of SRDF and TimeFinder mirror operations. The

automated sequence, cycle, is performed on a user-defined interval called cycle time.

The replication cycles automatically loop indefinitely or to the number of cycles specified by the users.

Users perform all SRDF/AR operations, setup, start, stop, restart, and query, through the

symreplicate command. Even though the SRDF link can be set to all SRDF operational mode,

except Asynchronous, it is usually set to operate in Adaptive Copy mode due to the long distance

between local and remote sites. This allows the users to save on network bandwidth thus minimizing

the network costs without compromising the integrity of the data.




SRDF/AR Requirements

� Software requirements • Solutions Enabler 4.3.1 (or later, if using PowerPath for consistent

split)

• Solutions Enabler 5.1 (or later, if using ECA for consistent split)

• Requires TimeFinder, SRDF, and SRDF/AR licenses

� TimeFinder Consistency Options• Enginuity 5x66 or later for use with PowerPath 2.1 or later

�Single server, single Symmetrix consistency

• Enginuity 5567 for use with “Enginuity Consistency Assist” (ECA) �Multiple server, single Symmetrix consistency

� Enginuity 5568, 5569

� Patch Q203 for 5x67

� Supported open systems hosts• Solaris, Windows NT/2000/2003, HP-UX, AIX

SRDF/Automated Replication (SRDF/AR) provides the ability to automate data copies across SRDF

links, providing a restartable image of the data at the remote site in the event of a disaster at the

production site. It combines both SRDF and TimeFinder to complete its operations. This slide lists the

requirements for both SRDF and TimeFinder in an SRDF/AR environment. Please refer to the EMC

Support Matrix for the latest information.




SRDF/AR Single-Hop Configuration

� Uses 2 Symmetrix arrays

� Uses STD, R1-BCV, R2 and BCV device types

� SRDF Mode of Operation• Adaptive copy

� Controlled data loss • Remote BCV can be used for disaster restart

Local Remote

Host R1 or

STD

R1BCV

BCV

R2

The copy path for a single-hop configuration is from the local R1/BCV pair (1) to the SRDF pair (2) to

the remote BCV pair (3). The remotely associated BCV holds the DBMS restartable copy. The

amount of data loss is a function of the replication cycle time (period of time between the start of one

copy cycle and the start of another copy cycle). Copy cycle time is affected by distance, bandwidth,

I/O update rate, and locality of reference for the updates. Update rate and locality of reference tend to

equate to changed tracks. The maximum data loss would be one copy cycle, thus makes the RPO ~

One Cycle Time.

Single Hop benefits include:

•The ability to perform incremental resynchronization between the intermediate SRDF target site and

the final SRDF target site, reducing required network bandwidth

•Reduction in communication link cost and improved resynchronization time for long-distance SRDF

implementations




Single Hop Set-up: Create a Device Group

1. Create a device group:

symdg create single –type

regular

2. Add standard device to device group:

symld –g single add dev 100

3. Associate R1/BCV device:

symbcv –g single associate

dev 101

4. Associate remote BCV device:

symbcv –g single associate

dev 102 –rdf –bcv

Local Remote

STD

R1BCV

BCV

R2

100

101 102

101




Local Remote

STD

R1BCV

BCV

R2

100

101 102

101

Single Hop Set-up: Initialize Mirrors

1. Establish the STD and R1/BCV (Note1):

symmir -g single est –full

This step also automatically suspends the SRDF link

2. Split the STD and R1/BCV:

symmir –g single split -consistent

3. Resume SRDF linkNote1:

symrdf –g single resume -bcv

4. Establish the R2 and remote BCV (Note1):

symmir –g single est –full –rdf –bcv

5. Split the R2 and remote BCV:

symmir –g single split –full –rdf –bcv

6. Establish the STD and R1/ BCV:

symmir -g single est

2 3 51 46

There are 6 steps to prepare the states of all mirrors involved in SRDF/AR Single Hop setup prior to

running symreplicate command. To begin a replication session in the single-hop configuration, the

mirrors must be in the following states:

− Local BCV pair must be synchronized

− SRDF pair must be suspended

− Remote BCV pair must be split

Note 1:

Users must wait and check for the full synchronization after issuing the command before proceeding to

the next step.




SRDF/AR Multi-Hop Configuration

� Uses 3 Symmetrix arrays:

• Local site, Hop1 bunker site, and Hop2 target site

• Uses R1, R2, R1-BCV, and BCV (optional) device types

• SRDF Mode of Operation

� Local to Hop 1 Bunker: Synchronous

� Hop 1 Bunker to Hop 2 Target: Adaptive Copy

• Zero data loss at Bunker site

• One cycle data loss at Target site

Host BCVR2

R1BCV

R1

R2

Local Bunker Target

The copy path for a multi-hop configuration is from the local SRDF pair (1) to the remote BCV pair

(2) to the remote SRDF pair (3) to the Target BCV (4). If your configuration does not include Target

BCVs, the path stops at (3).

Automated replication with the BCVs at Target is applicable if you want a zero data loss solution but

cannot risk the loss of both the Local site and Bunker site at the same time. With this configuration,

there are two possible disaster restart possibilities:

• If only the Local site is lost, the result is zero data loss at the Target restart site.

• If both the Local and Bunker site are lost, the result is a DBMS restartable copy at the Target restart

site with controlled data loss. The amount of data loss is a function of the replicate copy cycle time

between the Bunker site and the Target restart site.

Multi-Hop benefits include:

•The ability to perform incremental resynchronization between the intermediate SRDF target site and

the final SRDF target (Multi-Hop) site, reducing required network bandwidth

•Reduction in communication link cost and improved resynchronization time for long-distance SRDF

implementations

•The ability to use the SRDF Multi-Hop site to provide disaster recovery testing, point-in-time

backups, decision support operations, third-party software testing, and application upgrade

testing or the testing of new applications




Multi-Hop Set-up: Create a Device Group

1. Create a device group:

symdg create multi –type r1

2. Add the R1 device to the device group:

symld –g multi add dev 100

3. Associate R1/BCV device:

symbcv –g multi associate dev 101 -rdf

4. Associate remote BCV device from the Target:

symbcv –g multi associate dev 102 –rrdf

BCVR2

R1BCV

R1

R2

100 100 102

101101

Local Bunker Target




Multi-Hop Set-up: Initializing Mirrors

1. Establish the R2 and R1/BCV on Bunker (Note1 & Note 2):

symmir -g multi est -rdf

2. Split the R2 and R1/BCV and establish SRDF link between Bunker and TargetNote1:

symmir –g multi split -remote -consistent –rdf

3. Re-establish the R1/BCV and R2 on Bunker (Note 2):

symmir –g multi est –rdf

4. Establish the R2 and remote BCV on Target (Note1):

symmir –f <devfile> –sid <Target> est

or

symmir –g multi est –rrbcv (Note 3)

5. Split the R2 and remote BCV on Target:

symmir –f <defile> -sid <Target> split

or

symmir –g multi est –rrbcv (Note 3):

BCVR2

R1BCV

R1

R2

100 100 102

101101

Local Bunker Target

2 5

SRDF/S

1

2

3 4

There are 5 steps to prepare the states of all mirrors involved in SRDF/AR Multi Hop setup prior to

running symreplicate command. To begin a replicate session in the multi-hop configuration, the

mirrors must be in the following states:

− Local SRDF pair must be synchronized

− BCV pair on Bunker must be synchronized

− Remote SRDF link between Bunker and Target must be suspended

− BCV pair on Target must be split

Note 1:

Users must wait and check for the full synchronization after issuing the command before proceeding to

the next step.

Note 2:

This automatically suspends the SRDF link between the R1/BCV on Bunker and R2 on the Target

sites.

Note 3:

The BCV must associated with the device group with the –rrdf option.




SRDF/AR

� The continuous movement of dependent write consistent

data to a remote site in an asynchronous mode

• Standard to BCV/R1 => R2 to BCV Single Hop

• R1 => R2 to BCV/R1 => R2 [to BCV] Multi-Hop

� Support for Composite Groups (CG)

• CG can span multiple Symmetrix arrays

• A device group is limited to a single Symmetrix array

• Can use a CG and consistent split to create a DBMS-restartable copy of a database that spans multiple hosts and multiple Symmetrix

arrays




The symreplicate Command

� SYMCLI binary that integrates symmir and symrdf commands with the appropriate arguments and options depending on the configuration

� Used to propagate automated data copies• Incremental (changed tracks only)

• Behavior is based on pre-defined parameters (symreplicate options file)

� When used with TimeFinder consistent split, ensures a remotely-associated BCV contains a DBMS restartable copy of data• Requires PowerPath or ECA

� Used in both single-hop and multi-hop configurations

� Used to start, stop, restart, or query replication sessions

� Executed on local host

Symreplicate invokes a replicate session that generates automated, recurrent background copies of the

standard data across SRDF links and cascading BCVs. You can start, stop, and restart the replicate

session.




The symreplicate Options File

� User created, named, and configurable text file that defines andcontrols replication behavior

SYMCLI_REPLICATE_HOP_TYPE=<RepType>SYMCLI_REPLICATE_CYCLE=<CycleTime>SYMCLI_REPLICATE_CYCLE_OVERFLOW=<OvfMethod>SYMCLI_REPLICATE_CYCLE_DELAY=<Delay>SYMCLI_REPLICATE_NUM_CYCLES=<NumCycles>SYMCLI_REPLICATE_USE_FINAL_BCV=<TRUE|FALSE>SYMCLI_REPLICATE_LOG_STEP=<TRUE|FALSE>SYMCLI_REPLICATE_GEN_TIME_LIMIT=<TimeLimit>SYMCLI_REPLICATE_GEN_SLEEP_TIME=<SleepTime>SYMCLI_REPLICATE_RDF_TIME_LIMIT=<TimeLimit>SYMCLI_REPLICATE_RDF_SLEEP_TIME=<SleepTime>SYMCLI_REPLICATE_BCV_TIME_LIMIT=<TimeLimit>SYMCLI_REPLICATE_BCV_SLEEP_TIME=<SleepTime>SYMCLI_REPLICATE_MAX_BCV_SLEEP_TIME_FACTOR=<Factor>

SYMCLI_REPLICATE_MAX_RDF_SLEEP_TIME_FACTOR=<Factor>

The option file is a text file that contains parameters such as SRDF/AR hop type and cycle time. This

file is required and is used in conjunction with the symreplicate command to start a replication session.

This slide lists the variables available to be placed in the SRDF/AR options file. There are a minimum

of two options that must exist for the symreplicate to be executed. The two required options are:

SYMCLI_REPLICATE_HOP_TYPE and one of SYMCLI_REPLICATE_CYCLE or

SYMCLI_REPLICATE_CYCLE_DELAY.

As of Solutions Enable 6.2 additional SYMCLI REPLICATE variables have been added.. Ref the

“SE” product guide for a complete detailed explanation.

SYMCLI_REPLICATE_CONS_SPLIT_RETRY,

SYMCLI_REPLICATE_R1_BCV_EST_TYPE,

SYMCLI_REPLICATE_R1_BCV_DELAY,

SYMCLI_REPLICATE_FINAL_BCV_EST_TYPE,

SYMCLI_REPLICATE_FINAL_BCV_DELAY, and

SYMCLI_REPLICATE_PERSISTENT_LOCKS




The symreplicate Options File (Cont.)

SYMCLI_REPLICATE_HOP_TYPE=[SINGLE|MULTI]

** Must be specified

SYMCLI_REPLICATE_CYCLE=[minutes|hh:mm]

• Defines the period to wait between copy operations if more than 1 cycle

• Can be specified in total minutes or hours and minutes

• Defaults to 0; when one cycle ends another begins

**Either CycleTime or Delay is a required parameter

SYMCLI_REPLICATE_CYCLE_DELAY=[minutes]

• Specifies the minimum time to wait between the end of one cycle and the beginning of the next cycle

• Defaults to 0

** Either CycleTime or Delay is a required variable

The two options that must exist for the symreplicate to be executed are:

SYMCLI_REPLICATE_HOP_TYPE and either SYMCLI_REPLICATE_CYCLE or

SYMCLI_REPLICATE_CYCLE_DELAY.

SYMCLI_REPLICATE_HOP_TYPE is self explanatory and is directly dependent on the

configuration.

SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY requires a time

interval for its value.





� Time limit parameters

• Control how long symreplicate retries operations

�Applies only when an error occurs continuously

or

�When no data has flowed for duration of the timer

• Can set a time limit that is smaller than the actual time the operation

takes

• Timer will not expire as long as data flows between device

EMC recommends using default settings initially. Time limit parameter changes should only be made

based on recommendations from EMC.





� Sleep time parameters

• Specifies the minimum amount of that symreplicate sleeps:

Before checking again to see if devices have entered a specific state

or

To retry an operation when a recoverable error occurs

• symreplicate must wait for prior operations to complete before

moving to subsequent operations

• If the pair state is not reached, symreplicate sleeps for a period

of time and then checks device state again

• When checking device state, symreplicate calculates how long to

sleep based on the number of invalid tracks and the rate data ismoving

• Value must be greater than 0

EMC recommends using default settings initially. Sleet time parameter changes should only be made

based on recommendations from EMC.




Cycle, Delay and Overflow Parameters

The three parameters SYMCLI_REPLICATE_CYCLE, SYMCLI_REPLICATE_CYCLE_OVERFLOW,

and SYMCLI_REPLICATE_CYCLE_DELAY significantly contribute to how the next replication cycle

starts after the current cycle.

The next two slides show different combinations of cycle time, delay time, and overflow behavior to

achieve various results. Reference points on the cycle time lines are marked ACT (actual completion

time), NCS (next cycle start), and MDT (minimum delay time). Short cycle and delay times (in

minutes) were chosen for illustration purposes only.

Copy cycles #1 and #2 have the same cycle time (2) and no delay time. When copy cycle #1 runs

longer than two minutes, the overflow setting of “Next” results in a new copy cycle beginning at the

four-minute mark.




Cycle, Delay and Overflow Parameters (Cont.)

Copy cycles #3 and #4 have the same cycle time (2) and the same delay time (3). When copy cycle #3

runs longer than two minutes, the system waits three minutes and then resumes copying at the next

scheduled copy cycle (the six-minute mark). When copy cycle #4 runs longer than two minutes, the

system waits three minutes and then resumes copying as soon as the wait period completes.




Cycle, Delay and Overflow Parameters (Cont.)

Copy cycle #5 finishes before its scheduled cycle time of three minutes, waits two minutes, and begins

copying again at the next scheduled copy cycle (the six-minute mark). Copy cycle #6 performs

continuous copy cycles when its cycle time is set to zero. At the end of a cycle, the system waits three

minutes and then begins another copy cycle, regardless of the overflow setting.




More Options

SYMCLI_REPLICATE_USE_FINAL_BCV=[TRUE|FALSE]

Specifies whether a BCV should be used in the Target site of a multi-hop replication

SYMCLI_REPLICATE_PROTECT_BCVS=[NONE|LOCAL|REMOTE|FIRST_HOP|SECOND_HOP|BOTH]

Specifies if a protected BCV establish operation should be performed.

SYMCLI_REPLICATE_TF_CLONE_EMULATION=[TRUE|FALSE]

Specifies if TimeFinder Clone emulation mode should be used for RAID-5 BCVs.

Protected BCV establish:

NONE: normal TimeFinder operations

LOCAL: protected establish of local BCVs in a single-hop replication

REMOTE: protected establish of remote BCVs in a single-hop replication

FIRST_HOP: protected establish of Hop-1 BCV pairs in a multi-hop replication

SECOND_HOP: protected establish of Hop-2 BCV pairs in a multi-hop replication

BOTH: protected establish of both Hop-1 and Hop-2 BCV pairs in a multi-hop replication, and both

local and remote BCV pairs in single-hop replication

Clone Emulation: Beginning with Solutions Enabler 6.0, if ALL BCVs are RAID-5 BCVs,

TimeFinder/Mirror commands automatically map to corresponding TimeFinder/Clone commands.

However, if there is a mixed environment, i.e. some RAID-5 BCVs and others are RAID-

1/unprotected, then this option must be explicitly set to TRUE. Default is FALSE.




Defining and Adjusting Parameters

� EMC recommends starting with loose time constraints for

cycle time parameters

� Adjust parameters once basic information is gathered or

determined from initial cycles

• Data size

• SRDF throughput

• Operation timings

� Session progress can be monitored using the query

argument for the symreplicate command and settings

can be adjusted as required

� As of SE 6.1, the symreplicate stats command

It is recommended to be generous with time parameters for cycles, and adjust once more information is

collected for various cycles. The times configured in the options file should be configured for worst

case scenarios.

Beginning with Solutions Enabler 6.1, you can display statistical information for cycle time and

invalid tracks by using the symreplicate stats command. The command can be issued by device group

(-g) or composite group (-cg) for a specified Symmetrix (-sid) and information can optionally be

written to a specified log file (-log).

The -all option is the default and will display both the cycle time and invalid tracks statistics. The

following example will display both cycle time and invalid track for device group abcdg on Symmetrix

123, symreplicate -g abcdg -sid 23 -all stats –log abcdg.log




Overview of Implementation

� Perform initial synchronization tasks

• Devices must be in specific pair states for SRDF/AR cycle to execute

� Issue the symreplicate command to begin the

replication cycle

• Specify the options file

• Specify TimeFinder consistent split operations

• Automatic replication continues until the session completes the number of predefined cycles or until symreplicate command

issued to stop the session

• Sessions can be started, stopped, restarted, or queried

• Pre-action and post-action scripts can be applied to the data

replication process

Before using the symreplicate command, perform the initial synchronization tasks. All devices must be

in a specific paired state before executing the SRDF/AR cycle.




# symreplicate -g single setup -option /chai/test

Execute a symreplicate 'Start' operation

for device group 'single' (y/[n]) ? y

# symreplicate -g single start -option /chai/test –setup -consistent

Execute a symreplicate 'Start' operation

for device group 'single' (y/[n]) ? y

Execute one cycle to prepare the mirror states

Prepare the mirror states and start the session

Use ECA for TimeFinder

Consistent split. –ppath

and –vxfs could also be used

Automatic Initialization to Required Mirror States

Beginning in SE 5.4, SRDF/AR supports automatic setup using:

� symreplicate setup

� symreplicate start -setup

The setup command:

� Sets-up required pair states

− Options file indicates single or multi hop type setup

− Same options file used for setup and subsequent start command

� Executes one cycle, which may take some time, and then exits

− Setup command executes just one cycle, regardless of the number of cycles specified in the

options file

The start command with the -setup option:

� Sets-up the required pair states

� If successful, begins the symreplicate session

− Options file defines the hop type and the copy cycle parameters that you have chosen for the

session




Single Hop Replication Cycle

1. Split BCV pairs (wait for any ongoing establish to

complete...)

2. Establish RDF pairs

3. Suspend RDF pairs (wait for the establish operation to

complete...)

4. Establish BCV pairs

5. Establish BRBCV pairs

6. Split the BRBCV pairs (wait for the establish operation

to complete...)

The symreplicate command automatically executes the above steps in each cycle during a replication

session. These steps are similar to those used to initializing the mirror states. It is important to

understand these inner working steps because users can choose to terminate a session after the current

step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the

option file must be to TRUE for symreplicate to log the information after each step to the SYMAPI log

file. The location of the SYMAPI varies from one platform to another. The format of the file name,

symapi-YYYYMMDD.log, is identical across platforms.




Single Hop Example: Options File

DMX800ibm1 /var/symapi/config more rep_options.txt

SYMCLI_REPLICATE_HOP_TYPE=SINGLE

SYMCLI_REPLICATE_CYCLE=10

SYMCLI_REPLICATE_CYCLE_OVERFLOW=NEXT

SYMCLI_REPLICATE_NUM_CYCLES=6

A simple Options file specifying single-hop replication of 10 minute cycle times, to be repeated 6

times.




Single Hop Example: Setup

DMX800ibm1 / symreplicate -g single_hopdg setup -optimize -options

/var/symapi/config/rep_options.txt -foreground –nop

Checking for valid group configuration...

Checking for valid initial group state...

Setting up local BCV pairs...

Optimizing Local BCV pairs...

Waiting for local BCV synchronization...

Splitting local BCV pairs...

Incrementally establishing RDF pairs...

Setting up remote BCV pairs...

Optimizing remote BCV pairs...

Incrementally establishing remote BCV pairs...

Waiting for RDF synchronization...

Waiting for remote device synchronization...

Splitting remote BCV pairs...

Incrementally establishing local BCV pairs...

Setup complete; exiting symreplicate...

The setup command runs one cycle and exits. The –foreground option provides details of each of the

steps being executed.




Single Hop Example: Query and Start

DMX800ibm1 / symreplicate -g single_hopdg query

Device Group (DG) Name : single_hopdg



Replicate Cycle Current Max

Hop Type Status Step Period Cycle Cycles

--------- --------- ----------------------------- -------- -------- --------

SINGLE Completed Setting up devices 10 m 1 1

DMX800ibm1 / symreplicate -g single_hopdg start -options

/var/symapi/config/rep_options.txt -consistent -nop

Checking for valid group configuration...

Checking for valid initial group state...

symreplicate process launched.

Query shows the setup has completed. We are now ready to start the single-hop replication cycles.




Single Hop Device States: R1/BCVsDMX800ibm1 / symmir -g single_hopdg query

Device Group (DG) Name: single_hopdg

DG's Type : REGULAR


Standard Device BCV Device State

-------------------------- ------------------------------------- ------------

Inv. Inv.

Logical Sym Tracks Logical Sym Tracks STD <=> BCV

-------------------------- ------------------------------------- ------------

DEV001 001C 0 BCV001 0021 * 0 Synchronized

DEV002 001D 0 BCV002 0022 * 0 Synchronized

DEV003 001E 0 BCV003 0023 * 0 Synchronized

DEV004 001F 0 BCV004 0024 * 0 Synchronized

DEV005 0020 0 BCV005 0025 * 0 Synchronized

Total ------- -------

Track(s) 0 0

MB(s) 0.0 0.0

Legend:

(*): The paired BCV device is associated with this group.

At the start of a cycle, the R1/BCVs are established and synchronized with the Standard devices.




Single Hop Device States: Target BCVsDMX800ibm1 / symmir -g single_hopdg query -bcv -rdf

Device Group (DG) Name: single_hopdg

DG's Type : REGULAR



R E M O T E S Y M M E T R I X

Standard Device BCV Device State

-------------------------- ------------------------------------- ------------

Inv. Inv.

Logical Sym Tracks Logical Sym Tracks STD <=> BCV

-------------------------- ------------------------------------- ------------

BCV001 0021 0 BRBCV001 002B * 0 Split

BCV002 0022 0 BRBCV002 002C * 0 Split

BCV003 0023 0 BRBCV003 002D * 0 Split

BCV004 0024 0 BRBCV004 002E * 0 Split

BCV005 0025 0 BRBCV005 002F * 0 Split

Total ------- -------

Track(s) 0 0

MB(s) 0.0 0.0

Legend:

(*): The paired BCV device is associated with this group.

The BCVs on the Target site are split from their R2s.




Single Hop Device States: R1/BCV – R2 (SRDF)DMX800ibm1 / symrdf -g single_hopdg query -bcv


DG's Type : REGULAR


R E M O T E S Y M M E T R I X


-------------------------------- ------------------------ ----- ------------

ST LI ST

BCV A N A



-------------------------------- -- ------------------------ ----- ------------

BCV001 0021 NR 0 0 NR 0021 WD 0 0 S.. Suspended





Total -------- -------- -------- --------

Track(s) 0 0 0 0

MB(s) 0.0 0.0 0.0 0.0

The link between the R1/BCVs and their R2s on the Target site is Suspended.

Legend for MODES:


C = Adaptive Copy






Single Hop: Stop a SessionDMX800ibm1 / symreplicate -g single_hopdg stop

Execute a symreplicate 'Stop' operation

for device group 'single_hopdg' (y/[n]) ? y

Stop operation underway.







--------- --------- ----------------------------- -------- -------- --------

SINGLE Stopped Waiting for next cycle 10 m 3 6




Single Hop: Restart a Stopped SessionDMX800ibm1 / symreplicate -g single_hopdg restart -recover

Execute a symreplicate 'Restart' operation

for device group 'single_hopdg' (y/[n]) ? y

symreplicate process launched.







--------- --------- ----------------------------- -------- -------- --------

SINGLE Active Establishing RDF pairs 10 m 3 6




Failure/Recovery: Single Hop

� When a failure occurs in the primary site, we can have

one of the three situations on the target site

• All BCVs on the target are split from their corresponding R2 devices

• All BCVs on the target are established with their corresponding R2

devices

• Some of the BCVs on the target are split from their corresponding

R2 devices, while others are still established with their corresponding

R2 devices

� Recovery/Restart involves identifying devices with the most current and consistent copy of data

Determination of the states of the devices and deducing the cycle step using the states can be

performed from a host on the target side, using appropriate device files. These operations can be

performed from a different host on the source side, if the failure only affected the primary host, and the

array and site are still available and accessible.




Failure/Recovery: Single Hop (Cont.)

� All BCVs at target site are split:

• Are R1/BCVs established with the Standard devices?

• Are R1/BCVs transmitting owed data to their R2 devices?

• Are R1/BCVs – R2 pairs in a partitioned state?

� All BCVs at target site are established:

• R1/BCVs have completed transmission of data. The BCVs are in the

process of being synchronized with their R2s

� At the target site, some BCVs are split while others are still established:

• R1/BCVs have completed transmission of data. The BCVs are in theprocess of being split from their R2s

All BCVs are split: If the host on the primary site has failed, but the array and links are available, the

R1/BCVs can either be established with the Standard devices or can be split from the Standard devices

and transmitting data to the R2. In the former case, both the R2 and BCV contain identical and

consistent data, and restart can be done from either set of devices. In the latter case, the most recent

data received by the R2 and BCV has consistent data up to the last cycle. In this case, one can wait for

transmission to complete and then restart from the R2. Prior to restart it would be advisable to re-

establish the BCVs, synchronize, and split to maintain a gold copy. In the third case, R1/BCVs still

have tracks owed to the R2. So the consistent data resides on the BCV. One can restart using these or

preferably restore the BCV data to the R2s and restart from the R2.

All BCVs are established: It is clear that both R2 and BCVs have the most recent and consistent data

at the end of the BCV synchronization. One can then split the BCVs (to maintain a gold copy) and

restart from the R2s.

Some BCVs are split while others are still established: The R2s have the most recent and consistent

data. Re-establish all the BCVs with the R2s, synchronize, and split (to maintain a gold copy), prior to

restarting from the R2s.




Multi-Hop Replication Cycle

1. Split the first hop BCV pairs (wait for the ongoing

establish to complete…)

2. Establish remote RDF pairs

3. Suspend remote RDF pairs

4. Establish the first hop BCV pairs (wait for the RDF establish to complete...)

5. Establish the second hop BCV pairs

6. Split the second hop BCV pairs (wait for the establish

operation to complete...)

The symreplicate command automatically executes the above steps in each cycle during a replication

session. These steps are similar to those used to initialize the mirror states. It is important to understand

these inner working steps because users can choose to terminate a session after the current step instead

of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must

be set to TRUE for symreplicate to log the information after each step to the SYMAPI log file. The

location of the SYMAPI varies from one platform to another. The format of the file name, symapi-

YYYYMMDD.log, is identical across platforms.




Clustered SRDF/AR

Host A

BCVR2

R1BCV

R1

R2

Local Bunker Target

Host B SFS

Since Enginuity 5669, Symmetrix arrays support clustered SRDF/AR environments for multiple node (host) capability. Clustered SRDF/AR provides the capability to start, stop, and restart replication sessions from any host connected to any local Symmetrix array participating in the replication session.

The clustered SRDF/AR environment allows the replication log file to be written directly to the Symmetrix File System (SFS) instead of the local host directory of the node that began the session. If the primary node should fail, then any locally attached host to the Symmetrix array containing the log file would then be able to restart the SRDF/AR session from where it left off. If you begin a session and specify a user log file name (-log), you must specify the -log option for all other commands in the session sequence.

To write the log file to the SFS, you must specify the ID of the Symmetrix array (-sid) where the log file is to be stored at the start of the replication session, along with a group name (-g, -cg) and an optional user log filename (-log).

For example:

symreplicate start -g session1 -log srdfar1.log -sid 201

Note:

Not specifying the Symmetrix ID (-sid) at the start of the session, causes the log file to be written to local disk using the default SYMAPI log directory, which is not restartable from another node.




Failure/Recovery: Multi-hop

� In the event of a primary site failure, R2s on the bunker

site always contain the most recent and consistent data (SRDF/S between R1 – R2). The state of the R1/BCVs in

the bunker site dictates the restart/recovery procedures at

the target site:

• Are R1/BCVs established with the R2 devices?

• Are R1/BCVs transmitting owed data to their R2 devices at the target

site – i.e. All R1/BCVs are split from their R2s?

• Are some R1/BCVs split, while others are established?

� As in the case of single-hop failure, recovery/restart

involves identifying devices with the most current and

consistent copy of the data, but at the bunker site

Note that BCVs at the target site are optional in a multi-hop configuration

(SYMCLI_REPLICATE_USE_FINAL_BCV=<TRUE|FALSE>). In case BCVs are used at the

target site, we must consider their pair states if we wish to preserve copy of data from previous cycle.




Failure/Recovery: Multi-hop (Cont.)

� R1/BCVs are established with their R2s

• Bunker R2s are synchronizing the most recent data with the

R1/BCVs. Target R2 and optional BCV contain consistent data up to

the last cycle

� R1/BCVs are split from their R2s

• R1/BCVs are transmitting data and optional BCVs are split from the

target R2s

• R1/BCVs have completed data transmission, and optional BCVs are either split from or established with the target R2s

� Some R1/BCVs are split while others are established with

their respective R2s

• Optional target BCVs are necessarily in a split state

R1/BCVs are established with their R2s: In this case we can wait for synchronization between

R1/BCVs and the bunker R2s. Then split the R1/BCVs and resume the link between R1/BCVs and

target R2s. The optional BCVs at the target site can be left in a split state to preserve consistent data

from the previous cycle.

R1/BCVs are split from their R2s: If the target BCVs are split, then one can re-establish the

R1/BCVs with the bunker R2s, wait for synchronization, split, and resume the link between R1/BCVs

and the target R2s. The BCVs at the target will again have consistent data up to the last cycle. If BCVs

at the target site are in an established state, one can wait for or verify synchronization, split the BCVs

(to preserve data from the previous cycle), then perform the re-establish, synchronization, split, and

resume of the R1/BCVs.

Some R1/BCVs are split while others are established with their respective R2s: In this instance,

one can re-establish ALL the R1/BCVs with the bunker R2s, synchronize, split, and resume links to get

the most recent and consistent data over to the target R2s.




General Recovery Procedures

� Once the R2 devices on the target site are populated with the most recent and consistent data, they must be RW enabled for the host

� Application restart times must be factored in to the overall Recovery Time Objective

� Availability of necessary device file definitions at the target site can help in reducing the RTO. Device files must be updated on any configuration changes – adding more Standard devices or R1s

� It is preferable to restart/recover from the R2 devices• Facilitates easier return home

• BCVs can be used to preserve previous cycle’s data

RW enabling of R2s can be performed via the symrdf failover command with the appropriate device

file definitions.




Module Summary


� Benefits of integrating EMC’s SRDF and TimeFinder applications

� Business needs and requirements using an SRDF single

or multi-Hop Symmetrix configuration

� Clustered SRDF/AR environmens

� SRDF/AR configuration requirements

� The symreplicate options file

� Replication cycle steps for a single and multi-hop

environment

These are the key points covered in this module. Please take a moment to review them.




Course Summary

Key points covered in this course:

� Relevancy of SRDF solutions with different (RPO) Recovery Point Objective needs

� SRDF concepts, terminology and functionality

� Use of the SYMCLI command set to perform SRDF operations

� SRDF host considerations and configurations within Sun Solaris, HP-UX, IBM AIX, and Windows – LVM environments

� SRDF/A operations

� SRDF/A theory of operations and application

� Architectural components of SRDF/A

� SRDF/AR operations

� Architectural components of SRDF/AR

� SRDF/AR theory of operations

� SRDF/Star environment

� EMC Consistency Technology

These are the key points covered in this training. Please take a moment to review them.

This concludes the training. In order to receive credit for this course, please proceed to the Course

Completion slide to update your transcript and access the Assessment.




Closing Slide

This concludes, Module VI “SRDF Consistency Technology” for Symmetrix Business Continuity –

SRDF Solutions.

Date post:	20-Feb-2015
Category:	Documents
Upload:	amaleshanagani
View:	3,847 times
Download:	8 times

SRDF Solutions

Documents