RecoverPoint Deployment with KDriver on Sun Solaris...The driver itself is directly inserted into...

EMC Proven Professional Knowledge Sharing 2010

RecoverPoint Deployment with KDriver on Sun Solaris

Arthur Johnson

Arthur JohnsonAdvisory Technical Education Consultant EMC [email protected]

2010 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Overview ........................................................................................................................... 3 KDriver software components ........................................................................................... 3

Software I/O stack with Solaris KDriver ........................................................................ 4 KDriver SAN Discovery ................................................................................................. 5 KDriver Host Control ..................................................................................................... 5 KDriver Host Replication ............................................................................................... 5 KDriver Splitter Modes .................................................................................................. 6 KDriver Disaster States ................................................................................................. 6

Installing RecoverPoint KDriver Software on Sun Solaris ................................................. 9 Requirements for Installation ........................................................................................ 9 Performing the Installation .......................................................................................... 10 Checking the Installation ............................................................................................. 12

Installing KDriver in a Host-Cluster Environment ............................................................ 12 Solaris Containers and KDriver ................................................................................... 12 Files in a KDriver Implementation ............................................................................... 13

Managing the Solaris Environment with RecoverPoint KDriver ...................................... 13 Kdrv Operations .......................................................................................................... 13 Logging: Details, Gathering logs, Checking for Connectivity ...................................... 16 Using Kutils to Manage Volumes/LUNs in the Solaris Environment ........................... 18

Un-Installing and Upgrading KDriver on Solaris .............................................................. 19

Disclaimer: The views, processes, or methodologies published in this article are those of the author. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.


Overview RecoverPoint KDriver is a host-based splitter with support for the following operating environments: Sun Solaris; IBM AIX; and Microsoft Windows Server. The KDriver software has several distinct components which will vary depending upon the operating environment. This paper discusses Sun Solaris KDriver theory, installation, management, and basic troubleshooting in a Sun Solaris 10/RecoverPoint 3.2 environment.

In a standard RecoverPoint environment, I/O writes are split and sent to the appliances by a splitter. A splitter is either a host-based KDriver, CX3/CX4 splitter, or intelligent fabric splitter deployed on either a Brocade SAS or Cisco SANTap platform. This paper deals with KDriver and the theory of operations associated with this type of splitter.

The following diagram illustrates a basic KDriver implementation with RecoverPoint and the associated theoretical steps as writes are split, sent to native FC storage and the appliance with their associated acknowledgements.

RecoverPoint – KDriver Site Architecture

KDriver software components RecoverPoint Solaris KDriver has three main components:


• Watchdog/Service o Own separate process

o Daemon which spawns other processes – e.g., KDriver, HLR (host log retriever)

o Monitors KDriver processes and can restart as needed

• KDriver, in User Space o Contains replication logic, builds the view of the SAN from the host

perspective and communicates that with the RPAs

o Performs SAN discovery

o Communicates and updates with the RPAs

• Splitter, in Kernel Space o Actual splitter driver

o Manages the splitting logic – not the writer, but manages the splitting

Other components within the KDriver

Process Monitor; checks high and low watermark for memory usage

Kernel Log Reader; reads from kernel memory and writes to the host log events which have occurred within the KDriver/Host system

Software I/O stack with Solaris KDriver The splitter driver runs within the host operating environment within the Kernel space. The driver itself is directly inserted into the I/O stack before multipath management software such as PowerPath® or MPxIO. The driver will split writes for any FC LUN which the hosts sees and has had the splitter attached to that LUN/Volume from the perspective of the RPA. Attaching a volume to a splitter is done on the RecoverPoint management console and would essentially begin the operation of protecting that LUN with RecoverPoint replication.

There are cases when the LUN is attached to a splitter but writes are not split and sent to the RPA. This may be the case if the module is not running in the Kernel or if there were specific operational conditions which prevented the KDriver from splitting writes (e.g., volume in pass-through mode).

There are six operational stages with the KDriver in a write splitting scenario:

1) Write from host production application, split the write

2) Send the write to the assigned RPA

3) Receive acknowledgement from the RPA

4) Write to FC storage

5) Receive acknowledgement from FC storage


6) Acknowledge to host that the transaction is complete

KDriver SAN Discovery As previously mentioned, the KDriver in user space is responsible for SAN discovery. The KDriver will discover all entities in the SAN that it needs to manage and sends this host-based perspective to the RPAs. This view, derived from SAN discovery, is stored in the kernel splitter and in a file called “Volumes.db”. This file exists in order to persist host reboots. However, this discovery view is built after both installation and system reboots.

The splitter is seen as an initiator within the environment. The discovery of SAN components includes:

Mapping: Physical entities → Code objects (GUID)

SAN View: Storage, RPAs

– Sends SAN View to any RPA cluster

– Sends SAN View to Kernel Splitter

KDriver Host Control There are three states of KDriver host control: Send, Get, and FC LUN connectivity monitoring.

The Send State reads the state from the host Kernel and notifies the control process. In this state, the KDriver will communicate to the RPA regarding Host-to-RPA connectivity and different disaster states including ICW (incomplete writes), OCW (over complete writes), disconnects, etc.

The Get State communicates with the control process within a RPA and gets states on volumes and splitter states per volume from the RPA. Additionally, the Get state will report on any flags for volumes (e.g., ICW, OCW, disconnect) and if state has changed, it will update and configure the Kernel splitter to respond to the state change. The Get State is performed via SAN communication/discovery and is an ongoing process. The Get State also will perform base connectivity monitoring for Host-to-RPA and Host-to-storage connectivity.

KDriver Host Replication The KDriver Host Replication logic contains information and configuration details on the host-to-RPA data path, backlog, pipe between delta marker and backlog, and the ability to update backlog from Kernel to user space accordingly. The data path is the communication space between the host KDriver and RPA replicating the hosts’ LUNs. In normal operations, the data path is open and the KDriver is splitting writes and sending them to the RPA.


When communication is interrupted between the host and RPA, the KDriver will begin working off the backlog. The backlog maintains write metadata, offsets, and lengths of writes which have occurred while the data path has been interrupted. The delta marker is part of the replication process and works with the backlog to update the RPA/KDriver when the data path returns to a normal operational state.

KDriver Splitter Modes There are currently six different KDriver splitter modes. The modes suggest different operational states that might be effecting the entire replication system. The modes essentially describe the state of the splitter as it changes depending on the conditions within the environment. The states the splitter may be in are crucial in troubleshooting the environment. The six splitter states are:

N/A (not attached) - In this state, the splitter will not be splitting and sending writes to the RPA.

Split – In this mode, LUNs have been attached to splitters and splitters are sending copies of writes to the RPAs so that writes can be replicated.

Pass-through – While in pass-through, I/O still goes through the splitter; however, the writes go directly to FC storage, no I/O is sent to the RPA.

Delay – This mode starts due to an I/O error. As the name suggests, it delays the I/O. Depending on the application, there will be differences in how the application behaves during the delay. Delays expire after 60 seconds; however, if there is still a delay after 60 seconds (I/O error), the KDriver moves into pass-through mode and lets the host continue writing to its FC LUNs.

Fail-all – A configured state for a consistency group in which the splitter is allowed to interrupt access to the host volumes. If the operator has configured the RPA to regulate the application, writes will fail and the application will be unable to continue.

Redirect – This mode describes the environment when a consistency group is in virtual access mode. In this mode, the RPAs present a virtual image to the host which believes it is writing to actual storage. In this state, both writes and reads will be managed by the RPA managing the consistency group.

KDriver Disaster States RecoverPoint replication environments are complex integrations of many different components. All components within the system need to be able to account for differences in the environment and change behavior to best meet the defined working state for replication.


With regard to KDriver splitters, there are the following defined disaster states:

Host Reboot: – Marking is done on the host. This is done to make sure no I/O is

missed while the host may come up, application may start and the KDriver has not yet started. In this event, MOH (Marking on Host) is performed and when the data path to the RPA becomes available to the KDriver, it will then sync the backlog with the RPA.

Incomplete Write (ICW): – Incomplete Write

An incomplete write is when I/O reached the storage but didn't reach the RPA. ICW and OCW flags in the volumes database – these entries mark the incomplete or over-complete writes. When the problem is resolved, it will have to do an undo since the write went to storage without acknowledgement of the RPA. It will verify this via the delta marker.

Over-complete write (OCW): – Over-complete Write


– – With an OCW, the RPA gets the write, but the volume is in pass-

through mode and writes are made to FC LUNs. With MOH, the RPA will be able to see the writes made, verify with the delta marker, then replicate those missed writes over the WAN.

Double (multiple) disasters: – When there is more than a single issue within the replication

environment.


Installing RecoverPoint KDriver Software on Sun Solaris

Requirements for Installation Installing RecoverPoint KDriver software on Solaris requires a supported SAN-attached patched version of Sun Solaris within the customer environment. For supported platforms and operating environment versions, please refer to the EMC Support Matrix and reference the Host OS version, patch update, and compatible RecoverPoint versions.

Solaris KDriver software is posted and available on EMC Powerlink. Prior to installation, the RecoverPoint operating environment version should be known so that a compatible version of KDriver is downloaded and subsequently installed on the host. This software can be installed from local media such as CD/DVD or copied over the network (FTP/SCP) and then installed from a local disk.

The following activities are required for performing a successful standard installation of RecoverPoint KDriver on a Sun Solaris host. Although these steps may be negated in the event of pre-installation of SAN or RPAs, these steps describe ideal and most supported conditions.

Step One: Checking for Storage Requirements

The host should have assigned and accessible SAN storage. The KDriver will split writes which are going to FC SAN-attached LUNs. It is worthwhile to determine which LUNs the hosts access, from which arrays, and which will be protected with RecoverPoint. This is easily ascertained from the output of inquiry (inq). These LUNs will be attached to the KDriver splitter upon completion of the installation.

Inq


Additionally, Solaris KDriver software will require approximately ~500Kb. However, log sizes and retention duration for Solaris KDriver are configurable upon installation. The default size for logging is 500MB.

Step Two: Host-to-RPA Zoning

The servers will need to be zoned to RPAs in order for the splitters to send writes to the RPAs.

On current RPAs, all ports can be used for both initiator and target functionality. At a minimum, zone each host HBA port with an RPA port for each fabric. EMC best practices dictate that each zone will contain two members: an initiator port (Host HBA) and a target port (RPA port per fabric). Ideal host-to-RPA zoning would be:

• Fabric A: Host_to_RPA_zone: HBA port 0 + RPA port 0 (first port, first HBA)

• Fabric B: Host_to_RPA_zone: HBA port 1 + RPA port 2 (third port, second HBA)

Once the zoning is complete and the fabric configuration refreshed, the host may be able to discover the new RPAs as targets. To refresh the hosts’ FC device list, run the command: devfsadm

Inq Showing RPAs asTtargets

Step Three: Permissions, Log Directories, Base Space Requirements

Use the command: pkgadd to install RecoverPoint KDriver on Sun Solaris. By default, the software base directory is / and will require approximately ~500 on a local disk. If it is desired to install the software in a different base directory location, that needs to be defined at the time of installation.

The software must be installed as root and will be done interactively with pkgadd

Performing the Installation Log in to the host as root and prepare to install the required KDriver software. The software can be made either available on a local disk, network share, or copied from CD/DVD.

1. Copy the software to a temporary location on the host (e.g., /tmp)

2. Unzip the KDriver software


gunzip EMC_RecoverPoint_driver_sol5x_rel3.2_e.131_md5_42dc0b0cf7026bfa5f59d5c752ec3ebe.pkg.gz 3. Begin installing the package, using pkgadd

pkgadd -d EMC_RecoverPoint_driver_sol5x_rel3.2_e.131_md5_42dc0b0cf7026bfa5f59d5c752ec3ebe.pkg 4. Follow the prompts and install the KDriver software

5. SAN Diagnostics runs, as part of the installation. This is a SAN discovery

process for the KDriver with unique requirements which must be met for the environment. If there are issues, the installation will not complete (e.g., no connectivity to RPAs). There are false issues too, e.g., finding non-SAN attach disks

6. Enter a directory for the log files. The default location is the base

directory/log. e.g., /kdriver/log

7. Accept default size (500MB) or change size

8. This concludes installation of RecoverPoint KDriver on Solaris


Checking the Installation As part of the conclusion of the installation, the KDriver will be started and run on the host. However, there may be a need to verify the installation and check that the daemon is running and that the splitter can be discovered from the RecoverPoint management console.

1. To verify that the software is running and properly installed:

ps –ef |grep –i kdriver

2. check the status of the running kdriver

/etc/init.d/kdrv status

3. After the KDriver is installed and running, volumes can be attached to this

splitter and a consistency group with defined replication parameters can be created.

Installing KDriver in a Host-Cluster Environment The installation process is the same for installing the KDriver package in a host-based cluster. In an environment where the cluster consists of an active and passive node, the KDriver can be installed but not running and not splitting writes. If the active node fails, the passive node will have to have a dependency created to start the KDriver and begin write splitting in the event production moves to the passive node.

Solaris Containers and KDriver Containers (zones) within Solaris provide a mechanism to run isolated environments for applications. Processes from individual applications running in a zone will be prevented from interfering or interrupting other activities within the system. Access to system-wide processes, network interfaces, file systems, and


devices are restricted to prevent interference from different processes within the system and other zones.

A KDriver installed in a single zone in a Solaris RecoverPoint KDriver environment can only be attached to devices available to that zone and those devices can only be attached to the splitter within that zone.

Files in a KDriver Implementation A majority of the files in a KDriver installation are placed in the base directory, /kdriver.

From there, the major areas of interest include the following subdirectories:

/kdriver/bin -> location of the kdriver software, scripts, utilities

/kdriver/etc -> start and stop init files, start and stop scripts

/kdriver/info_collector -> host info collector is the logs collector for a KDriver

/kdriver/log -> installation and configuration logs

/kdriver/tweak -> collection of parameters passed to the KDriver upon configuration/startup. These files will dictate any unique parameters which the driver may need/require

Managing the Solaris Environment with RecoverPoint KDriver

Kdrv Operations It may be required to occasionally stop, manually start/restart, or check the status of KDriver on a Solaris host running the software based splitter. There may be different reasons for stopping, starting, or restarting the KDriver but without it running, writes will not be split and sent to the RPAs to be replicated. To check the status of a currently running KDriver, you can simply run the start up script, kdrv located in /etc/init.d. Running this command will allow the operator to check to see if the KDriver, the watchdog service, and host log retriever are up and running: /etc/init.d/kdrv status

For various reasons it may be required to stop the KDriver splitter entirely. In some cases it may be acceptable to stop the KDriver but still allow the host access to volumes (option 2). In other instances, it may be required to regulate I/O (slow access, option 1) or prevent access to the volumes entirely (option 3). Halting all access to the volumes


will preserve the replication environment and prevent a full sweep from occurring for either the volumes or consistency group(s) involved. Stopping the KDriver: /etc/init.d/kdrv stop

Starting the KDriver may be required when either the service failed to start upon boot or has been stopped. Starting the KDriver: /etc/init.d/kdrv start

The kdrv script also allows an operator to diagnostically check the volumes which have been attached to splitters; this command option is “check_devs”. Checking devices will test all path connectivity to RPAs (referred to as kbox from this utility). The check_devs option will also provide details for volumes attached to the splitters including GUID identification, storage paths including major and minor numbers, volume size, host cluster state, SCSI reservations, and number of reads and writes performed to the volume. Check Devs: /etc/init.d/kdrv check_devs


Details on volume 0 (first volume attached to the splitter) shown from check_devs: volumes : *********** splitter vol 0: kbox lun = 1 name = device name: major=0 guid=0x383a571bbeb4ebfb storage path 0 : path = major:118 minor:154 storage path 1 : path = major:118 minor:178 storage path 2 : path = major:280 minor:114 kbox-1 = major:118 minor:18 mode = split NonDomino mode = ON Boot Device = NO Dirty flag = clear Incomplete write = clear Meta Data Available = clear Splitter Initiated Unregulate = clear reservation state = reserved cluster type = none active = 0 size = 8256000 blocks delayExpiryTimeout = 60 fo_time = 0 basket id = 1 b0=0 b1=0 open=1 bytes_per_sector=512 number_of_heads=15 sectors_per_track=128 number_of_cylinders=4300 num_writes=0 num_end_writes=0 num_reads=1 num_end_reads=1 num_split=0 num_kbox_io_done=0 num_redirect_reads=0 num_redirect_reads_done=0 num_redirect_writes=0 num_redirect_writes_done=0 num_send_to_storage=0 num_storage_io_done=0

A final operation from kdrv is to remove it from the start up process and to stop the KDriver and watchdog processes. A reason for running this option of the command is in cases when the host may need to be rebooted but not restart the splitter upon boot (maintenance reasons).


Removing the start option and stopping the processes:

/etc/init.d/kdrv remove_all

Logging: Details, Gathering logs, Checking for Connectivity In the event that logs are required to check the environment, logs should be collected via the RPAs in the environment. An option which may assist in expediting log collection is to set the splitter to be enabled for “Automatic host info collection”. This setting is for the KDriver on a specific host within the environment. To set the host for automatic info collection by the RPAs:

/kdriver/bin/kutils/manage_auto_host_info_collection enable Alternatively, if host info collection is not required to be automatic, it can be disabled:

/kdriver/bin/kutils/manage_auto_host_info_collection disable To collect the logs for KDriver/Splitter:

1. Login to an RPA within the environment as user boxmgmt

ssh [email protected]

2. Select [3] Diagnostics, [4] Collect system information

3. Follow prompts to select the following options:

a. Date and time for collection start (beginning of the range)

b. Time and date for collection stop (end of range)

c. Chose to copy the collected logs to an FTP server or output in default location (on the RPA available via ftp via user: webdownload)

d. Chose number of sites, only one or both – you must choose the site for which the splitter is installed and requiring collection

e. Select to get data for splitters only – use the default collection file name

Completed splitter collection:


After the collection has completed, the log file will be available either on the specified FTP server or via webdownload (as shown above). The log file will be a tar with the gathered details as the name of the file:

sysInfo-no-rpas-splitters-from-l1-l2-2009.09.15.15.43.05.tar The format of the file name is: systemInformation-no RPAs – only splitters – gathered details from site1, box1 and site1, box2- date and time stamp of the log collection.

After untarring the log file, the following similar files will be extracted from the tar file:

HLR-l1-2009.09.15.15.43.10 -> directory containing the collected details

HLR-l1_success -> details on the collection (success or fail)

HLR_hosts -> Host Log Retriever (HLR) hosts with splitters installed which were collected from

HostsCollectionRemote.log -> log of collection activity

The directory containing the actual logs will have the date and time stamp as part of the directory name: HLR-l1-2009.09.15.15.43.10. This directory will contain a zipped/tarred file for each splitter which was collected from.

These files should be sent to support for analysis.


Using Kutils to Manage Volumes/LUNs in the Solaris Environment It is recommended to source the /kdriver/bin/kenv.sh file before running kutils. Kutils on a Sun Solaris platform provides the ability to an operator to perform the following actions:

1. Enable or disable automatic host info collection

2. Show volume details by volume name

3. Show all volumes

4. Start/Stop the KDriver (similar to directly running /etc/init.d/kdrv)

Kutils in a Sun Solaris environment does not need to support functionality such as managing disk cache, buffers, or mounting and umounting of devices. To manage devices in a Solaris/RecoverPoint environment, follow the system administration practices normally deployed in the environment. Practices such as using volume managers, mounting, umounting, and syncing disks all work outside of KDriver/Kutils and should be conducted in the manner which meets the needs of the host operating environment.

Managing SCSI reservations within a Sun Solaris/RecoverPoint environment is primarily managed by RecoverPoint. If the consistency group for a given host has been defined to support SCSI reservations, the RPAs will know how to handle and


manage reservations based on the desired, defined behaviors for the consistency group.

Un-Installing and Upgrading KDriver on Solaris Beginning with RecoverPoint 3.1, splitters and the RecoverPoint operating environment can be at different versions. With 3.1, the RecoverPoint environment (RPAs) can communicate with older versions of the splitters and their protocols (up to two versions different). However, there will come a time in which the splitters need to be upgraded.

Within the Solaris operating environment, only one unique instance of a package can exist at any given time. Given this requirement, it is necessary to first uninstall the currently installed KDriver.

To uninstall (for either upgrade or removal purposes):

1. Stop the currently running KDriver on the Solaris host

/etc/init.d/kdrv stop


2. If the host is going to maintain access to the volumes, choose the appropriate shut down method. Either allow device access or restrict access to the production volumes. If the host is allowed to continue accessing the volumes, a full sweep will have to be performed. If re-sync time is to be minimized, the stopping KDriver should halt all access to the host production volumes and the host should not access the volumes to avoid the full sweep

3. From the RecoverPoint console,

a. Detach volumes from the splitter

b. Delete the splitter from RecoverPoint

4. Remove the KDriver

pkgrm kdriver 5. Select to either preserve or discard logs

6. The upgrade process will be the same as the installation procedure previously written in this document. Install the new version of the KDriver package.

7. Check the status of the KDriver after re-installing (reboot usually not necessary). The driver should be up and installed.

/etc/init.d/kdrv status


8. On the RecoverPoint console, discover the newly installed splitter.

9. Re-attach volumes to the splitter – if the host was blocked from accessing

the volumes during this activity, the volumes can be attached “as clean” and a full sweep will be avoided.


10. Check replication consistency groups to make sure that replication can now

continue normally.

Date post:	20-Apr-2020
Category:	Documents
Upload:	others
View:	6 times
Download:	0 times

RecoverPoint Deployment with KDriver on Sun Solaris...The driver itself is directly inserted into...

Documents