IM B09 Best Practices for Backup and Recovery of VMware...

IM B09 Best Practices for Backup and Recovery of VMware - DRAFT v1

George Winter, Symantec Corporation Technical Product Manager

1

David Hendrix, FedEx Corporation

Technical Principal

Product Marketing Engineer

Abdul Rasheed, Symantec Corporation

IM B09 - Best Practices for VMware Backup

SYMANTEC VISION 2012

Please Note…

“This information is about pre-release software. Any unreleased update to the product or other planned modification is subject to ongoing evaluation by Symantec and therefore subject to change. This information is provided without warranty of any kind, express or implied. Customers who purchase Symantec products should make their purchase decision based upon features that are currently available.”

IM B09 - Best Practices for VMware Backup 2


Agenda

• What’s a “VADP”?

• VADP – Backup Process

• Performance Baseline Testing

• Backup Application & VADP Perf Considerations

• Virtual Machine Restore Considerations

• FedEx VADP Implementation

• NetBackup Deduplication Appliance

• Q & A

3 IM B09 - Best Practices for VMware Backup


What’s a “VADP”?



vStorage API for Data Protection (VADP)

• A family of storage related API’s created by VMware

– vStorage API for Array Integration (VAAI)

– vStorage API for Data Protection (VADP)

– vStorage API for Site Recovery Manager (VASRM)

– vStorage API for Multi-pathing (VAMP)

• Not a backup application – a true API

– Backup vendors use this API to access advanced VMware backup capabilities

• VADP replacement (not enhancement) of VCB

– VCB still supported with vSphere 4

– VCB no longer supported with vSphere 5



VADP - Provides Powerful Backup Features

• Changed Block Tracking (CBT)

– True incremental backups

– I consider this most important enhancement

– Image (vmdk) level incremental backup & restore

– Some vendors support single file restores from CBT incremental

• Incremental (CBT) backups are key component of efficient backups

– Less back end storage required

– Quicker backups

– Less snapshot impact (more on this in a minute)



VADP – Backup Process



vStorage API Backup Process

8

Backup Target

VMware Backup Host

1. Snapshot is created

ESX/ESXi

VMDK VMDK VMDK VMDK VMDK VMDK

VM1 VM2 VM3 ESX

Datastore

ESX/ESXi


VM4 VM5 VM6

1

1



NetBackup 7 for VMware Backup Process

9

Tape

VMware Backup Host


2. VM data copied directly to tape (VTL, etc.)

ESX/ESXi


VM1 VM2 VM3 ESX

Datastore

ESX/ESXi


VM4 VM5 VM6

2

2



NetBackup 7 for VMware Backup Process

10

Tape

VMware Backup Host


2. VM data copied directly to tape (VTL, etc.)

3. VM Snapshot released

ESX/ESXi


VM1 VM2 VM3 ESX

Datastore

ESX/ESXi


VM4 VM5 VM6

3

3



NetBackup 7 for VMware Backup Data Path

• Network Backups (NBD)

– NFS or DAS fully supported

– No loss in backup or restore functionality

– Direct communication with ESX server required (DNS, etc.)

– ESX server directly impacted

11

= Backup Impact / Path

ESX Datastore

LAN



NetBackup 7 for VMware Backup Data Path






• Shared storage configuration

– Fibre or iSCSI

– Near zero impact on ESX


– No communication with ESX required

• Hotadd

12


LAN

ESX Datastore

LAN

ESX Datastore



ESX Server Load – Client Backups vs. VADP

13

Std

Clie

nt

Bac

kup

s St

d C

lien

t B

acku

ps

NBU 7 for VMware

SAN Backups

NBU 7 for VMware

SAN Backups



Performance Baseline Testing



Know Your Hardware!

• Determine maximum performance of backup environment

– Tests are designed to simulate VADP backup paths (SAN, Network)

– Verify that hardware configured and working correctly

– Never assume hardware or software working optimally until performance verified

• Cisco UCS, VMware, NetBackup Benchmark:

http://www.symantec.com/business/products/whitepapers.jsp?pcid=pcat_business_cont&pvid=2_1



Backup Application & VADP Performance Settings & Considerations



What Happens During a Backup?

• During VM backups, VMware creates temp snapshot of VM. Process is as follows:

– VSS provider flushes OS buffers in VM

– Snapshot of VM is taken (vmdk(s) are frozen) (SCSI reservation of LUN)

– Redo log created – all writes redirected to redo log

– VM is backed up

– Redo log data applied to original vmdk(s)

– Snapshot released - backup completed

• Why does this matter?

– Every one of these steps involves I/O

– Reducing random, simultaneous backups limits I/O impact

– Improves backup performance & reliability

A Detailed Look at the VADP Snapshot Process

ESX/ESXi


VM1 VM2 VM3



Reducing Backup I/O Impact – Improve B/U Reliability

• Shorten time backup snapshot is open

– Backup during periods of low VM activity

– Use incremental backups (CBT) liberally

– Limit simultaneous backups per ESX / Datastore

• Configure backup application for optimal performance

– Design backup policies to evenly balance load across ESX / Datastores

– Faster backups = shorter open snapshots

– Tune backup application buffers for optimal performance

• Result:

– Snapshots more reliable

– Overall backup processing faster

– Higher level of backup success



• Single VADP backup stream typically won’t saturate backup path

• Max backup performance achieved by creating simultaneous backup streams

• Design backups so that data is streamed across multiple VMware components (ESX/DS)

• Improve backup speeds, shorten backup window

Ensure Optimal Backup Performance

Number of simultaneous backups?

19

0

20

40

60

80

100

120

1 2 4 8 12 15

Bac

kup

Th

rou

ghp

ut

in M

B/s

Total # of concurrent VMguests

Aggregated Backup Speed (MB/s)



VADP Performance Characteristics

• Single ESX server – Creating Multiple Backup Streams

20

Stream #: 1 2 3 4

200

MB/Sec

200 MB/sec

175

MB/Sec

75

MB/Sec

45

MB/Sec

Aggregate Throughput = 495 MB/sec

Based on Cisco/VMware/NetBackup Benchmark Testing

Fibre = 8 Gb

495 MB/sec Creates Significant

Backup Load on ESXi Datastore



VADP Performance Characteristics

• Four Separate ESX servers – Creating One Backup Stream / ESXi

21

Stream #: 1 2 3 4

200

MB/Sec

200 MB/sec

Based on Cisco/VMware/NetBackup Benchmark Testing

Fibre = 8 Gb

200

MB/Sec

200

MB/Sec

200

MB/Sec Aggregate

Throughput = 800 MB/sec

800 MB/sec Creates Less Load

Per ESX Server



VMware Intelligent Policy


Improve Backup Performance & Reliability

VIP Designed for Two Major Tasks

1) Automatically add and backup new and moved VM’s

2) Automatically balance backups across entire vSphere environment (Fibre or network)

• VMs protected based on physical location

– ESX server

– ESX Datastore

– VMware cluster

• VMs protected based on logical attributes

– vCenter folder

– Resource pool

– vApp


Network

LAN

Without VIP – Backup Activity Unbalanced

6 JOBS

0 JOBS

8 JOBS

1 JOBS

Backup Server

ESX 1


VM1 VM2 VM3

ESX 2


VM1 VM2 VM3

ESX 3


VM1 VM2 VM3

ESX 4


VM1 VM2 VM3

Backup Load

• Slower backups

• More backup failures

• Longer backup window


Network

LAN

With VIP – Automatic Load Balancing

2 JOBS

2 JOBS

2 JOBS

2 JOBS

Backup Server

ESX 1


VM1 VM2 VM3

ESX 2


VM1 VM2 VM3

ESX 3


VM1 VM2 VM3

ESX 4


VM1 VM2 VM3

Backup Load

• Faster backups

• Improved reliability

• Less load on ESX

• New VM’s automatically backed up


VMware Intelligent Policy Use Cases Improve Backup Performance & Reliability


Problem: Equalize Backup Load in SAN Environment

• Manage SAN (iSCSI - shared storage) backups at Datastore level

• Managing backups at ESX server does not balance load at storage level

Solution: Will protect every existing and new VM on any Datastore with “PROD” in name

Fully compliant with Storage vMotion

Set “Datastore” resource limit to balance backup load across all Datastores


SAN Backup Data Path

• Shared storage configuration

– Fibre or iSCSI

– Near zero impact on ESX


– No communication with ESX required

28


LAN

ESX Datastore


SAN

SAN Backups - Equalize Load at Datastore

UCS

Fibre or iSCSI

• Design resource limit setting for each Datastore to “2” (total=8)

• Jobs evenly stream data across every SAN connection

Backup Server

2 JOBS

2 JOBS

2 JOBS

2 JOBS

ESX 1


VM1 VM2 VM3

ESX 2


VM1 VM2 VM3

ESX 3


VM1 VM2 VM3

ESX 4


VM1 VM2 VM3


Problem: Network backups – Equalize Load at ESX NIC

• All backup traffic over network – no shared storage available

• Don’t want to saturate ESX network interface

Solution: All powered on VMs on any ESX server in “Production” will be protected.

Unimportant VMs in the “DEVELOPMENT” folder (only used for development) will be skipped

Set “ESX” resource limit to balance backup load across all ESX NICs


Network (NBD) Backup Data Path






31


ESX Datastore

LAN


Network

LAN

Network backups – Equalize Load at ESX NIC

2 JOBS

2 JOBS

2 JOBS

2 JOBS

UCS

• Design resource limit setting for each ESX server to “2” (total = 8)

• No ESX NIC ever saturated with backup traffic

Backup Server

ESX 1


VM1 VM2 VM3

ESX 2


VM1 VM2 VM3

ESX 3


VM1 VM2 VM3

ESX 4


VM1 VM2 VM3


So, How Fast Can I Backup?

Where’s the beef?


0 100 200 300 400 500 600

Translate Performance Into Protection

Normal Backup

Total Number of VMs Protected – Full Backups

3240 VMs

2430 VMs

340 VMs

Single VMware Backup Host Weekend B/U = 60 hours Avg. VM size = 40GB Data Change = 5%

500

1000

1500

2000

2500

3000

Test 2:

450 MB/sec

Test 1:

63 MB/sec

Test 3:

600 MB/sec

Incremental Backup 92 Virtual Machines

5% Data Change

12 minutes! 460 VM’s per hour



Virtual Machine Restore Considerations



General Restore Considerations

• Restore process involves more I/O than backup process

– Disks (vmdk’s) must be first created as target of restore

– Type of vmdk can impact restore speed and I/O required

• Single restore typically won’t saturate restore path

– DR – create simultaneous restore jobs

– As with backup – balance restores across ESX or Datastore

• Slow vCenter can also cause restore perf issues

– Optional: bypass vCenter by restoring directly to ESX(i) server

– Known to significantly improve restore perf in some cases



VMDK Type Can Impact Restore Speeds

• thin

– Space not allocated during creation

– Space is supplied then zeroed out – on demand

– Creation slow if vmdk turns out to be full

• zeroedthick

– 100% of space allocated during creation

– Zeroed out on demand

– Can be faster than “thin” – especially if vmdk is nearly full

• eagerzeroedthick

– 100% of space allocated during creation

– 100% of disk zeroed out during creation

– Could take long time (and create lots of I/O) to complete entire process

37

0%

100%

0% 100%



FedEx VADP Implementation



Technical Principal

FedEx Corporate Overview

• Founded in 1971

• Based in Memphis, TN

• 300,000+ employees

– Spread across 5 operating companies

• Shipping presence in over 200 countries

• 5 major data centers in US

– Additional data centers in EMEA, APAC


FedEx VMware Environment

• DEV / Test

– 8400+ VM’s (growing ~30/mo)

– 90+% Linux

– 500+ ESX servers

– 3 vCenter domains – vSphere 4.1

– Backup methods:

– Guest, Array Snapshot, VADP

• Production

– 4000+ VM’s (growing ~170/mo)

– 90+% Linux

– 400+ ESX servers

– 6 vCenter domains – vSphere 4.1

– Methods: Guest, VADP

• Storage

– EMC, NetApp

– Block / NFS


FedEx Backup Environment

• Backup Environment - NetBackup

• Need for both physical system and VM backups

• Physical environment

– Array based snapshots

– RMAN (Oracle) backups

– NDMP

• Backup targets

– Data Domain

– Tape

– Goal is tapeless

• NetBackup 7.1 servers associated with VM backups ~ 40


VM Protection Criteria

• Simple and scalable, minimal impact to ESX servers

• Disaster Recovery: Legal offsite Requirements

– Backup and restore locally

– Offsite = replication and tape

• Occasional restore of entire VM

– Restore entire environment if necessary

• Single file restore – Linux and Windows

– Most common restore request


Test Environment


Backup Performance Testing

• Critical for meeting SLA’s

– 8 months testing performance/functionality

– 8 hour Backup Window

– RTO is based on tier definitions – normally within 4 hours

• Performance testing methodology

– Multiple Protocol: FCoE, iSCSI, NFS

– CBT, and mapped (single file restore) VMs mandatory

– Transport: NBD (network) and SAN

– Scale tests (# ESX servers, # of VMs, # of datastores)

• Results of performance testing

– 2-3TB/hr (limiting factor target storage) for a single backup host

– Overhead of vCenter

– Media server CPU capacity

– SAN transport method preferred


Restore Functionality Testing

• Ultimate goal is to focus on restore functionality

• What can be restored?

• How fast can restores be processed? (single files indexed, searchable)

• How simple is restore process?

– Single files

– Entire VM


Lessons Learned

• vStorage API Advantages

– Simplified Client Management

– Future Enhancements

• vStorage API – Challenges

– High IO activity on snapshot rollback for applications that do not integrate well with VMware snapshots – Weblogic, Oracle in VMs, et al

– VAAI array support important – affects snapshot performance and # of simultaneous snapshots

– Critical VM’s are sensitive to any VMware activity, DRS, VMotion, VADP

• NetBackup configuration suggestions

– Split backup hosts across vCenter servers

– Use query (VIP) method for VM client selection


Moving Forward

• vStorage API vs proprietary storage snapshots vs guest deduplication

• Measure success as roll out progresses

• 12 month VM count exceed 15,000


NetBackup Appliance

Abdul Rasheed


What is NetBackup Appliance?

NetBackup with Intelligent Deduplication

Symantec Hardened Operating System

Veritas Storage Foundation

Optimized Hardware

Redundant Storage

Sym

ante

c C

riti

cal S

yste

m

Pro

tect

ion

Bu

ilt-i

n W

AN

Op

tim

izat

ion

New! New!

Symantec™ NetBackup Appliances 49

http://www.iconfinder.com/icondetails/35528/128/disk_harddrive_storage_icon




Why NetBackup Appliances?

Building a backup solution

Hardware Acquisition - Updates

Software Acquisition - Patches

Integration – Test – Deploy

Multiple Consoles

Multiple Vendor Relationships

NetBackup Appliances

Simplified Backup – Lower OpEx

Elimination of Integration Complexity

Predictable Performance

One Vendor – Easier Acquisition

Single Pane of Glass Management


Symantec NetBackup Appliances

NetBackup 5220 NetBackup 5020

Backup appliance with dedupe Deduplication appliance

2U to 8U form factor 4U form factor

Master/Media server roles Storage pool role

4TB to 72TB usable capacity 32TB ; up to 192TB with 6 nodes

Deduplication at source and target

Multiple Connectivity Options: Ethernet and Fiber Channel

Transferable software licensing

Replication at no charge


End-to-end protection for Virtualized Enterprise

• Protect up 3,000 virtual machines in one appliance

• Direct vSphere backup – No proxy server required

• Equipped with Symantec V-Ray

• Multiple types of recovery from a single backup pass – single file, applications, application objects and DR image

• Protect through SAN or Ethernet


Copyright © 2012 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

George Winter, Symantec Corporation


Abdul Rasheed, Symantec Corporation


Date post:	20-Aug-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

IM B09 Best Practices for Backup and Recovery of VMware...

Documents