DDN & iRODS · DDN & iRODS iRODS User Group Meeting Cambridge, MA June 18, 2014 David Martin WOS...

DDN & iRODS iRODS User Group Meeting Cambridge, MA June 18, 2014

David Martin WOS Product Line Manager

ddn.com ©2013 DataDirect Networks. All Rights Reserved.

Storage should improve collaboration

… Not make it harder

▶  Distributed, not centralized

▶  Minutes to install, not hours

▶  Milliseconds to retrieve data, not seconds

▶  Replication built in, not added on

▶  Instantaneous recovery from disk failure, not days

▶  Built in data integrity, not silent data corruption


DDN is the World’s Largest Privately Held Data Storage Company

Key Statistics

Overview: Providing the data backbone for the world’s most extreme, data-intensive environments – enabling organizations across multiple industries to maximize the value of their information globally. •  Established: 1998 •  Headquarters: Chatsworth, California USA •  Over 1,000 Customers in 5 Continents

Key Industries: •  High Performance Computing & Life Science •  Cloud & Web Content •  Rich Media •  Intelligence/Federal

Go To Market: •  Global Partners, VARs, Resellers

Industry Validation

World’s Largest Privately-Held Storage Co.

Fast500 Technology Company

Inc. 500|5000 High-Growth Company

Best HPC Storage Product

Best Practice for Digital Media

World Class Set of Global Customers


Big Data & Cloud Infrastructure DDN Announced & Pending Product Portfolio

Analytics Reference Architectures

EXAScaler™ 10Ks of Clients 1TB/s+, HSM

Linux HPC Clients NFS & CIFS

Petascale Lustre® Storage

Enterprise Scale-Out File Storage

GRIDScaler™ ~10K Clients 1TB/s+, HSM

Linux/Windows HPC Clients NFS & CIFS

SFA12KX 48 GB/s 1.7M IOPS 1,680 Drives, Embedded Option

SFA7700 12.5 GB/s 450K IOPS 60 Drives in 4U; 228 Drives in 12U

Storage Fusion Architecture™ Core Storage Platforms

SATA SSD

Flexible Drive Configuration SAS

SFX Automated Flash Caching

WOS® 3.0 32 Trillion Unique Objects

Geo-Replicated Cloud Storage 256 Million Objects/Second

Self-Healing Cloud Parallel Boolean Search

Cloud Foundation

Big Data Platform Management

DirectMon

Cloud Tiering

* Technology Demonstration

Infinite Memory Engine™

Distributed File System Buffer Cache* [Demo]

WOS7000 60 Drives in 4U

Self-Contained Servers

4

Adaptive cache with application & filesystem support for pre-staging, alignment & by-pass

S3

ddn.com ©2013 DataDirect Networks. All Rights Reserved. 5

DATA CENTER

Tape Archive

Users

Pub

lic C

loud

WOS Live Archive Storage

IME

GRIDScaler

High speed processing

Scalable storage engine

WOS Bridge

High Performance Data Center With iRODS tie-in


IRODS Data Grid & WOS Made for Big Data

•  Hyperscale •  Distributed •  Collaborative

•  Accessible •  Secure •  Flexible

DDN products assure data longevity with low latency over geographies and iRODS adds the important policy management & accessibility layers


WOS Key Feature Breakdown

Fully-Integrated Object Storage Appliance •  60 Drives in 4U •  WOS7000 nodes support in-

node failover for ideal HA

Federated, Global Object Storage Namespace •  Up to 256 nodes per cluster •  Across 64 geographies, •  Replicate data with smart policies for

performance and/or storage efficiency on a per-object basis

Object Disk Architecture •  NOFS-no Linux file I/Os, no fragmentation, •  Fully contiguous object read and write

operations for maximum disk efficiency

Latency-Aware Access Manager WOS intelligently makes decisions on the best geographies to get from based upon location access load and latency

User Defined Metadata and Metadata Search •  Applications can assign their own

metadata via object storage API •  Supports batch search of user metadata

DeClustered Data Protection •  No hard tie between physical disks & data. •  Failed drives are recovered through

dispersed data placement •  Rebuilds only data, not drives

Flexible Data Protection Supports any combination of replication, erasure coding & Distributed Data Protection

REPLICATION

Self-Healing Architecture •  Protection from node, Disk &

Data Center failure •  Completely automated


Why DDN WOS & iRODS are perfect complements

WOS extends iRODS by… ▶  WOS makes iRODS more performant & scalable

•  Single iCAT DB entry references multiple WOS copies •  iRODS retrieves lowest latency WOS object instance

▶  WOS enables iRODS users to implement better QOS •  High speed access to distributed project data •  Long tail data can be stored on deep archive WOS Nodes

▶  WOS dramatically improves iRODS storage efficiency & availability •  Self healing – at local node SAS bus speeds which increases data durability •  Highly available – no interruption if disk, node, or site failure •  WOS implements highly efficient Global Object Assure •  Lowest TCO in the industry, <1 FTE for multi petabyte multi-site deployments

iRODS extends WOS by… •  Integration with heterogeneous storage environments (multi-vendor disk & tape) •  Additional clients and access methods •  Richer storage rules and migration policies

8


iRODS Standard Operation

iRODS Server

JBOD ICAT DB

.. iRODS Server

SF

iRODS Server

ICAT DB

.. iRODS Server

iRODS Server

RAID

.. iRODS Server

Paris

BOS

iRODS Operation •  iRules determine file store locations •  iCat maintains file location metadata •  All retrieves get primary file location info from

iCAT (all retrieves use primary)

Store & Retrieve Operation •  iRule policy determines where file is stored

• FileA stored in BOS is replicated to SF automatically by iRODS

•  iCat maintains file location metadata •  File Access always reads primary copy

• FileA read in SF or Paris will go to BOS server (primary)

iRODS Zones

iRule

Replicas: BOS, SF

A

A


Site 1

iRODS Server

RAID ICAT DB

.. iRODS Server

BOS

iRODS Server

JBOD ICAT DB

.. iRODS Server

iRODS Server

ICAT DB

.. iRODS Server

WOS + iRODS Integration WOS Location Intelligence

SF

Paris

WOS as a iRODS Cloud resource • WOS Cluster is defined as IRODS “Cloud Resource” • Multiple WOS replicas stored once in iCAT

•  iRODS utilizes WOS data location intelligence File store / retrieve operation •  iRule policy specifies cloud storage method

•  FileA stored in SF writes to closest node of WOS Cluster

• WOS replicates file to BOS per WOS storage policy • Single entry in iCAT DB

•  File read always reads from closest WOS location •  FileA read in BOS will retrieve from BOS WOS node •  FileA read from Paris will retrieve from closest location

(normally Boston)

WOS Cluster

iRODS Zones

iRule

Replicas Cloud

A

A

iRODS optimizations for WOS is available today


Cluster “Acme WOS 1”

3

40 ms

80 ms 10 ms

Los Angeles Latency Map

San FranciscoNew YorkLondonTokyo

Power of WOS Location intelligence Minimize Latency & Enable True Collaboration

Data Locality & Collaboration 1.  LA site user edits video “A”, which replicates to

Mexico City & Tokyo based on policy 2.  MP Gateway immediately synchronizes iCat

metadata DB with Tokyo user 3.  Tokyo user requests video “A” for processing,

WOS-Lib selects Tokyo site (lowest latency) & retrieves for the user

4.  The Tokyo user extracts frames from the video & writes to WOS (new object), which replicates to Mexico City & LA & synchronizes the iCAT DB’s

Zone Los Angeles

10.8.24.101 . .

10.8.24.105

10.8.25.101 . .

10.8.25.105

10.8.26.101 . .

10.8.26.105

A A iRODS Server

WOS-Lib

Tokyo User

A

30 ms

80 ms

10 ms

Tokyo Latency Map

Best viewed in presentation mode

Los Angles Mexico City Tokyo

IRODS Server

WOS-Lib

2

Real Time Editing App

BB

A A

1

Zone Mexico City

Zone Tokyo

B

44

3

Los Angles User

Real Time Editing App

1

2

3

4


WOS Replication

Replication delivers both multi-site collaboration + data & disaster protection

▶  Select either Asynchronous or Synchronous replication per policy •  Definable on a per-object basis

▶  Synchronous Replication replicates to remote sites to insure data protection before acknowledging a successful “write” to the application

▶  Asynchronous replication to remote WOS nodes increases performance •  Two copies of data are always written to “local” (lowest latency) node before “PUT” call

returns OID to minimize/eliminate risk of data loss •  Replication occurs in background, once replication successfully completes, extra local

copy is removed •  Especially useful for big files and big data sets


WOS Local & Replicated Object Assure

Replicated Object Assure •  DR for OA storage methods •  Sync & Async replication supported •  Optimized for large objects •  Two OA replicas provide equal protection as 3

standard replicas

NDA Confidential Information

Splits files into 10 shards

WOS Replicated OA (ROA)

Application Servers

Data Center 2

Data Center 1

1 2 1 2 9 8

1 2 1 2 9 10

1 2 1 2 9 8 1 2 1 2 9 10

… 1 2 1 2 9 10

Asynchronous Replication

Writes @ LAN line speed WAN speed

ONLY on read failure

Data Center Failure

All shards stored locally

Reads @ LAN line speed

5 6 7 8

P1

P2

1 2 3 4 1

“PUT”“GET”

Client App``

WOS-Lib

WOS Local OA (LOA)

Local Object Assure: •  Erasure-code based declustered data protection

•  8+2 (Data + Parity) for single copy protection •  Each WOS node can withstand up to

two concurrent drive failures without loss of data or availability

•  Local Data rebuilds


WOS Global Object Assure & Replicated Object Assure

Global Object Assure

•  A minimum of 3 data centers must be deployed to provide disaster protection

•  High storage efficiency •  Data rebuilds from drive failures occur @ SAS Bus

speeds

Fastest LAN Path = Performance (unless DR scenario)

•  Two data centers required for full DR •  Store/retrieve data @ full LAN line speeds •  Data replicates to remote asynchronously @ WAN

Speeds •  Only retrieve data over slower WAN in DR scenarios

14

Dual encodes files & spreads across 3

data centers

Stores single data copy across 3 WOS

zones Splits files into

10Shards across 2 data centers

WOS OA Replicated

Application Servers

Data Center 2

Data Center 1

1 2 1 2 9 8

1 2 1 2 9 10

1 2 1 2 9 8 1 2 1 2 9 10

… 1 2 1 2 9 10

Asynchronous Replication

Writes @ LAN line speed

Data Center 2

Data Center 1

Data Center 3

1 2 3 1 2 3 15 16 15 16 17 18 17 18

WAN speed ONLY on failure

WOS Global Object Assure

Data Center Failure

All shards stored locally

Application Servers

The Net-Net - Ultimate customer flexibility

Reads @ LAN line speed

Best viewed in presentation Mode


Performance Optimized Tier

Geo-location & Collaboration (Active Archive)

Capacity Optimized (Deep Archive)

DDN-IRODS Summary Data Center View

15

WOS

IME Bridge

PFS

ES/GS

TSM / HPSS

Applications and Users

WOS

IME Bridge

PFS

ES/GS

TSM /ACSLS

Data Center 1

Bridge Bridge

Data Center 2 Additional Data Centers

Namespace Federation


Building a DDN & iRODS storage Grid

Seattle

Austin Tx (UT) UNC (RENCI)

Karlsruhe, DE

Replication &

Multi-site Collaboration


Summary WOS & iRODS

WOS – iRODS Integration"•  Petabyte scalability: Scale out by

simply adding storage modules"•  Unrivaled Simplicity: Management

simplicity translates directly to lower cost of ownership"

•  Self-Healing: Zero intervention required for failures, automatically recovers from lost drives"

•  Rapid Rebuilds: Fully recover from lost drives in moments"

•  Replication Ready: Ingest & distribute data globally"

•  Disaster Recoverable: For uninterrupted transactions no matter what type of disaster occurs"

•  File Layout: Capacity and performance optimized"

•  Object Metadata: User-defined metadata makes files smarter"

UserCan Search, Access, Add and Manage Data

& Metadata

iRODSMetadataCatalogTrack data

iRODS Data SystemiRODSRule Engine

Track policies

WOS Object StoreWOS ObjectSupervisorGlobal Data Management

WOS Data Locality MgrPolicy-‐driven Replication

Date post:	21-Aug-2018
Category:	Documents
Upload:	dinhhanh
View:	217 times
Download:	0 times

DDN & iRODS · DDN & iRODS iRODS User Group Meeting Cambridge, MA June 18, 2014 David Martin WOS...

Documents