Developer Summit Agenda:

Nathan Thompson – CEO Spectra LogicSteven Easter – BlackPearl Product ManagerJeff Braunstein – Senior Product Manager, Client DevelopmentSharon Shabtai – Senior Software Engineer, Client DevelopmentSven Howarth-Moore – Professional Services Lead for CatDV

BlackPearl 2016 Developers Conference

Nathan ThompsonCEO & FounderSpectra Logic Corporation

Spectra Logic HeadquartersBoulder, CO

2U BlackPearl + T50e Tape Library BlackPearl + T120 Tape Library BlackPearl + T200 Tape Library BlackPearl + T380 Tape Library BlackPearl + T680 Tape Libraries

80TB Raw Disk Storage 160TB Raw Disk Storage 928TB Raw Disk Storage 1,696TB Raw Disk Storage 6.1PB+ Raw Disk Storage

300TB Uncompressed Tape Storage 720TB Uncompressed Tape Storage 1,200TB Uncompressed Tape Storage 2,280TB Uncompressed Tape Storage 4,020TB Uncompressed Tape Storage

Multi–RackFull RackHalf RackQuarter Rack

Spectra’s BlackPearl S3 Gateway To Tape

Eighth Rack

System Configuration:

BlackPearl 4U Deep Storage Gateway• 6.1PB Raw Capacity Cache

5 Frame TFinity Tape Library5,590 LTO Slots• 13.9PB Uncompressed Capacity

using LTO-6 tape technology

• 33.5PB Uncompressed Capacity using LTO-7 tape technology

4,239 TS Enterprise Slots• 42.4PB Uncompressed Capacity

using TS1150 tape technology

BlackPearl™ ExaScale Edition TFinity™

Exploring The New World Of Storage

• At the last conference event I spoke about two upcoming publications– Spectra’s Storage Forecast which you can download from the Spectra Logic

web site for free (http://spectra.cc/Igh)– The Book “Society’s Genome”

• Both are now published– I will tell you a few of our findings– Jeff can send you an electronic copy of the book

Major Change In Tape Roadmap

• Tape is in a position to more rapidly deliver declining cost per bit over the next 10 years– Tape has plenty of potential for capacity improvement to reach at least four or five

more doublings of current capacity, entirely due to surface area and Tunneling Magnetoresistive (TMR) heads

– Fujifilm and IBM have demonstrated BaFe media has been proven to reach 220TB or more / cartridge

– IBM and HP have partnered on the same head design at LTO-7 and beyond which gives them 98% market share (at LTO-7 and beyond) with plans to double every two years• This gives users confidence in a roadmap that can span for at least four more generations of drives

– Nearly all of the large cloud companies see a significant role for tape

Here Is The Economic Opportunity For Tape

• With or without HAMR, disk prices will level out at $300 for capacity 3.5” disk drives, for 16TB, or about 2 cents per Gigabyte (raw) in 2019 and will increase with inflation from there

• Assuming constant media price and the industry stays true to a two year improvement cycle– LTO-7 is about 1.6c per Gigabyte now and will reach

~1c in 2017– LTO-8 will .8c per Gigabyte in 2018– LTO-9 tape will be .4c per Gigabyte in 2020– LTO-10 in ~2022 will be .2c per Gigabyte

What Does This Mean For Large BlackPearl/Tape Opportunities?

• Ultra cost effective solutions can be built• Declining cost point into the future• Let’s look at some examples:

10 PB Archive System (no compression)5 PB Nearline Copy + 5 PB Ejected Copy of DataSystems Configured with 12 LTO Drives7x24 Support adds 6%-9% to total US Price, annually

100 PB Archive System (no compression)50 PB Nearline Copy + 50 PB Ejected Copy of DataSystems Configured with 48 LTO Tape Drives7x24 Support adds 6%-9% to total US Price, annually

1000 PB / 1 Exabyte System (no compression)500 PB Nearline Copy + 500 PB Ejected Copy of DataSystems Configured with 144 LTO Tape Drives7x24 Support adds 6%-9% to total US Price, annually

Applications Written To BlackPearl Can Take Advantage Of All Storage Types

• Tape in libraries– LTO– IBM TS

• Ejected tape• SAS disk• Power-Down SMR SATA

disk• Public Cloud (late ‘16/early

‘17)• Future storage media types

THANKS

Questions?

BlackPearl Product UpdatesPresenter: Steven Easter – Product Manager

What Makes BlackPearl RevolutionaryBlackPearl – Is a living archive, preserving digital

data

The most data durability available

Highly efficient tape writing

Replication Targets now available

Two Types of Replication TargetBlackPearl to BlackPearl replication

BlackPearl to 3rd Party (public) Cloud

BlackPearl Replication• Provide two copies of data – two locations• Data is more ‘available’, • if one site is down your app can point to the other BlackPearl

to GET data• Solution then adds some HA capability

• More flexible platform to fit more customer needs

BlackPearl Replication ExampleProcess to setup bi-directional replication

1. At each site: Setup other BlackPearl Replication target • Use Data Port for replication• Set default read preference• Optional – Enable User Replication (access control replication)

2. At each site: Add BlackPearl Target to Data Policy• Different policy for each BP:

• HQ system has a Data Policy for: One Copy on a tape Storage Domain and One Replication Target

• Colo system has a Data Policy for: One Copy on a ArcticBlue Storage Domain and One Replication Target (pointing back at HQ system)

3. Create a Bucket using these Data Policies• Either a new PUT Job will use default Policies for the user and

replication target, or • Best to create the same bucket on both systems first (specify the

Data Policy created in step 2), before a PUT job is done

Tape LibrarySite – Primary – “HQ”

Site – Secondary – “Colo”

BlackPearl Replication Example (cont.)Data flow in bi-directional replication

A. User does A Bulk PUT into Bucket “EditingArchive” at HQ BlackPearli. Creates a S3 Job – e.g. Job 1 on HQ BlackPearl, and Replication Job 1 on Colo BlackPearlii. Both databases are synced with Replicated Job 1 at time of Job 1 creation, that way the same data cannot be sent to

other BlackPearlB. The Application sends data to HQ BlackPearl

i. Data lands in HQ BlackPearl cache, stays here until the Job 1 data has been persisted to all Storage Domainsii. For Example we’ll say the data is replicated to the Colo BlackPearl first, iii. Colo BlackPearl writes it to ArcticBlue quickly

• Replication Job 1 is now complete. Data in Colo BlackPearl cache can be reclaimed when needediv. Tape Drive resources become available at HQ BlackPearl and the Data is now written to Tape

• Job 1 is now complete. Data in HQ BlackPearl can now be reclaimed. Host application can remove any file locks for deleting data from primary storage.

Site – Primary – “HQ"

Cache:

Tape Storage Domain:

Site – Secondary – “Colo”

Cache:

ArcticBlue Storage Domain:

EditingArchiveData Policy: Tape+Replication

EditingArchiveData Policy: ArcticBlue+Replication

Job 1Replication Job 1

HostUser

data datadata

datadata

BlackPearl P Seriesspectralogic.com/products/BlackPearl/specs/

• Very high performance system – up to 3GBps• Customers will need scale-out application & client• Or customer may have multiple different

applications

• Also useful if BlackPearl makes multiple copies

BlackPearl Developer Updates - Agenda• General Program Updates• Certification Program• SDK Updates• End to End Data Integrity• When Files are Safely Archived to BlackPearl• New Deep Storage Browser Demonstration• Java Command Line Interface Demo and Updates

Developer Program Updates• Mature Program• Certification• Simulator and Remote BlackPearl• Spectra BlackPearl Lab• Forums

BlackPearl Certification Program• Goal: Ensure Clients are Robust and Reliable

• Includes the Following Components:• Program Overview and Timeline

• Legal Agreements• Questionnaire

• Test Plan• Testing Performed in Spectra Lab in Boulder

• Certified Clients Able to Participate in Joint Marketing• Retesting Available for New Releases

• Currently have 3 Spectra staff doing certification testing

• Goal: Ensure Clients are Robust and Reliable• Includes the Following Components:

• Program Overview and Timeline• Questionnaire• Test Plan

• Testing Performed in Spectra Lab in Boulder• Certified Clients Able to Participate in Joint Marketing• Retesting Available for New Releases• Currently have 2 dedicated staff doing certification testing

Software Development Kit (SDK) Updates• Four Complete SDKs• Java• C#/.NET• Python• C

• Java and C#/.NET Include “Helper” Functions• All SDKs Can Now Access All 250+ BlackPearl

APIs

End-To-End Data Integrity• BlackPearl includes comprehensive data integrity once data

received• Developers to be concerned with “in flight” data integrity

between client and BlackPearl• Client can use checksums to guarantee integrity• Checksums done on each file or file parts sent to BlackPearl• HTTPS/SSL is a good optional checksum data integrity

method

When Are Files Safely Archived to BlackPearl?• Clients archive groups of files to BlackPearl in “jobs”• Job is “Complete” when files are on all storage targets• BlackPearl’s database tracks the location of files on

storage targets• This database is typically backed up daily• Files are not truly, safely archived until database backup with

their location has been made• Clients should not purge/delete files off of their system

until safely archived to BlackPearl

Deep Storage Browser

Java Command Line Interface

The .Net SDK and Demonstration of Avid PAM pluginSharon Shabtai - Senior Software Engineer

Table Of Contents

• Spectra Logic .Net SDK

• Helper functions

• The Spectra S3 Client

• Instantiating the API example

• NuGet Package

• AVID Plugin

Spectra Logic .Net SDK• Open source (GitHub) can be imported to VS and modify to you needs

• Using .Net 4.5.1

• Latest SDK release version v3.2.7

• The SDK consists of two levels of abstraction:

1. A high level interface (Ds3.Helpers.IDs3ClientHelpers)

2. The core client interface (Ds3.IDs3Client)

Helper Functions• A high level interface that abstracts several very common application requirements.

• Constantly working on adding and improving the API:

o StartWriteJob

o StartReadJob

o StartReadAllJob

o StartPartialReadJob

Helper Functions• A high level interface that abstracts several very common application requirements.

• Constantly working on adding and improving the API:

o StartWriteJob

o StartReadJob

o StartReadAllJob

o StartPartialReadJob

o ListObjects - with and without prefix

Helper Functions• A high level interface that abstracts several very common application requirements.

• Constantly working on adding and improving the API:

o StartWriteJob

o StartReadJob

o StartReadAllJob

o StartPartialReadJob

o ListObjects - with and without prefix

o EnsureBucketExists

Helper Functions• A high level interface that abstracts several very common application requirements.

• Constantly working on adding and improving the API:

o StartWriteJob

o StartReadJob

o StartReadAllJob

o StartPartialReadJob

o ListObjects - with and without prefix

o EnsureBucketExists

o RecoverWriteJob

o RecoverReadJob

Helper Functions• The Helper functions support:

o Parallel Requests

o Using the right Chunking strategy:

Random access

Streaming

o Metadata

o Checksum

o Cancellation token

o Job Events (“Data transferred”, “Item completed”, “On failure”)

The Spectra S3 Client• The Spectra S3 client implements all of the BlackPearl REST API and can be use when the desire

behavior is not implemented in the helper function.

• You can find the full API in our IDs3Client interface

Instantiating the API exampleThe example below shows how to configure and instantiate IDs3Client and IDs3ClientHelpers.using Ds3;using Ds3.Helpers;using System.Configuration;

namespace YourApplication{ class YourClass { public void YourMethod() { // Configure and build the core client. IDs3Client client = new Ds3Builder( "http://ds3-endpoint", new Credentials("access key", "secret key") ).Build(); // Set up the high-level abstractions. IDs3ClientHelpers helpers = new Ds3ClientHelpers(client); // Use functionality from 'helpers' and 'client', preferring 'helpers'. } }}

• More detailed examples can be found in our Examples Project

NuGet Package• The SDK is distributed as a NuGet package for .Net 4.5.1 and above.

• How to get the NuGet package:

1. Download the .nupkg file from the Releases page

2. Use Spectra Logic Bintray NuGet repository

• Avid Technology, Inc.

• Avid 3rd party plugin API

o Archive

o Restore (including partial restore)

o Delete

o Pause and Resume

• Avid and Spectra Logic

o We Created a plugin for BlackPearl

• Avid Technology, Inc.

• Avid 3rd party plugin API

o Archive

o Restore (including partial restore)

o Delete

o Pause and Resume

• Avid and Spectra Logic

o Created a plugin for BlackPearl

• Avid Technology, Inc.

• Avid 3rd party plugin API

o Archive

o Restore (including partial restore)

o Delete

o Pause and Resume

• Avid and Spectra Logic

o Created a plugin for BlackPearl

Best of Breed Partners

Application Architecture

CatDV and Spectra Logic Black Pearl

CatDV UI

Configuration