things

THE DPP’S

YOU NEED TO KNOW

ABOUT DIGITAL STORAG

E

INTRODUCTION

PAGE 2

Maybe you already have a digital archive – but if you have, congratulations!

You’re in the minority. The DPP’s conversations with production teams tell us

it’s more likely you’ve got a room full of dusty tapes, or a shelf full of external

hard drives. Maybe with an Excel spreadsheet of what’s on them. Or maybe

not. Either way, you are not alone.

And if that’s what your programme storage is like now, you’re probably dreading

what it’s going to be like when you move to digital delivery and storage.

In practice the principles behind storing files are not all that different from

those for keeping tapes. You’ll still have to decide what material to keep, for

how long, and how you’re going to find it again. But as you begin to store your

programme masters digitally, you’ll notice you are also starting to do things

very differently. Some things will prove more difficult and complicated; others

will feel much better.

This introductory guide is designed to assist you. Without any jargon.

It will help you think about how to store your master files and valuable rushes

in a cost effective, well-organised and commercially useful way. Whether

you’re designing your own in-house archive or paying someone else to store

your material, the questions you’ll need to ask are the same. This guide will

work through those questions and help you decide what sort of storage you

really need.

Next year the DPP will be publishing a more definitive, and more technical

guide to storage in the broadcast industry. In the meantime we hope this

simple introduction will get you underway.

Broadcasters may or may not compel you to keep your masters: but just like

with tape masters, it’ll almost certainly be in your own company’s interests

to maintain a copy anyway.

The primary reason for this is that you can’t rely on the broadcaster to

preserve your content. Many broadcasters delete their copy of the file after

their rights window expires, and, just as with tapes, there’s nothing to oblige

them to provide a copy of your programme back to you at a later date. The

same goes for distributors. Don’t make any assumptions about how they’ll

store your masters or how long they’ll keep them: look at your contract to

see what they’ve committed to do. Ultimately if you own the Intellectual

Property (IP), it’s for you to decide how best to keep your master material

safe and accessible.

Master is in itself a not entirely straightforward term. In a file-based world,

the copy of your programme you send to the broadcaster in the AS-11

DPP format is your delivery master: the one used by the broadcaster for

transmission, and for their own storage. In the event any late changes are

made to your programme, or if you have different programme versions as

part of your deliverables, you may also have different delivery master versions.

Why store my media?

PAGE 3 VERSION 2.0

But you will also have an edit master: the version of your programme as you

completed it on whatever editing platform you were using. If you wanted to

re-edit the programme at a later date, you would probably want to use this

master, together with your original edit information (your EDL).

So you will want to consider what to do with delivery masters, delivery

versions, and edit masters – as well as rushes and other programme elements.

You are likely to need to store each of these types of material in different

formats. You might keep your delivery master as an AS-11 DPP file; an edit

master in its edit format; and rushes and programme elements in the medium

in which they were originally created. Seek expert advice if you’re unsure.

WHY STORE MY MEDIA?

1

PAGE 4 VERSION 2.0

It’s very common for people to refer to any storage arrangement as an archive,

but strictly speaking it almost certainly wont be. Archives – tape or file – are

actually very rare. The word archive means not just a store but a rigorous set

of policies and preservation standards that are formally, and internationally,

defined. All archives incorporate systems that manage the storage so you can

administer, search and find your material. You may be able to build or buy into

such a service – although this guide will help you consider whether an archive

in this formal sense is what you really need.

But if what you want to do is simply store your programme or other files in

a place that’s secure and where you’ll be able to get them out again in many

years time, then what you are after is probably best referred to as a library.

The table below describes storage in three different categories, defined not

by the technology but by the way it is used. These categories have recently

been adopted by the BBC, ITV and Channel 4.

What is a store?

PAGE 5 VERSION 2.0

Material In the WiP material is being actively used to make a show or series. Once the edit is complete and the programme successfully delivered and signed off by the broadcaster, this material is likely to be moved out of the WiP.

Speed, Resilience and Quality

The performance of WiP storage may vary according to the task. Is it serving multiple, simultaneous edits or just a single edit? How close to transmission deadline is the show being cut? Generally speaking however WiP storage is high performance, and expensive.

Access will be limited to only those involved in producing and post-producing the programmes.

Frequency Of Use:

almost constantly while in post-production.

Business Requirements:

will change frequently, and vary by production; different productions may need different kinds of WiP storage.

Volumes: will vary for each show. They can be enormous while the programme is in post-production – but then suddenly reduced when that programme finishes its edit.

Costs: are allowed for within the production budget, and sit in the post-production lines of that budget.

Work in Progress Store (WiP):A WiP holds and manages media which enables

productions to edit and post-produce programmes.

2WHAT IS A STORE?

PAGE 6 VERSION 2.0

Material in a Library material is held close to production teams so it can be exploited commercially or creatively in the future. It will be relatively quick to access compared with an Archive, but it doesn’t necessarily have to be instantly to hand.

Access will be determined by the production company’s own policies. It is likely that access will be limited to named individuals for reasons of security.

Frequency Of Use:

The Library is likely to be accessed frequently for quick re-versioning or the re-use of memorable sequences or shots. However usage won’t be as constant and time-critical as the WiP.

Business Requirements:

will change over time, based on production workflows and working practices; but it will nonetheless be possible to have some standard company policies and processes.

Volumes: may vary dramatically over time in response to commissions, international sales, editorial priorities and the technical quality of the material. In theory the amount of storage required could go up or down. But in practice it only ever seems to go up – so you might as well accept storage volumes will gradually grow!

Costs: will be met by the production company and weighed up against the commercial and creative value of keeping the material in the Production Library. These costs may be charged back to a progamme or series, or may become part of the company overhead.

Production Library:A Production Library holds any digital assets (rushes, edit projects,

‘wavs’, stills, audio, key production paperwork, masters etc.) which are by-products of the production process.

2WHAT IS A STORE?

PAGE 7 VERSION 2.0

Material An Archive should hold finished material and any other material (rushes, edit masters, stills, audio etc.) which the company decides to retain, or it is legally required to retain. A company could decide to keep all finished programmes and rushes, but it is more likely they will decide to retain only those that have potential re-use value.

Access may be granted, as required, to production teams or potential buyers. Depending on the design of the Archive, it may have a self-serve capability which enables users to view, select and download. The administration of the Archive itself however will be restricted to a small number of authorised individuals.

Frequency Of Use:

Occasional. This is a long term preservation store rather than storage designed for quick and frequent access.

Business Requirements:

Requirements of Archive storage will change relatively slowly. The greatest area of innovation and change will be in functions which enable people to search and access the material.

Volumes: will increase relatively predictably over time.

Costs: are likely to be borne centrally by the company or group.

Archive:An Archive is where a company or group keeps its valued material

and makes sure it is safe for the future. It is also sometimes referred to as the ‘deep archive’ or ‘preservation archive.’

2WHAT IS A STORE?

PAGE 8 VERSION 2.0

At the beginning of the production process, you will need to consider the type

of programme you’re making, and the kind of life you want it to have. Will it

be sold internationally, or just be shown in the UK? Are the rushes likely to

have re-use value within other programming? Are you likely to re-version or

update the show at a later date?

Thinking about re-use may encourage you to shoot differently, with a view to

the longer term value of the rushes. It may lead you to log high-value material

carefully and to ensure that this information is created in such a way that it

can be stored with the material in the longer term.

And if you think you’re now getting into a degree of complexity, you’d be right!

If you are creating a log, then you are creating metadata (which essentially

means data about data – your programme or rushes material themselves

being data). As soon as you start creating metadata you need to know it will

remain accessible and readable when you share or keep it. Make sure you

get some expert advice.

What should I keep?

PAGE 9 VERSION 2.0

Above all, be honest with yourself. How often have you actually re-used

footage? When have you ever really had time to hang around on a shoot to

capture extra GVs? If you don’t do these things now, you might want a policy

of saving no rushes at all! It would certainly avoid a whole lot of hassle and

cost. If you do want to store rushes however, they will need to be carefully

logged and catalogued if you want to stand a chance of finding them again.

The key point here is if you’re going to do it, you’ll need to do it properly.

3WHAT SHOULD I KEEP?

PAGE 10 VERSION 2.0

The danger in asking an expert what kind of technology you should choose for

the storage of your file-based material is that it’s rather like asking a financial

advisor how you should keep your money: they might actually tell you. Then

you’ll be sorry.

So here’s as simple an answer as we can manage:

Portable hard drivesBrace yourself for an inconvenient truth: you shouldn’t even think about

doing this. Pity, because it’s very probably the way you’re doing it right

now. The problem is that hard-drives fail – especially if they are not

in constant use. A shelf full of neatly labelled hard drives looks very

organised. But it’s a disaster waiting to happen; and lots of broadcasters

and indies have already found they didn’t have to wait very long.

Storage serverThis is the non-portable kind of hard drive. Sometimes it’s referred to

as spinning disk. It actually consists of a number of hard drives brought

together in a server. It’s industrial strength and although it isn’t infallible,

Where should I keep it?

A

B

PAGE 11 VERSION 2.0

it does have built-in safeguards against going wrong. The key thing

will be to set this up as a RAID, so that failure of any individual disk or

other component won’t corrupt or destroy the contents of the server.

The storage server can be accessed from a desktop machine that sits

on the same computer network. Files can be copied backwards and

forwards providing the access to store and retrieve media. It may be

possible to review clips and programmes directly, depending on the

number of people doing this at the same time and the connection

speeds within your office. This is a perfectly good form of file-storage,

but gets expensive if you have a lot of it.

Data TapeConfusingly this is sometimes referred to as tape! It’s actually very

different since it’s designed to keep lots of computer files rather than

a single stream of video. This format is tailor-made for long-term

storage and can provide a more stable way of keeping content than

spinning disk. This is most likely to be an LTO or other proprietary

format. It’s relatively cheap and very reliable – as long as you take

care to control environmental conditions like temperature, humidity

and dust.

The only downside is that you need a way of reading the data tapes.

Larger stores of data tapes tend to be contained inside a cabinet with

barcodes to identify them, and a robotic arm to collect and insert

B

C

4WHERE SHOULD I KEEP IT?

PAGE 12 VERSION 2.0

them into data tape machines. Usually the material is then moved

onto a storage server. So commonly a storage server is used in

conjunction with a data tape set-up, with material being moved off the

expensive server onto the cheaper data tape store once it is needed

less often. It will be necessary to migrate these data tapes to a newer

format periodically, as data stored on older data tapes can’t always

be restored by newer systems.

So although a data tape in itself may be cheap, the system you need

to put around it to make it usable is actually pretty costly to create

and maintain.

You can keep and read data tapes without a robotic system. Desktop

tape drives are available and can be connected to a computer to allow

data tapes to be read and written. It will be necessary to copy files to

a local desktop before viewing. In this case, it may be easier to keep

a lower quality copy, or a proxy, of media which is needed frequently

somewhere more accessible such as a local server. These proxies can

be removed when no longer needed, or recreated if a disk drive fails.

In addition to data tapes, there are other removable media storage

options, such as optical disks, now available.

C

4WHERE SHOULD I KEEP IT?

PAGE 13 VERSION 2.0

LibraryShelf

HDCamSR

DigibetaHard Disk

DriveDataTape

Storage asa service

RAID StorageSystem

Storage Server

RoboticTape

Library Cloud

Film

Local VTR

Edit Suite

Local Desktop

Folders Spreadsheet Web Browser

Cataloguing System

Storage Management System

Database

MAM

MAM

API

Web Browser

Spreadsheet

CataloguingSystem

Edit Suite

VHS dub

Local Desktop

Web-basedmedia player

Library Shelf Library Shelf

How will you store

your digital media?

TAPE STORAGE

OLD WAY

HARD DISK DRIVE STORAGE

DATA TAPE

STORAGE

NEW WAYS

STORAGE AS A

SERVICE

What do you

store it on?

Where do you

store it?

How do you

retrieve media?

How do you catalogue

and search for things?

How do you view clips and

programmes?

PAGE 14

We’ve already warned against the most common pitfalls: failing hard drives;

corrupted disks within servers; and data tapes rendered obsolete by the

launch of a new versions.

Beyond these big three, if you had to boil considerations about file storage

down to one single word it would probably be security. How can you be

sure your precious material is safe? How do you know it can’t be deleted by

accident? How can you protect it from fire or theft? Who can you trust with

it? And how do you know you’ll still be able to get at it in ten, twenty or fifty

years time?

This is a huge subject. But here are three key things to consider:

Two + OneThere is a simple rule of thumb for keeping file-based material safe, and

it is known as two plus one. This means you have two copies of your file

kept in separate places (ideally in different buildings far away from each

other), and a further off-line copy. By off-line we mean that it can’t be

deleted by the system that can delete the first two.

How will I know it’s safe?

A

PAGE 15 VERSION 2.0

In many ways it’s that plus one copy that you should think about

most, since it is your ultimate safety copy. So make sure it is in a

place with good physical security, fire systems and so on. Of course

if that third place is the cloud, then the building in question may be a

purpose-built, high spec, data centre in another country.

To the ninesIf you are buying your file-storage as a service, make sure you look

at the contractual small print. What is the company promising you

about how accessible your material will be? Often this is expressed

as percentage availability, e.g. 99.99% (which is equivalent to less

than one hour across a whole year when your footage may not be

accessible to you). A typical enterprise agreement with a cloud

storage provider, for example, could be 99.999%, or five nines – or

just a few minutes a year when you can’t get your material. This level

of availability should be considered in conjunction with the number

of separate stored copies in A above.

Forever is a long timeEver drawn up a will? It’s amazing how being forced to think about

the unthinkable leads you to ask all sorts of questions you never

thought to ask before. The same is true when thinking about precious

programme material that you want to maintain over a period of

decades rather than months or years.

B

A

C

5HOW WILL I KNOW IT’S SAFE?

PAGE 16 VERSION 2.0

If you have material you want to keep for as long as you can imagine,

then you can be sure that during its lifetime there will have been

technical advances that will have rendered obsolete the format on

which it was originally stored. So you will need to know that, over time,

your material can be migrated to whatever is the new format. If you

are creating your own file store, you should build this upgrade into

your cost model. If you are buying a service, then take a good look at

the small print around upgrading.

C

HOW WILL I KNOW IT’S SAFE?

5

PAGE 17 VERSION 2.0

This is really important. It’s no use storing something unless you can find

it again when you need it. For a library or an archive you’ll need four key

functions:

1 Searchwhich enables you to look for content – or elements of that content;

2 Catalogue which organises your metadata, or information, about your content;

3 Physical management which looks after the storage and moves the files;

4 Review which enables you to browse, view and even publish your content.

These elements could all come together in a Media Asset Management

(MAM) system, or as a number of different tools all linked together. In either

case there are some golden rules to consider:

How will I find it?

PAGE 18 VERSION 2.0

1 Give each file a Unique Identifier. This is a number which is distinct from all others in the archive or library.

2 Decide how you’ll name each file. You’ll need a logical system (File Naming Convention) to make this easy and consistent.

3 Make sure your unique identifiers and file names enable you to distinguish between, and keep track of, different versions of a file.

4 Decide what information (metadata) you want to keep about a file.

5 Decide whether to keep that metadata with the file itself, or in a separate database, or both. If both, decide which you’ll regard as the master set of information.

6 Work out how best to catalogue (organise) the metadata so it’s most useful.

7 Figure out how you will search for a file and view it. This will require either a media asset management system (MAM) or separate tools linked together.

All this is just as tricky as it might appear, and may be one reason why you

ask a third party to store your material. If you create a library or archive within

your own company you’ll need to hire someone who understands all this!

The Magnificent Seven:Tips for organising a digital store

6HOW WILL I FIND IT?

PAGE 19 VERSION 2.0

There are two types of access: the first is direct access, which enables people

to directly control the material itself. This means they can move it, edit it and,

most importantly, copy or delete it. The second type of access enables people

to see and use a copy of the material, commonly known as a proxy, which will

not change the master file, unless it is later conformed. This is rather like the

public having access to copies of BBC programmes through iPlayer – but not

having access to the original material.

So think carefully first about who is going to administer the material in the

library or archive. Then think about who you want to be able to make use of it.

Is it just for the benefit of the production company or group? Or do you want

to offer a shop window for potential buyers?

If you have answers to all these questions, congratulations: you have just

written what archive experts call an access rights policy.

How can I stop the wrong people from getting in?

PAGE 20 VERSION 2.0

We all know how painful clearing the hard drive on your computer can be if

it’s full to bursting and you need to free some space. What should I delete and

what should I keep? There’s real danger of an even more painful and expensive

scenario with a digital archive if you don’t have a pre-agreed set of criteria

which help you not to become overwhelmed by the sheer volume of stuff.

To avoid such a nightmare, you’ll need to consider a couple of things which

at first glance sound like the same thing: what to retain, and what to delete.

Retention is about what you actively plan to build up in your store. As

we mentioned in Section 3, deciding what to keep is important. Based

on your businesses’ needs, you can devise a simple retention policy for

different types of material. Will you keep a copy of every programme in

which you have the rights for as long as those rights exist? If so, don’t

forget to also consider the underlying rights: for example, you might

have the rights to the programme, but do you still have rights to re-show

the contributions within it? Also bear in mind any obligations you have

through your contracts with broadcasters or others.

How long should I keep it?

A

PAGE 21 VERSION 2.0

Deletion is about what you actively plan to get rid of – to save money on

storage, and keep your store more manageable – so it’s also important

to have a deletion policy. Perhaps you’ll delete material after a certain

period if it hasn’t been reused? Perhaps you’ll delete material to which

you haven’t got rights? Maybe you’ll delete old rushes material twelve

months after programme transmission?

Both retention and deletion policies will be influenced by how much you’re

willing to invest in storing your material.

Programme files are big.

A one hour programme in the DPP format is approximately 50 GB, so twenty one hour programmes take up 1 TB. But that’s the easy bit. An hour of HD rushes takes around 50 GB too. And an hour of rushes shot in 4k requires a cool 100 GB… at least.

FACT!

8HOW LONG SHOULD I KEEP IT?

B

PAGE 22 VERSION 2.0

You don’t! Just as in the analogue world, you’ll need to work hard to avoid

your stored content becoming technologically obsolete. Older ones among

us may have Betacam or U-matic copies of our programmes in the attic. We

can’t bring ourselves to throw them away; but we’ll never find a machine on

which to play them.

In a digital world you will over time need to migrate your digital media from

one hardware or software generation to the next. The aim is to preserve the

content and maintain the ability to retrieve, display and use it. Keeping up

with new generations of data tape (e.g. LTO) so that your stored media does

not become inaccessible (see Section 4 C ) is just one reason why you’ll need

to have preservation and migration policies in place. Don’t forget to also make

someone responsible for implementing them.

Some types of material are likely to have more value to you than others, so

your migration policy could be tiered (e.g. Gold for highly valued, or culturally

or legally important material; Silver for items of middling value; Bronze for the

less important pieces). You could prioritise your time and money on migrating

How do I know it will

always play?

PAGE 23 VERSION 2.0

Gold valued content, you may even decide to delete the Bronze content rather

than migrate it. Just as with other policies, your approach to migration will be

determined by the content you’re storing and your own businesses’ needs.

You will also need to keep an eye on the long-term readability of the data

itself. Data Tapes can be written in a variety of formats most of which are not

interchangeable. You’d be wise to take expert advice on this as well as on the

choice of file format in which to store media and metadata. Openly accessible

(rather than proprietary) formats are generally safer bets in the long term.

9HOW DO I KNOW IT WILL ALWAYS PLAY?

PAGE 24 VERSION 2.0

Storing your file-based material will carry a cost. But there are costs in the

tape model too. Although the cost of storing tapes doesn’t show up in a

specific budget line; to get a sense of the true cost you only have to consider

the square footage of the space they occupy, the cost of hiring a tape deck

or edit suite to look at them, and the courier bill for shipping them around.

In a file-based world the costs of storage are more transparent. You’ll either

be paying someone for a service, or you’ll be paying for servers, software

and connectivity. A well-managed file-store can unlock potential income,

however – making international distribution and re-purposing of content a

whole lot quicker and easier. And if you decide to create your own storage

capability, the infrastructure you’ll require to store your programme masters

will also benefit you in other regards – such as sending file-based material to

customers or post-houses, accessing web-based viewing and editing services,

and so on. So the investment may go further than you think.

In 2013, Mediasmiths authored a report for the DPP entitled The Coming Storm

( http://dpp-assets.s3.amazonaws.com/wp-content/uploads/2013/09/

The-Coming-Storm.pdf ). The report provides an in-depth assessment of

how far the television industry has progressed to being able to make effective

use of cloud-based services. The report also contained a comparison of the

relative cost (in 2013) of storing 1 GB of material in various ways for one year:

How much will it cost me?

PAGE 25 VERSION 2.0

http://dpp-assets.s3.amazonaws.com/wp-content/uploads/2013/09/The-Coming-Storm.pdfhttp://dpp-assets.s3.amazonaws.com/wp-content/uploads/2013/09/The-Coming-Storm.pdf

Cloud £0.67

Portable Hard Drives (not recommended) £0.20 *

Data Tapes (e.g. LTO 6) £0.05

* This varied between £0.16 and £0.38 depending on the capacity of the

drive, the interfaces and the brand. The figure of £0.20 is based on using

two 2 TB drives.

Of course, it is impossible for a simple guide like this to help you calculate

the total cost of ownership of a storage system; or to calculate the long-term

cost of a storage service. Your decision about which storage medium to go for

will also depend not just on its relative affordability, but also on your existing

technology and IT infrastructure; the value and quantity of your content; and

other business drivers unique to your company or group. In the case of Cloud

the costs will be highly dependent upon how often you’ll need to access your

content: it’s a rule of thumb for cloud services that it’s cheap to put your

material in, but expensive to take it out.

10HOW MUCH WILL IT COST ME?

PAGE 26 VERSION 2.0

A FINAL WORD

We hope this guide has helped you think through how best to store your

digital media in a cost effective, well-organised and commercially useful

way. Even if you’ve decided to outsource your storage, we hope we’ve helped

you reflect on all the key questions you’ll need to ask your service provider.

The DPP plan to publish a more comprehensive guide to digital storage

in 2015, but in the meantime here’s a reminder of the key considerations

covered in this concise guide:

PAGE 27

Checklist

1Have you decided what to store and written a

retention policy?

2Will you store your material in a library or an

archive or both?

3Will you create your own library or archive or outsource these to a post production company or other service provider?

4 If you’re creating an in-house store, which storage medium(s) will you use?

5 Have you considered how much storage you’ll require?

6 Have you thought through the level of resilience, access speed and security you’ll need?

7 Have you decided how to search, catalogue, and physically manage and review your content?

8Have you written an access policy and figured out how to give the correct people access to review and edit?

9 Have you written a deletion policy and worked out how best to implement it?

10Is someone in your organisation now responsible for preservation and migration and are policies in place for this?

PAGE 28

This DPP production was brought to you by Rachel Baldwin, with Mark

Harrison. Vlad Cohen has yet again made everything look lovely. Huge thanks

are due to ITV, Channel 4 and BBC colleagues whose work we have drawn

upon, especially Tim Davis, Heather Powell, Steve Jupe, Andy Tennant, Kevin

Burrows, Alan Whiston, Andy Corp, Andy King, Holly Flanagan, Dale Grayson

and Steve Daly. Thanks also to Niall Duffy from Mediasmiths, whose report

The Coming Storm is an important addition to this simple guide.

Copyright © 2014-2015 Digital Production Partnership. All rights reserved.

First published 2014.

PAGE 29