YOU NEED TO KNOW
ABOUT DIGITAL STORAG
Maybe you already have a digital archive – but if you have, congratulations!
You’re in the minority. The DPP’s conversations with production teams tell us
it’s more likely you’ve got a room full of dusty tapes, or a shelf full of external
hard drives. Maybe with an Excel spreadsheet of what’s on them. Or maybe
not. Either way, you are not alone.
And if that’s what your programme storage is like now, you’re probably dreading
what it’s going to be like when you move to digital delivery and storage.
In practice the principles behind storing files are not all that different from
those for keeping tapes. You’ll still have to decide what material to keep, for
how long, and how you’re going to find it again. But as you begin to store your
programme masters digitally, you’ll notice you are also starting to do things
very differently. Some things will prove more difficult and complicated; others
will feel much better.
This introductory guide is designed to assist you. Without any jargon.
It will help you think about how to store your master files and valuable rushes
in a cost effective, well-organised and commercially useful way. Whether
you’re designing your own in-house archive or paying someone else to store
your material, the questions you’ll need to ask are the same. This guide will
work through those questions and help you decide what sort of storage you
Next year the DPP will be publishing a more definitive, and more technical
guide to storage in the broadcast industry. In the meantime we hope this
simple introduction will get you underway.
Broadcasters may or may not compel you to keep your masters: but just like
with tape masters, it’ll almost certainly be in your own company’s interests
to maintain a copy anyway.
The primary reason for this is that you can’t rely on the broadcaster to
preserve your content. Many broadcasters delete their copy of the file after
their rights window expires, and, just as with tapes, there’s nothing to oblige
them to provide a copy of your programme back to you at a later date. The
same goes for distributors. Don’t make any assumptions about how they’ll
store your masters or how long they’ll keep them: look at your contract to
see what they’ve committed to do. Ultimately if you own the Intellectual
Property (IP), it’s for you to decide how best to keep your master material
safe and accessible.
Master is in itself a not entirely straightforward term. In a file-based world,
the copy of your programme you send to the broadcaster in the AS-11
DPP format is your delivery master: the one used by the broadcaster for
transmission, and for their own storage. In the event any late changes are
made to your programme, or if you have different programme versions as
part of your deliverables, you may also have different delivery master versions.
Why store my media?
PAGE 3 VERSION 2.0
But you will also have an edit master: the version of your programme as you
completed it on whatever editing platform you were using. If you wanted to
re-edit the programme at a later date, you would probably want to use this
master, together with your original edit information (your EDL).
So you will want to consider what to do with delivery masters, delivery
versions, and edit masters – as well as rushes and other programme elements.
You are likely to need to store each of these types of material in different
formats. You might keep your delivery master as an AS-11 DPP file; an edit
master in its edit format; and rushes and programme elements in the medium
in which they were originally created. Seek expert advice if you’re unsure.
WHY STORE MY MEDIA?
PAGE 4 VERSION 2.0
It’s very common for people to refer to any storage arrangement as an archive,
but strictly speaking it almost certainly wont be. Archives – tape or file – are
actually very rare. The word archive means not just a store but a rigorous set
of policies and preservation standards that are formally, and internationally,
defined. All archives incorporate systems that manage the storage so you can
administer, search and find your material. You may be able to build or buy into
such a service – although this guide will help you consider whether an archive
in this formal sense is what you really need.
But if what you want to do is simply store your programme or other files in
a place that’s secure and where you’ll be able to get them out again in many
years time, then what you are after is probably best referred to as a library.
The table below describes storage in three different categories, defined not
by the technology but by the way it is used. These categories have recently
been adopted by the BBC, ITV and Channel 4.
What is a store?
PAGE 5 VERSION 2.0
Material In the WiP material is being actively used to make a show or series. Once the edit is complete and the programme successfully delivered and signed off by the broadcaster, this material is likely to be moved out of the WiP.
Speed, Resilience and Quality
The performance of WiP storage may vary according to the task. Is it serving multiple, simultaneous edits or just a single edit? How close to transmission deadline is the show being cut? Generally speaking however WiP storage is high performance, and expensive.
Access will be limited to only those involved in producing and post-producing the programmes.
Frequency Of Use:
almost constantly while in post-production.
will change frequently, and vary by production; different productions may need different kinds of WiP storage.
Volumes: will vary for each show. They can be enormous while the programme is in post-production – but then suddenly reduced when that programme finishes its edit.
Costs: are allowed for within the production budget, and sit in the post-production lines of that budget.
Work in Progress Store (WiP):A WiP holds and manages media which enables
productions to edit and post-produce programmes.
2WHAT IS A STORE?
PAGE 6 VERSION 2.0
Material in a Library material is held close to production teams so it can be exploited commercially or creatively in the future. It will be relatively quick to access compared with an Archive, but it doesn’t necessarily have to be instantly to hand.
Access will be determined by the production company’s own policies. It is likely that access will be limited to named individuals for reasons of security.
Frequency Of Use:
The Library is likely to be accessed frequently for quick re-versioning or the re-use of memorable sequences or shots. However usage won’t be as constant and time-critical as the WiP.
will change over time, based on production workflows and working practices; but it will nonetheless be possible to have some standard company policies and processes.
Volumes: may vary dramatically over time in response to commissions, international sales, editorial priorities and the technical quality of the material. In theory the amount of storage required could go up or down. But in practice it only ever seems to go up – so you might as well accept storage volumes will gradually grow!
Costs: will be met by the production company and weighed up against the commercial and creative value of keeping the material in the Production Library. These costs may be charged back to a progamme or series, or may become part of the company overhead.
Production Library:A Production Library holds any digital assets (rushes, edit projects,
‘wavs’, stills, audio, key production paperwork, masters etc.) which are by-products of the production process.
2WHAT IS A STORE?
PAGE 7 VERSION 2.0
Material An Archive should hold finished material and any other material (rushes, edit masters, stills, audio etc.) which the company decides to retain, or it is legally required to retain. A company could decide to keep all finished programmes and rushes, but it is more likely they will decide to retain only those that have potential re-use value.
Access may be granted, as required, to production teams or potential buyers. Depending on the design of the Archive, it may have a self-serve capability which enables users to view, select and download. The administration of the Archive itself however will be restricted to a small number of authorised individuals.
Frequency Of Use:
Occasional. This is a long term preservation store rather than storage designed for quick and frequent access.
Requirements of Archive storage will change relatively slowly. The greatest area of innovation and change will be in functions which enable people to search and access the material.
Volumes: will increase relatively predictably over time.
Costs: are likely to be borne centrally by the company or group.
Archive:An Archive is where a company or group keeps its valued material
and makes sure it is safe for the future. It is also sometimes referred to as the ‘deep archive’ or ‘preservation archive.’
2WHAT IS A STORE?
PAGE 8 VERSION 2.0
At the beginning of the production process, you will need to consider the type
of programme you’re making, and the kind of life you want it to have. Will it
be sold internationally, or just be shown in the UK? Are the rushes likely to
have re-use value within other programming? Are you likely to re-version or
update the show at a later date?
Thinking about re-use may encourage you to shoot differently, with a view to
the longer term value of the rushes. It may lead you to log high-value material
carefully and to ensure that this information is created in such a way that it
can be stored with the material in the longer term.
And if you think you’re now getting into a degree of complexity, you’d be right!
If you are creating a log, then you are creating metadata (which essentially
means data about data – your programme or rushes material themselves
being data). As soon as you start creating metadata you need to know it will
remain accessible and readable when you share or keep it. Make sure you
get some expert advice.
What should I keep?
PAGE 9 VERSION 2.0
Above all, be honest with yourself. How often have you actually re-used
footage? When have you ever really had time to hang around on a shoot to
capture extra GVs? If you don’t do these things now, you might want a policy
of saving no rushes at all! It would certainly avoid a whole lot of hassle and
cost. If you do want to store rushes however, they will need to be carefully
logged and catalogued if you want to stand a chance of finding them again.
The key point here is if you’re going to do it, you’ll need to do it properly.
3WHAT SHOULD I KEEP?
PAGE 10 VERSION 2.0
The danger in asking an expert what kind of technology you should choose for
the storage of your file-based material is that it’s rather like asking a financial
advisor how you should keep your money: they might actually tell you. Then
you’ll be sorry.
So here’s as simple an answer as we can manage:
Portable hard drivesBrace yourself for an inconvenient truth: you shouldn’t even think about
doing this. Pity, because it’s very probably the way you’re doing it right
now. The problem is that hard-drives fail – especially if they are not
in constant use. A shelf full of neatly labelled hard drives looks very
organised. But it’s a disaster waiting to happen; and lots of broadcasters
and indies have already found they didn’t have to wait very long.
Storage serverThis is the non-portable kind of hard drive. Sometimes it’s referred to
as spinning disk. It actually consists of a number of hard drives brought
together in a server. It’s industrial strength and although it isn’t infallible,
Where should I keep it?
PAGE 11 VERSION 2.0
it does have built-in safeguards against going wrong. The key thing
will be to set this up as a RAID, so that failure of any individual disk or
other component won’t corrupt or destroy the contents of the server.
The storage server can be accessed from a desktop machine that sits
on the same computer network. Files can be copied backwards and
forwards providing the access to store and retrieve media. It may be
possible to review clips and programmes directly, depending on the
number of people doing this at the same time and the connection
speeds within your office. This is a perfectly good form of file-storage,
but gets expensive if you have a lot of it.
Data TapeConfusingly this is sometimes referred to as tape! It’s actually very
different since it’s designed to keep lots of computer files rather than
a single stream of video. This format is tailor-made for long-term
storage and can provide a more stable way of keeping content than
spinning disk. This is most likely to be an LTO or other proprietary
format. It’s relatively cheap and very reliable – as long as you take
care to control environmental conditions like temperature, humidity
The only downside is that you need a way of reading the data tapes.
Larger stores of data tapes tend to be contained inside a cabinet with
barcodes to identify them, and a robotic arm to collect and insert
4WHERE SHOULD I KEEP IT?
PAGE 12 VERSION 2.0
them into data tape machines. Usually the material is then moved
onto a storage server. So commonly a storage server is used in
conjunction with a data tape set-up, with material being moved off the
expensive server onto the cheaper data tape store once it is needed
less often. It will be necessary to migrate these data tapes to a newer
format periodically, as data stored on older data tapes can’t always
be restored by newer systems.
So although a data tape in itself may be cheap, the system you need
to put around it to make it usable is actually pretty costly to create
You can keep and read data tapes without a robotic system. Desktop
tape drives are available and can be connected to a computer to allow
data tapes to be read and written. It will be necessary to copy files to
a local desktop before viewing. In this case, it may be easier to keep
a lower quality copy, or a proxy, of media which is needed frequently
somewhere more accessible such as a local server. These proxies can
be removed when no longer needed, or recreated if a disk drive fails.
In addition to data tapes, there are other removable media storage
options, such as optical disks, now available.
4WHERE SHOULD I KEEP IT?
PAGE 13 VERSION 2.0
Storage asa service
Folders Spreadsheet Web Browser
Storage Management System
Library Shelf Library Shelf
How will you store
your digital media?
HARD DISK DRIVE STORAGE
STORAGE AS A
What do you
store it on?
Where do you
How do you
How do you catalogue
and search for things?
How do you view clips and
We’ve already warned against the most common pitfalls: failing hard drives;
corrupted disks within servers; and data tapes rendered obsolete by the
launch of a new versions.
Beyond these big three, if you had to boil considerations about file storage
down to one single word it would probably be security. How can you be
sure your precious material is safe? How do you know it can’t be deleted by
accident? How can you protect it from fire or theft? Who can you trust with
it? And how do you know you’ll still be able to get at it in ten, twenty or fifty
This is a huge subject. But here are three key things to consider:
Two + OneThere is a simple rule of thumb for keeping file-based material safe, and
it is known as two plus one. This means you have two copies of your file
kept in separate places (ideally in different buildings far away from each
other), and a further off-line copy. By off-line we mean that it can’t be
deleted by the system that can delete the first two.
How will I know it’s safe?
PAGE 15 VERSION 2.0
In many ways it’s that plus one copy that you should think about
most, since it is your ultimate safety copy. So make sure it is in a
place with good physical security, fire systems and so on. Of course
if that third place is the cloud, then the building in question may be a
purpose-built, high spec, data centre in another country.
To the ninesIf you are buying your file-storage as a service, make sure you look
at the contractual small print. What is the company promising you
about how accessible your material will be? Often this is expressed
as percentage availability, e.g. 99.99% (which is equivalent to less
than one hour across a whole year when your footage may not be
accessible to you). A typical enterprise agreement with a cloud
storage provider, for example, could be 99.999%, or five nines – or
just a few minutes a year when you can’t get your material. This level
of availability should be considered in conjunction with the number
of separate stored copies in A above.
Forever is a long timeEver drawn up a will? It’s amazing how being forced to think about
the unthinkable leads you to ask all sorts of questions you never
thought to ask before. The same is true when thinking about precious
programme material that you want to maintain over a period of
decades rather than months or years.
5HOW WILL I KNOW IT’S SAFE?
PAGE 16 VERSION 2.0
If you have material you want to keep for as long as you can imagine,
then you can be sure that during its lifetime there will have been
technical advances that will have rendered obsolete the format on
which it was originally stored. So you will need to know that, over time,
your material can be migrated to whatever is the new format. If you
are creating your own file store, you should build this upgrade into
your cost model. If you are buying a service, then take a good look at
the small print around upgrading.
HOW WILL I KNOW IT’S SAFE?
PAGE 17 VERSION 2.0
This is really important. It’s no use storing something unless you can find
it again when you need it. For a library or an archive you’ll need four key
1 Searchwhich enables you to look for content – or elements of that content;
2 Catalogue which organises your metadata, or information, about your content;
3 Physical management which looks after the storage and moves the files;
4 Review which enables you to browse, view and even publish your content.
These elements could all come together in a Media Asset Management
(MAM) system, or as a number of different tools all linked together. In either
case there are some golden rules to consider:
How will I find it?
PAGE 18 VERSION 2.0
1 Give each file a Unique Identifier. This is a number which is distinct from all others in the archive or library.
2 Decide how you’ll name each file. You’ll need a logical system (File Naming Convention) to make this easy and consistent.
3 Make sure your unique identifiers and file names enable you to distinguish between, and keep track of, different versions of a file.
4 Decide what information (metadata) you want to keep about a file.
5 Decide whether to keep that metadata with the file itself, or in a separate database, or both. If both, decide which you’ll regard as the master set of information.
6 Work out how best to catalogue (organise) the metadata so it’s most useful.
7 Figure out how you will search for a file and view it. This will require either a media asset management system (MAM) or separate tools linked together.
All this is just as tricky as it might appear, and may be one reason why you
ask a third party to store your material. If you create a library or archive within
your own company you’ll need to hire someone who understands all this!
The Magnificent Seven:Tips for organising a digital store
6HOW WILL I FIND IT?
PAGE 19 VERSION 2.0
There are two types of access: the first is direct access, which enables people
to directly control the material itself. This means they can move it, edit it and,
most importantly, copy or delete it. The second type of access enables people
to see and use a copy of the material, commonly known as a proxy, which will
not change the master file, unless it is later conformed. This is rather like the
public having access to copies of BBC programmes through iPlayer – but not
having access to the original material.
So think carefully first about who is going to administer the material in the
library or archive. Then think about who you want to be able to make use of it.
Is it just for the benefit of the production company or group? Or do you want
to offer a shop window for potential buyers?
If you have answers to all these questions, congratulations: you have just
written what archive experts call an access rights policy.
How can I stop the wrong people from getting in?
PAGE 20 VERSION 2.0
We all know how painful clearing the hard drive on your computer can be if
it’s full to bursting and you need to free some space. What should I delete and
what should I keep? There’s real danger of an even more painful and expensive
scenario with a digital archive if you don’t have a pre-agreed set of criteria
which help you not to become overwhelmed by the sheer volume of stuff.
To avoid such a nightmare, you’ll need to consider a couple of things which
at first glance sound like the same thing: what to retain, and what to delete.
Retention is about what you actively plan to build up in your store. As
we mentioned in Section 3, deciding what to keep is important. Based
on your businesses’ needs, you can devise a simple retention policy for
different types of material. Will you keep a copy of every programme in
which you have the rights for as long as those rights exist? If so, don’t
forget to also consider the underlying rights: for example, you might
have the rights to the programme, but do you still have rights to re-show
the contributions within it? Also bear in mind any obligations you have
through your contracts with broadcasters or others.
How long should I keep it?
PAGE 21 VERSION 2.0
Deletion is about what you actively plan to get rid of – to save money on
storage, and keep your store more manageable – so it’s also important
to have a deletion policy. Perhaps you’ll delete material after a certain
period if it hasn’t been reused? Perhaps you’ll delete material to which
you haven’t got rights? Maybe you’ll delete old rushes material twelve
months after programme transmission?
Both retention and deletion policies will be influenced by how much you’re
willing to invest in storing your material.
Programme files are big.
A one hour programme in the DPP format is approximately 50 GB, so twenty one hour programmes take up 1 TB. But that’s the easy bit. An hour of HD rushes takes around 50 GB too. And an hour of rushes shot in 4k requires a cool 100 GB… at least.
8HOW LONG SHOULD I KEEP IT?
PAGE 22 VERSION 2.0
You don’t! Just as in the analogue world, you’ll need to work hard to avoid
your stored content becoming technologically obsolete. Older ones among
us may have Betacam or U-matic copies of our programmes in the attic. We
can’t bring ourselves to throw them away; but we’ll never find a machine on
which to play them.
In a digital world you will over time need to migrate your digital media from
one hardware or software generation to the next. The aim is to preserve the
content and maintain the ability to retrieve, display and use it. Keeping up
with new generations of data tape (e.g. LTO) so that your stored media does
not become inaccessible (see Section 4 C ) is just one reason why you’ll need
to have preservation and migration policies in place. Don’t forget to also make
someone responsible for implementing them.
Some types of material are likely to have more value to you than others, so
your migration policy could be tiered (e.g. Gold for highly valued, or culturally
or legally important material; Silver for items of middling value; Bronze for the
less important pieces). You could prioritise your time and money on migrating
How do I know it will
PAGE 23 VERSION 2.0
Gold valued content, you may even decide to delete the Bronze content rather
than migrate it. Just as with other policies, your approach to migration will be
determined by the content you’re storing and your own businesses’ needs.
You will also need to keep an eye on the long-term readability of the data
itself. Data Tapes can be written in a variety of formats most of which are not
interchangeable. You’d be wise to take expert advice on this as well as on the
choice of file format in which to store media and metadata. Openly accessible
(rather than proprietary) formats are generally safer bets in the long term.
9HOW DO I KNOW IT WILL ALWAYS PLAY?
PAGE 24 VERSION 2.0
Storing your file-based material will carry a cost. But there are costs in the
tape model too. Although the cost of storing tapes doesn’t show up in a
specific budget line; to get a sense of the true cost you only have to consider
the square footage of the space they occupy, the cost of hiring a tape deck
or edit suite to look at them, and the courier bill for shipping them around.
In a file-based world the costs of storage are more transparent. You’ll either
be paying someone for a service, or you’ll be paying for servers, software
and connectivity. A well-managed file-store can unlock potential income,
however – making international distribution and re-purposing of content a
whole lot quicker and easier. And if you decide to create your own storage
capability, the infrastructure you’ll require to store your programme masters
will also benefit you in other regards – such as sending file-based material to
customers or post-houses, accessing web-based viewing and editing services,
and so on. So the investment may go further than you think.
In 2013, Mediasmiths authored a report for the DPP entitled The Coming Storm
The-Coming-Storm.pdf ). The report provides an in-depth assessment of
how far the television industry has progressed to being able to make effective
use of cloud-based services. The report also contained a comparison of the
relative cost (in 2013) of storing 1 GB of material in various ways for one year:
How much will it cost me?
PAGE 25 VERSION 2.0
Portable Hard Drives (not recommended) £0.20 *
Data Tapes (e.g. LTO 6) £0.05
* This varied between £0.16 and £0.38 depending on the capacity of the
drive, the interfaces and the brand. The figure of £0.20 is based on using
two 2 TB drives.
Of course, it is impossible for a simple guide like this to help you calculate
the total cost of ownership of a storage system; or to calculate the long-term
cost of a storage service. Your decision about which storage medium to go for
will also depend not just on its relative affordability, but also on your existing
technology and IT infrastructure; the value and quantity of your content; and
other business drivers unique to your company or group. In the case of Cloud
the costs will be highly dependent upon how often you’ll need to access your
content: it’s a rule of thumb for cloud services that it’s cheap to put your
material in, but expensive to take it out.
10HOW MUCH WILL IT COST ME?
PAGE 26 VERSION 2.0
A FINAL WORD
We hope this guide has helped you think through how best to store your
digital media in a cost effective, well-organised and commercially useful
way. Even if you’ve decided to outsource your storage, we hope we’ve helped
you reflect on all the key questions you’ll need to ask your service provider.
The DPP plan to publish a more comprehensive guide to digital storage
in 2015, but in the meantime here’s a reminder of the key considerations
covered in this concise guide:
1Have you decided what to store and written a
2Will you store your material in a library or an
archive or both?
3Will you create your own library or archive or outsource these to a post production company or other service provider?
4 If you’re creating an in-house store, which storage medium(s) will you use?
5 Have you considered how much storage you’ll require?
6 Have you thought through the level of resilience, access speed and security you’ll need?
7 Have you decided how to search, catalogue, and physically manage and review your content?
8Have you written an access policy and figured out how to give the correct people access to review and edit?
9 Have you written a deletion policy and worked out how best to implement it?
10Is someone in your organisation now responsible for preservation and migration and are policies in place for this?
This DPP production was brought to you by Rachel Baldwin, with Mark
Harrison. Vlad Cohen has yet again made everything look lovely. Huge thanks
are due to ITV, Channel 4 and BBC colleagues whose work we have drawn
upon, especially Tim Davis, Heather Powell, Steve Jupe, Andy Tennant, Kevin
Burrows, Alan Whiston, Andy Corp, Andy King, Holly Flanagan, Dale Grayson
and Steve Daly. Thanks also to Niall Duffy from Mediasmiths, whose report
The Coming Storm is an important addition to this simple guide.
Copyright © 2014-2015 Digital Production Partnership. All rights reserved.
First published 2014.