+ All Categories
Home > Documents > THE DPP’S · 2017-04-25 · PAGE 2 Maybe you already have a digital archive – but if you have,...

THE DPP’S · 2017-04-25 · PAGE 2 Maybe you already have a digital archive – but if you have,...

Date post: 08-Jul-2020
Author: others
View: 0 times
Download: 0 times
Share this document with a friend
Embed Size (px)
of 29 /29
  • things






    PAGE 2

    Maybe you already have a digital archive – but if you have, congratulations!

    You’re in the minority. The DPP’s conversations with production teams tell us

    it’s more likely you’ve got a room full of dusty tapes, or a shelf full of external

    hard drives. Maybe with an Excel spreadsheet of what’s on them. Or maybe

    not. Either way, you are not alone.

    And if that’s what your programme storage is like now, you’re probably dreading

    what it’s going to be like when you move to digital delivery and storage.

    In practice the principles behind storing files are not all that different from

    those for keeping tapes. You’ll still have to decide what material to keep, for

    how long, and how you’re going to find it again. But as you begin to store your

    programme masters digitally, you’ll notice you are also starting to do things

    very differently. Some things will prove more difficult and complicated; others

    will feel much better.

    This introductory guide is designed to assist you. Without any jargon.

    It will help you think about how to store your master files and valuable rushes

    in a cost effective, well-organised and commercially useful way. Whether

    you’re designing your own in-house archive or paying someone else to store

    your material, the questions you’ll need to ask are the same. This guide will

    work through those questions and help you decide what sort of storage you

    really need.

    Next year the DPP will be publishing a more definitive, and more technical

    guide to storage in the broadcast industry. In the meantime we hope this

    simple introduction will get you underway.

  • Broadcasters may or may not compel you to keep your masters: but just like

    with tape masters, it’ll almost certainly be in your own company’s interests

    to maintain a copy anyway.

    The primary reason for this is that you can’t rely on the broadcaster to

    preserve your content. Many broadcasters delete their copy of the file after

    their rights window expires, and, just as with tapes, there’s nothing to oblige

    them to provide a copy of your programme back to you at a later date. The

    same goes for distributors. Don’t make any assumptions about how they’ll

    store your masters or how long they’ll keep them: look at your contract to

    see what they’ve committed to do. Ultimately if you own the Intellectual

    Property (IP), it’s for you to decide how best to keep your master material

    safe and accessible.

    Master is in itself a not entirely straightforward term. In a file-based world,

    the copy of your programme you send to the broadcaster in the AS-11

    DPP format is your delivery master: the one used by the broadcaster for

    transmission, and for their own storage. In the event any late changes are

    made to your programme, or if you have different programme versions as

    part of your deliverables, you may also have different delivery master versions.

    Why store my media?

    PAGE 3 VERSION 2.0

  • But you will also have an edit master: the version of your programme as you

    completed it on whatever editing platform you were using. If you wanted to

    re-edit the programme at a later date, you would probably want to use this

    master, together with your original edit information (your EDL).

    So you will want to consider what to do with delivery masters, delivery

    versions, and edit masters – as well as rushes and other programme elements.

    You are likely to need to store each of these types of material in different

    formats. You might keep your delivery master as an AS-11 DPP file; an edit

    master in its edit format; and rushes and programme elements in the medium

    in which they were originally created. Seek expert advice if you’re unsure.



    PAGE 4 VERSION 2.0

  • It’s very common for people to refer to any storage arrangement as an archive,

    but strictly speaking it almost certainly wont be. Archives – tape or file – are

    actually very rare. The word archive means not just a store but a rigorous set

    of policies and preservation standards that are formally, and internationally,

    defined. All archives incorporate systems that manage the storage so you can

    administer, search and find your material. You may be able to build or buy into

    such a service – although this guide will help you consider whether an archive

    in this formal sense is what you really need.

    But if what you want to do is simply store your programme or other files in

    a place that’s secure and where you’ll be able to get them out again in many

    years time, then what you are after is probably best referred to as a library.

    The table below describes storage in three different categories, defined not

    by the technology but by the way it is used. These categories have recently

    been adopted by the BBC, ITV and Channel 4.

    What is a store?

    PAGE 5 VERSION 2.0

  • Material In the WiP material is being actively used to make a show or series. Once the edit is complete and the programme successfully delivered and signed off by the broadcaster, this material is likely to be moved out of the WiP.

    Speed, Resilience and Quality

    The performance of WiP storage may vary according to the task. Is it serving multiple, simultaneous edits or just a single edit? How close to transmission deadline is the show being cut? Generally speaking however WiP storage is high performance, and expensive.

    Access will be limited to only those involved in producing and post-producing the programmes.

    Frequency Of Use:

    almost constantly while in post-production.

    Business Requirements:

    will change frequently, and vary by production; different productions may need different kinds of WiP storage.

    Volumes: will vary for each show. They can be enormous while the programme is in post-production – but then suddenly reduced when that programme finishes its edit.

    Costs: are allowed for within the production budget, and sit in the post-production lines of that budget.

    Work in Progress Store (WiP):A WiP holds and manages media which enables

    productions to edit and post-produce programmes.


    PAGE 6 VERSION 2.0

  • Material in a Library material is held close to production teams so it can be exploited commercially or creatively in the future. It will be relatively quick to access compared with an Archive, but it doesn’t necessarily have to be instantly to hand.

    Access will be determined by the production company’s own policies. It is likely that access will be limited to named individuals for reasons of security.

    Frequency Of Use:

    The Library is likely to be accessed frequently for quick re-versioning or the re-use of memorable sequences or shots. However usage won’t be as constant and time-critical as the WiP.

    Business Requirements:

    will change over time, based on production workflows and working practices; but it will nonetheless be possible to have some standard company policies and processes.

    Volumes: may vary dramatically over time in response to commissions, international sales, editorial priorities and the technical quality of the material. In theory the amount of storage required could go up or down. But in practice it only ever seems to go up – so you might as well accept storage volumes will gradually grow!

    Costs: will be met by the production company and weighed up against the commercial and creative value of keeping the material in the Production Library. These costs may be charged back to a progamme or series, or may become part of the company overhead.

    Production Library:A Production Library holds any digital assets (rushes, edit projects,

    ‘wavs’, stills, audio, key production paperwork, masters etc.) which are by-products of the production process.


    PAGE 7 VERSION 2.0

  • Material An Archive should hold finished material and any other material (rushes, edit masters, stills, audio etc.) which the company decides to retain, or it is legally required to retain. A company could decide to keep all finished programmes and rushes, but it is more likely they will decide to retain only those that have potential re-use value.

    Access may be granted, as required, to production teams or potential buyers. Depending on the design of the Archive, it may have a self-serve capability which enables users to view, select and download. The administration of the Archive itself however will be restricted to a small number of authorised individuals.

    Frequency Of Use:

    Occasional. This is a long term preservation store rather than storage designed for quick and frequent access.

    Business Requirements:

    Requirements of Archive storage will change relatively slowly. The greatest area of innovation and change will be in functions which enable people to search and access the material.

    Volumes: will increase relatively predictably over time.

    Costs: are likely to be borne centrally by the company or group.

    Archive:An Archive is where a company or group keeps its valued material

    and makes sure it is safe for the future. It is also sometimes referred to as the ‘deep archive’ or ‘preservation archive.’


    PAGE 8 VERSION 2.0

  • At the beginning of the production process, you will need to consider the type

    of programme you’re making, and the kind of life you want it to have. Will it

    be sold internationally, or just be shown in the UK? Are the rushes likely to

    have re-use value within other programming? Are you likely to re-version or

    update the show at a later date?

    Thinking about re-use may encourage you to shoot differently, with a view to

    the longer term value of the rushes. It may lead you to log high-value material

    carefully and to ensure that this information is created in such a way that it

    can be stored with the material in the longer term.

    And if you think you’re now getting into a degree of complexity, you’d be right!

    If you are creating a log, then you are creating metadata (which essentially

    means data about data – your programme or rushes material themselves

    being data). As soon as you start creating metadata you need to know it will

    remain accessible and readable when you share or keep it. Make sure you

    get some expert advice.

    What should I keep?

    PAGE 9 VERSION 2.0

  • Above all, be honest with yourself. How often have you actually re-used

    footage? When have you ever really had time to hang around on a shoot to

    capture extra GVs? If you don’t do these things now, you might want a policy

    of saving no rushes at all! It would certainly avoid a whole lot of hassle and

    cost. If you do want to store rushes however, they will need to be carefully

    logged and catalogued if you want to stand a chance of finding them again.

    The key point here is if you’re going to do it, you’ll need to do it properly.


    PAGE 10 VERSION 2.0

  • The danger in asking an expert what kind of technology you should choose for

    the storage of your file-based material is that it’s rather like asking a financial

    advisor how you should keep your money: they might actually tell you. Then

    you’ll be sorry.

    So here’s as simple an answer as we can manage:

    Portable hard drivesBrace yourself for an inconvenient truth: you shouldn’t even think about

    doing this. Pity, because it’s very probably the way you’re doing it right

    now. The problem is that hard-drives fail – especially if they are not

    in constant use. A shelf full of neatly labelled hard drives looks very

    organised. But it’s a disaster waiting to happen; and lots of broadcasters

    and indies have already found they didn’t have to wait very long.

    Storage serverThis is the non-portable kind of hard drive. Sometimes it’s referred to

    as spinning disk. It actually consists of a number of hard drives brought

    together in a server. It’s industrial strength and although it isn’t infallible,

    Where should I keep it?



    PAGE 11 VERSION 2.0

  • it does have built-in safeguards against going wrong. The key thing

    will be to set this up as a RAID, so that failure of any individual disk or

    other component won’t corrupt or destroy the contents of the server.

    The storage server can be accessed from a desktop machine that sits

    on the same computer network. Files can be copied backwards and

    forwards providing the access to store and retrieve media. It may be

    possible to review clips and programmes directly, depending on the

    number of people doing this at the same time and the connection

    speeds within your office. This is a perfectly good form of file-storage,

    but gets expensive if you have a lot of it.

    Data TapeConfusingly this is sometimes referred to as tape! It’s actually very

    different since it’s designed to keep lots of computer files rather than

    a single stream of video. This format is tailor-made for long-term

    storage and can provide a more stable way of keeping content than

    spinning disk. This is most likely to be an LTO or other proprietary

    format. It’s relatively cheap and very reliable – as long as you take

    care to control environmental conditions like temperature, humidity

    and dust.

    The only downside is that you need a way of reading the data tapes.

    Larger stores of data tapes tend to be contained inside a cabinet with

    barcodes to identify them, and a robotic arm to collect and insert




    PAGE 12 VERSION 2.0

  • them into data tape machines. Usually the material is then moved

    onto a storage server. So commonly a storage server is used in

    conjunction with a data tape set-up, with material being moved off the

    expensive server onto the cheaper data tape store once it is needed

    less often. It will be necessary to migrate these data tapes to a newer

    format periodically, as data stored on older data tapes can’t always

    be restored by newer systems.

    So although a data tape in itself may be cheap, the system you need

    to put around it to make it usable is actually pretty costly to create

    and maintain.

    You can keep and read data tapes without a robotic system. Desktop

    tape drives are available and can be connected to a computer to allow

    data tapes to be read and written. It will be necessary to copy files to

    a local desktop before viewing. In this case, it may be easier to keep

    a lower quality copy, or a proxy, of media which is needed frequently

    somewhere more accessible such as a local server. These proxies can

    be removed when no longer needed, or recreated if a disk drive fails.

    In addition to data tapes, there are other removable media storage

    options, such as optical disks, now available.



    PAGE 13 VERSION 2.0

  • LibraryShelf


    DigibetaHard Disk


    Storage asa service

    RAID StorageSystem

    Storage Server


    Library Cloud


    Local VTR

    Edit Suite

    Local Desktop

    Folders Spreadsheet Web Browser

    Cataloguing System

    Storage Management System





    Web Browser



    Edit Suite

    VHS dub

    Local Desktop

    Web-basedmedia player

    Library Shelf Library Shelf

    How will you store

    your digital media?









    What do you

    store it on?

    Where do you

    store it?

    How do you

    retrieve media?

    How do you catalogue

    and search for things?

    How do you view clips and


    PAGE 14

  • We’ve already warned against the most common pitfalls: failing hard drives;

    corrupted disks within servers; and data tapes rendered obsolete by the

    launch of a new versions.

    Beyond these big three, if you had to boil considerations about file storage

    down to one single word it would probably be security. How can you be

    sure your precious material is safe? How do you know it can’t be deleted by

    accident? How can you protect it from fire or theft? Who can you trust with

    it? And how do you know you’ll still be able to get at it in ten, twenty or fifty

    years time?

    This is a huge subject. But here are three key things to consider:

    Two + OneThere is a simple rule of thumb for keeping file-based material safe, and

    it is known as two plus one. This means you have two copies of your file

    kept in separate places (ideally in different buildings far away from each

    other), and a further off-line copy. By off-line we mean that it can’t be

    deleted by the system that can delete the first two.

    How will I know it’s safe?


    PAGE 15 VERSION 2.0

  • In many ways it’s that plus one copy that you should think about

    most, since it is your ultimate safety copy. So make sure it is in a

    place with good physical security, fire systems and so on. Of course

    if that third place is the cloud, then the building in question may be a

    purpose-built, high spec, data centre in another country.

    To the ninesIf you are buying your file-storage as a service, make sure you look

    at the contractual small print. What is the company promising you

    about how accessible your material will be? Often this is expressed

    as percentage availability, e.g. 99.99% (which is equivalent to less

    than one hour across a whole year when your footage may not be

    accessible to you). A typical enterprise agreement with a cloud

    storage provider, for example, could be 99.999%, or five nines – or

    just a few minutes a year when you can’t get your material. This level

    of availability should be considered in conjunction with the number

    of separate stored copies in A above.

    Forever is a long timeEver drawn up a will? It’s amazing how being forced to think about

    the unthinkable leads you to ask all sorts of questions you never

    thought to ask before. The same is true when thinking about precious

    programme material that you want to maintain over a period of

    decades rather than months or years.





    PAGE 16 VERSION 2.0

  • If you have material you want to keep for as long as you can imagine,

    then you can be sure that during its lifetime there will have been

    technical advances that will have rendered obsolete the format on

    which it was originally stored. So you will need to know that, over time,

    your material can be migrated to whatever is the new format. If you

    are creating your own file store, you should build this upgrade into

    your cost model. If you are buying a service, then take a good look at

    the small print around upgrading.




    PAGE 17 VERSION 2.0

  • This is really important. It’s no use storing something unless you can find

    it again when you need it. For a library or an archive you’ll need four key


    1 Searchwhich enables you to look for content – or elements of that content;

    2 Catalogue which organises your metadata, or information, about your content;

    3 Physical management which looks after the storage and moves the files;

    4 Review which enables you to browse, view and even publish your content.

    These elements could all come together in a Media Asset Management

    (MAM) system, or as a number of different tools all linked together. In either

    case there are some golden rules to consider:

    How will I find it?

    PAGE 18 VERSION 2.0

  • 1 Give each file a Unique Identifier. This is a number which is distinct from all others in the archive or library.

    2 Decide how you’ll name each file. You’ll need a logical system (File Naming Convention) to make this easy and consistent.

    3 Make sure your unique identifiers and file names enable you to distinguish between, and keep track of, different versions of a file.

    4 Decide what information (metadata) you want to keep about a file.

    5 Decide whether to keep that metadata with the file itself, or in a separate database, or both. If both, decide which you’ll regard as the master set of information.

    6 Work out how best to catalogue (organise) the metadata so it’s most useful.

    7 Figure out how you will search for a file and view it. This will require either a media asset management system (MAM) or separate tools linked together.

    All this is just as tricky as it might appear, and may be one reason why you

    ask a third party to store your material. If you create a library or archive within

    your own company you’ll need to hire someone who understands all this!

    The Magnificent Seven:Tips for organising a digital store


    PAGE 19 VERSION 2.0

  • There are two types of access: the first is direct access, which enables people

    to directly control the material itself. This means they can move it, edit it and,

    most importantly, copy or delete it. The second type of access enables people

    to see and use a copy of the material, commonly known as a proxy, which will

    not change the master file, unless it is later conformed. This is rather like the

    public having access to copies of BBC programmes through iPlayer – but not

    having access to the original material.

    So think carefully first about who is going to administer the material in the

    library or archive. Then think about who you want to be able to make use of it.

    Is it just for the benefit of the production company or group? Or do you want

    to offer a shop window for potential buyers?

    If you have answers to all these questions, congratulations: you have just

    written what archive experts call an access rights policy.

    How can I stop the wrong people from getting in?

    PAGE 20 VERSION 2.0

  • We all know how painful clearing the hard drive on your computer can be if

    it’s full to bursting and you need to free some space. What should I delete and

    what should I keep? There’s real danger of an even more painful and expensive

    scenario with a digital archive if you don’t have a pre-agreed set of criteria

    which help you not to become overwhelmed by the sheer volume of stuff.

    To avoid such a nightmare, you’ll need to consider a couple of things which

    at first glance sound like the same thing: what to retain, and what to delete.

    Retention is about what you actively plan to build up in your store. As

    we mentioned in Section 3, deciding what to keep is important. Based

    on your businesses’ needs, you can devise a simple retention policy for

    different types of material. Will you keep a copy of every programme in

    which you have the rights for as long as those rights exist? If so, don’t

    forget to also consider the underlying rights: for example, you might

    have the rights to the programme, but do you still have rights to re-show

    the contributions within it? Also bear in mind any obligations you have

    through your contracts with broadcasters or others.

    How long should I keep it?


    PAGE 21 VERSION 2.0

  • Deletion is about what you actively plan to get rid of – to save money on

    storage, and keep your store more manageable – so it’s also important

    to have a deletion policy. Perhaps you’ll delete material after a certain

    period if it hasn’t been reused? Perhaps you’ll delete material to which

    you haven’t got rights? Maybe you’ll delete old rushes material twelve

    months after programme transmission?

    Both retention and deletion policies will be influenced by how much you’re

    willing to invest in storing your material.

    Programme files are big.

    A one hour programme in the DPP format is approximately 50 GB, so twenty one hour programmes take up 1 TB. But that’s the easy bit. An hour of HD rushes takes around 50 GB too. And an hour of rushes shot in 4k requires a cool 100 GB… at least.




    PAGE 22 VERSION 2.0

  • You don’t! Just as in the analogue world, you’ll need to work hard to avoid

    your stored content becoming technologically obsolete. Older ones among

    us may have Betacam or U-matic copies of our programmes in the attic. We

    can’t bring ourselves to throw them away; but we’ll never find a machine on

    which to play them.

    In a digital world you will over time need to migrate your digital media from

    one hardware or software generation to the next. The aim is to preserve the

    content and maintain the ability to retrieve, display and use it. Keeping up

    with new generations of data tape (e.g. LTO) so that your stored media does

    not become inaccessible (see Section 4 C ) is just one reason why you’ll need

    to have preservation and migration policies in place. Don’t forget to also make

    someone responsible for implementing them.

    Some types of material are likely to have more value to you than others, so

    your migration policy could be tiered (e.g. Gold for highly valued, or culturally

    or legally important material; Silver for items of middling value; Bronze for the

    less important pieces). You could prioritise your time and money on migrating

    How do I know it will

    always play?

    PAGE 23 VERSION 2.0

  • Gold valued content, you may even decide to delete the Bronze content rather

    than migrate it. Just as with other policies, your approach to migration will be

    determined by the content you’re storing and your own businesses’ needs.

    You will also need to keep an eye on the long-term readability of the data

    itself. Data Tapes can be written in a variety of formats most of which are not

    interchangeable. You’d be wise to take expert advice on this as well as on the

    choice of file format in which to store media and metadata. Openly accessible

    (rather than proprietary) formats are generally safer bets in the long term.


    PAGE 24 VERSION 2.0

  • Storing your file-based material will carry a cost. But there are costs in the

    tape model too. Although the cost of storing tapes doesn’t show up in a

    specific budget line; to get a sense of the true cost you only have to consider

    the square footage of the space they occupy, the cost of hiring a tape deck

    or edit suite to look at them, and the courier bill for shipping them around.

    In a file-based world the costs of storage are more transparent. You’ll either

    be paying someone for a service, or you’ll be paying for servers, software

    and connectivity. A well-managed file-store can unlock potential income,

    however – making international distribution and re-purposing of content a

    whole lot quicker and easier. And if you decide to create your own storage

    capability, the infrastructure you’ll require to store your programme masters

    will also benefit you in other regards – such as sending file-based material to

    customers or post-houses, accessing web-based viewing and editing services,

    and so on. So the investment may go further than you think.

    In 2013, Mediasmiths authored a report for the DPP entitled The Coming Storm

    ( http://dpp-assets.s3.amazonaws.com/wp-content/uploads/2013/09/

    The-Coming-Storm.pdf ). The report provides an in-depth assessment of

    how far the television industry has progressed to being able to make effective

    use of cloud-based services. The report also contained a comparison of the

    relative cost (in 2013) of storing 1 GB of material in various ways for one year:

    How much will it cost me?

    PAGE 25 VERSION 2.0


  • Cloud £0.67

    Portable Hard Drives (not recommended) £0.20 *

    Data Tapes (e.g. LTO 6) £0.05

    * This varied between £0.16 and £0.38 depending on the capacity of the

    drive, the interfaces and the brand. The figure of £0.20 is based on using

    two 2 TB drives.

    Of course, it is impossible for a simple guide like this to help you calculate

    the total cost of ownership of a storage system; or to calculate the long-term

    cost of a storage service. Your decision about which storage medium to go for

    will also depend not just on its relative affordability, but also on your existing

    technology and IT infrastructure; the value and quantity of your content; and

    other business drivers unique to your company or group. In the case of Cloud

    the costs will be highly dependent upon how often you’ll need to access your

    content: it’s a rule of thumb for cloud services that it’s cheap to put your

    material in, but expensive to take it out.


    PAGE 26 VERSION 2.0


    We hope this guide has helped you think through how best to store your

    digital media in a cost effective, well-organised and commercially useful

    way. Even if you’ve decided to outsource your storage, we hope we’ve helped

    you reflect on all the key questions you’ll need to ask your service provider.

    The DPP plan to publish a more comprehensive guide to digital storage

    in 2015, but in the meantime here’s a reminder of the key considerations

    covered in this concise guide:

    PAGE 27

  • Checklist

    1Have you decided what to store and written a

    retention policy?

    2Will you store your material in a library or an

    archive or both?

    3Will you create your own library or archive or outsource these to a post production company or other service provider?

    4 If you’re creating an in-house store, which storage medium(s) will you use?

    5 Have you considered how much storage you’ll require?

    6 Have you thought through the level of resilience, access speed and security you’ll need?

    7 Have you decided how to search, catalogue, and physically manage and review your content?

    8Have you written an access policy and figured out how to give the correct people access to review and edit?

    9 Have you written a deletion policy and worked out how best to implement it?

    10Is someone in your organisation now responsible for preservation and migration and are policies in place for this?

    PAGE 28

  • This DPP production was brought to you by Rachel Baldwin, with Mark

    Harrison. Vlad Cohen has yet again made everything look lovely. Huge thanks

    are due to ITV, Channel 4 and BBC colleagues whose work we have drawn

    upon, especially Tim Davis, Heather Powell, Steve Jupe, Andy Tennant, Kevin

    Burrows, Alan Whiston, Andy Corp, Andy King, Holly Flanagan, Dale Grayson

    and Steve Daly. Thanks also to Niall Duffy from Mediasmiths, whose report

    The Coming Storm is an important addition to this simple guide.

    Copyright © 2014-2015 Digital Production Partnership. All rights reserved.

    First published 2014.

    PAGE 29