YOUR BROADCAST & IT CONVERGENCE PARTNER
Digital MediaDigital MediaDigital MediaDigital Media
ArchiveArchiveArchiveArchive
YOUR BROADCAST & IT CONVERGENCE PARTNER
Digital Media Archive Overview
2
ApplicationsApplicationsApplicationsApplicationsAutomationAutomationAutomationAutomation
Media Asset ManagementMedia Asset ManagementMedia Asset ManagementMedia Asset ManagementNewsroom Newsroom Newsroom Newsroom
Media NetworkMedia NetworkMedia NetworkMedia Network
Broadcast Devices : Broadcast Devices : Broadcast Devices : Broadcast Devices :
Transmission Transmission Transmission Transmission ServersServersServersServersNewsRoomNewsRoomNewsRoomNewsRoom SystemsSystemsSystemsSystems
NLE editing NLE editing NLE editing NLE editing
Storage Area NetworkStorage Area NetworkStorage Area NetworkStorage Area Network
Archive Archive Archive Archive LibraryLibraryLibraryLibraryTape Library
ExternalisationExternalisationExternalisationExternalisationNearline StorageNearline StorageNearline StorageNearline Storage
Archive Archive Archive Archive GUIGUIGUIGUI
Data MoverIntegratedIntegratedIntegratedIntegrated
TranscoderTranscoderTranscoderTranscoder
Ethernet NetworkEthernet NetworkEthernet NetworkEthernet Network
Control Network
API Archive API Archive API Archive API Archive
Archive Archive Archive Archive Manager Manager Manager Manager
Data Data Data Data MoverMoverMoverMover
Option Option Option Option TranscoderTranscoderTranscoderTranscoderAnalyserAnalyserAnalyserAnalyser
New Media New Media New Media New Media IPTV, VOD, 3GPIPTV, VOD, 3GPIPTV, VOD, 3GPIPTV, VOD, 3GP
BrowserBrowserBrowserBrowser
Tape Library
DisasterDisasterDisasterDisaster recoveryrecoveryrecoveryrecovery
YOUR BROADCAST & IT CONVERGENCE PARTNER
We would like to present you
through
concrete existing projects an
overview
of
some topics of archive systems
YOUR BROADCAST & IT CONVERGENCE PARTNER
Some Case Stories in order to speak about :
• Different Files Format – Why ? What to choose ?
• Different Storage – Why ? – What to choose ?
• Different Library – Why ? What to choose ?
• Different Drives – Why ? – What to choose ?
• Media HSM – IT HSM ?
• Different Applications ?
• Transcoding – Analyse – Monitoring – Browse : In-Out ?
• Disaster Recovery – Multiple Site – Multiple Robot
• Workflow
YOUR BROADCAST & IT CONVERGENCE PARTNER
Central Play out system of RTL Group company, migration from a classic tape based play-out system to a fully digital file based automated system and very large centralized Archive System.
21 channels on air
Historical and Playout Archive System
YOUR BROADCAST & IT CONVERGENCE PARTNER
• achieved its migration from a classic tape based play-out
system to a fully “ tapeless ” digital file based automated
system with centralized media archiving
• Linked with National Archive project and used as disaste r
recovery site
• Remote Acces and link from Brussels (RTL-TVI ) to the
Archiving System and
• Already 200.000 hours of Video
Historical and Playout Archive System
YOUR BROADCAST & IT CONVERGENCE PARTNER
In the thematic Channels control room, 1 operator drives 10 channels of transmission with an automation system
Historical and Playout Archive System
YOUR BROADCAST & IT CONVERGENCE PARTNER
The Storagetek PowderHorn Library can store up to 1.000.000 hours !
Historical and Playout Archive System
YOUR BROADCAST & IT CONVERGENCE PARTNER
Up to 450 exchanges of cartridges per hour, the system isfocused on access time, throughput and reliability
Historical and Playout Archive System
YOUR BROADCAST & IT CONVERGENCE PARTNER
Our methodology ...
• Customized proposalin close relation with customer requests
• Confidentiality
• Respect of Budget & Time Frame
• Testing Design & Performances
• Follow up & Training
• Insurance of investments in the long run
• Specific procedures in order to provide to the customer the foreseen functionalities
YOUR BROADCAST & IT CONVERGENCE PARTNER
Full Total File Base Operation
Full Complete Tapeless Integration Production - Postproduction – News – Archive – Play Out :
- Ingest on central media storage and local work cent er
- Integration with NRCS
- Low res search and edit
- Hi res conforming/editing
- Playout of Rundown
- Very large Archive System integrating also the IT Storag e
YOUR BROADCAST & IT CONVERGENCE PARTNER
6 Avid Unity ISIS
GMII
Airspeed
Omneon
Arvid (lo res)
45 AvidCraft Editing
Option
Ardendo Browse and edit
SDI
Omneon
NRCSAvid iNews
SGT automation
or iNews Control Air
variant
SDI
Ardendo dual file ingest
MediaFile
Avid DMSTelestream FlipFactory
WoConline
Global Media Archive
HistoricalArchive
IT Archive Storage
IBM GPFS2 PETA BYTE DISK
YOUR BROADCAST & IT CONVERGENCE PARTNER
Global Multipurpose Archive System
The Archiving System
is used for different purposes :
- Historical Media Assets
- Global Media Assets : News
PlayOut
Production -
Postproduction
- IT Storage
YOUR BROADCAST & IT CONVERGENCE PARTNER
Global Multipurpose Archive System
The Archiving System
is driven by different applications:
- DIVArchive from Front Porch Digital
for Media
- Tivoli from IBM
for IT
YOUR BROADCAST & IT CONVERGENCE PARTNER
Global Multipurpose Archive System
The Archive Robot
is virtualy divided in different partitions:
- News – Play Out
- Production - Postproduction
- Historical Media Assets
- IT
YOUR BROADCAST & IT CONVERGENCE PARTNER
Global Multipurpose Archive System
The Archive Robot
is handling different type of drives
- LTO3 : Historical Media Assets
- LTO 4 : IT Storage - Production
- Titanium : Play Out – News
YOUR BROADCAST & IT CONVERGENCE PARTNER
Different Files FormatWhy ? What to choose ?
In that project what is used :
- Historical Assets : MPEG2 – Long GOP 15 Mb/s
- MPEG2 MXF OP1a @ 50 Mb/s
- MPEG2 MXF OP ATOM - For Editing not in Archive
- MPEG1 General Browse
- WM9 Browse Historical Assets
- 3G - Mobile TV
- Tomorrow HD …
YOUR BROADCAST & IT CONVERGENCE PARTNER
Different Files Format
RTL uses :
- Historical Assets : MPEG2 – Long GOP 10 & 15 Mb/s
- MPEG2 MXF OP1a @ 50 Mb/s
- MPEG1 for browse
- Tomorrow HD ….
So : 1 application = 1 format….or more
Several applications = several format
Digital Archive is Format Agnostic !
Giving thus Protection of the future and of investment
YOUR BROADCAST & IT CONVERGENCE PARTNER
Storage Tools
20
Media RobotMedia RobotMedia RobotMedia Robot
Tape Library
Hard DriveHard DriveHard DriveHard Drive
Tape Library
DisasterDisasterDisasterDisaster RecoveryRecoveryRecoveryRecovery –––– Media RobotMedia RobotMedia RobotMedia Robot
Off LineOff LineOff LineOff Line
On Line On Line On Line On Line ServersServersServersServers
YOUR BROADCAST & IT CONVERGENCE PARTNER
Different StorageWhy ? What to choose ?
21
Media from the archive library can be ejected
and stored on shelves (or offsite) while still
tracked by the Media HSM
Large and expandable robotic tape or optical
archive media device controlled by the Media
HSM
Less expensive spinning disk controlled by
the CSM. Requires a “copy” operation to
ONLINE disk for content access
Spinning disk connected directly to broadcast
devices (video servers, editing platforms,
newsroom systems, etc)
Description Cost and AccessStorage
Tier
Storage Cost: Very Cheap
Access Speed: Slow
OFFLINE
Storage Cost: $$
Access Speed: Fast
ARCHIVE
Storage Cost: $$$
Access Speed: Very Fast
NEARLINE
Storage Cost: $$$$$
Access Speed: Immediate
ONLINE
Online
Storage Management
YOUR BROADCAST & IT CONVERGENCE PARTNER
YOUR BROADCAST & IT CONVERGENCE PARTNER
- Hard Drives Advantages - Disadvantages :
Faster
Capacity is growing
Price is decreasing but still higher and needs maintenance
Reliability weaker
- Tape Robot Advantages - Disadvantages :
Best reliability
Price still much lower
Much Larger Capacity
Capacity is growing ( but little less than Hard Drive )
Price is decreasing ( but less than Hard Drive )
Different StorageWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
VRT uses :
- Hard Drives : - Standard Raid Storage SUN 6140 - 20 TB
( Low res WM9 historical assets – on line )
- Extremely large and Redundant Storage
IBM GPFS : 2 PetaByte !
All Media Assets type near on line
- Digital Robot : - Very large Robot SL 8500 Storagetek
after migration from L700 ( all assets )
- ADIC : first IT Robot
Different StorageWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
Internal Users
Filing - 6 x 1 inch
3 xLTO 3Drive
Diva ActorManager 72GB
Content Checking
&Sub clipping
Low Res Server 20 TB WM9
PATRIMONIALARCHIVING PROJECT
Preview ingest station
Restore
Filing Server
Append
Archivists
TelestreamFlipFactory
(WM9)
SwitchSwitch
Restore(Preview+)
DBOS Loader & Media storageinterface
DBOS DeviceController
DBOS Server
VRT Supernet
12xLTO4
Drive for IT
Externalisation
Image Processing
YOUR BROADCAST & IT CONVERGENCE PARTNER
6 Avid Unity ISIS
GMII
Airspeed
Omneon
Arvid (lo res)
45 AvidCraft Editing
Option
Ardendo Browse and edit
SDI
Omneon
NRCSAvid iNews
SGT automation
or iNews Control Air
variant
SDI
Ardendo dual file ingest
MediaFile
Avid DMSTelestream FlipFactory
WoConline
Global Media Archive
HistoricalArchive
IT Archive Storage
IBM GPFS2 PETA BYTE
YOUR BROADCAST & IT CONVERGENCE PARTNER
RTL uses :
- Hard Drives : - Standard Raid Storage SUN - 40 TB
- Digital Robot : - 2 Very large Robot, located on 2 sites
- PowderHorn Storagetek
- SL 8500 Storagetek ( CNA )
Pay Tv Channel uses :
- Hard Drives : - Standard Raid Storage
- Digital Robot : - 2 Robot, located on 2 separate sites
- Sony Petasite
- SL 700 Storagetek
Different StorageWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER27
Hard Disk - NearLine Storage
Type of Disk Related to access time and reliability :
SATA or SCSI
Type of Raid technology : Raid 1– Raid 3 - Raid 5 – Raid 6
And from simple storage to quite complex :
- IBM GPFS
- Exanet
- Isilon
- MediaGrid
YOUR BROADCAST & IT CONVERGENCE PARTNER28
Stockage NearLine ( Disques ) - Exanet
Comparatif Storagetek
YOUR BROADCAST & IT CONVERGENCE PARTNER
So No single answer, choice is related to the requested
fuctionalities, reliability, and……price.
For Archive Robot choice one important topic is :
Ability to grow and to integrate different type
of drives and as far as possible ability to
integrate future drive with future technology
Different StorageWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
Which Media / Support to choose :
DVD – Holographic – Data Tape ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
VRT uses :
- LTO 3 Historical Assets ( MPE2 Long GOP )
Browse ( WM9 )
This migrated already from LTO2
- LTO4 General Media ( Play out – Production ….)
MPEG2 MXF - General Browse ( MPEG1 )
- LTO4 IT This migrated already from LTO3
Different DrivesWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
RTL uses :
- 9840 Historical – Play Out
Short Assets only -( MPE2 Long GOP )
Why = very short access time ( but small capacity)
- 9940 Historical – Play Out
All other Assets -( MPE2 Long GOP – MPEG1 )
Why = large capacity –good access time
- Platinum : All items for new customer
CNA National Archive Institution
Different DrivesWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
Mixed Drives
• For different type like fast access and small capacity and
lower access but high capacity
• Typically SUN/StorageTek T9840C and T9940B or IBM LTO3,
LTO4, Sun/StorageTek T10000
• For a migration due to new technology:
– LTO2 to LTO3, LTO4
YOUR BROADCAST & IT CONVERGENCE PARTNER
So for Archive Robot actually Tape Data, are still
the best fit for the Broadcast applications
But new technologies are coming, like Holographic
and performance are increasing.
Different DrivesWhy ? What to choose ?
YOUR BROADCAST & IT CONVERGENCE PARTNER
Media HSM versus IT HSM
• Support for the widest base of asset management, automation,
video server, storage and archive devices
• Dedicated applications Not an HSM solution, but rather a content
aware, intelligent storage management solution
• IT HSM are Archive Manager which can of course manage the
different components of the archive but without dedicated tool
for Media
YOUR BROADCAST & IT CONVERGENCE PARTNER
Storage Resources Same for Media or IT HSM
Servers• Robot Tape Libraries
– Quantum/Adic
– IBM
– Qualstar
– Sony
– SUN/StorageTek
– SpectraLogic
• Tape drives
– LTO-2
– LTO-3
– LTO-4
– 9840C (STK)
– 9940B (STK)
– T10000 (STK)
– S-AIT1 (SONY)
Storage Resources• DVD jukeboxes
– Asaca
(Matsushita drives)
• Disk
– Any SAN storage
– Any NAS storage
– Omneon Mediagrid
– Seachange BML
YOUR BROADCAST & IT CONVERGENCE PARTNER
Storage Resources
SERVERSwith IT HSM
Servers
Storage Resources• FTP Servers
YOUR BROADCAST & IT CONVERGENCE PARTNER
Storage Resources
SERVERS with Media HSM
Servers • Avid
• EVS
• Incite
• Leitch
• Omneon
• Pinnacle
• Quantel
• SeaChange
• SGI
• Sony
• Thomson GV
• Vela
• Any FTP servers
Storage Resources
YOUR BROADCAST & IT CONVERGENCE PARTNER
• Ardendo
• ……
Limited Existing Interface with Automation / MAM/ Traffic System
IT HSM Interfaced Applications
YOUR BROADCAST & IT CONVERGENCE PARTNER
• Anywhere
• Ardendo
• Artesia
• Ascential
• Aveco
• Blue Order
• CIS
• Dalet
• Harris/Arkemedia
• Harris/Encoda
• Harris/Invenio
• Harris/Louth
• Ibis
• Incite
• Konan
• Marquis
• Mediagenix
• Netia
• Omnibus
• PebbleBeach
• Pharos
• Probel
• Proconsultant
• Publitronic
• S4M
• SGT
• Sobey
• Tedial
• TGV/ContentShare
• TGV/AuroraNews
• TMD
Many : Automation / MAM/ Traffic System
Media HSM Interfaced Applications
YOUR BROADCAST & IT CONVERGENCE PARTNER
Broadcast EnvironmentDifferent Applications can be interfaced
Media ManagementMedia Management
PostPost--ProductionProduction
Ingest, Playout,Ingest, Playout,
Production ServersProduction Servers
AutomationAutomation
Mid and Long term Storage
Traffic SystemTraffic System
Media Archive ManageMedia Archive ManageMedia Archive ManageMedia Archive Manager r r r
YOUR BROADCAST & IT CONVERGENCE PARTNER
Media HSM Features
Standard Features
• Near line disk and tape storage • Off-line storage (Externalization)• Tape de-fragmentation• Priorities management on requests• Queue Manager Analysis (QMA)• Estimated Time Completed (ETC)• Multiple Instances Object • Co-location of objects • Object copy• Tape Groups & Disk Array grouping• Library partitioning • Flow control management • Louth VACP (Harris)• SNMP agent• Metasource• Automatic spanning of content • TRU and FTU Utilities• Application Programming Interface
(API)• Graphical User Interface (GUI)
Other Features
• Application Filtering• Avid Unity support• Drop Folder Monitor (DFM)• Database Clustering• Export / Import • Mixed drives• Multiple robots • Object Transfer Utility (OTU)• Partial Restore • Multiple Parallel Restore • Storage Plan Manager (SPM)• Transcode Plug-in• Quality Analysis
YOUR BROADCAST & IT CONVERGENCE PARTNER
ResourcesAssignment &Management
ContentLocalization
Storage/Archive Manager: The Functions
ContentDuplication
MigrationPolicies
Interface ToVideo
Equipments
FaultsIdentification& Recovery
ErrorsReporting
EventsLogging
ContentGrouping
Interface ToStorage
Technologies
DataMovement
ContentTranscoding
VideoPartial
Retrieval
PriorityManagement
LoadBalancing
Interface ToLegacy
Systems
Interface ToBroadcast
Applications
CacheManagement
TapeDe-Fragment
TapesSpanning
YOUR BROADCAST & IT CONVERGENCE PARTNER
Partial Restore (1/2)
• Optimized access time for movie excerpts
• Parsing and selection direct from tape (no disk cache
required)
• Time-code offsets can be specified for video clips:
– Format or server type
– Start time-code
– End time-code
– New Name for the restored clip
• Uses frame index when available
YOUR BROADCAST & IT CONVERGENCE PARTNER
Partial Restore (2/2)
– AVID/Pinnacle • MediaStream: MPEG-2 LGOP Program or System Stream)
– Final Cut Pro• QT Embedded
– Leitch• Nexio LXF: DV25, MPEG-2 I-Frame and LGOP
– Matrox based servers AVI : DV25/DV50, MPEG-2 50I and LGOP– Omneon Spectrum or Mediagrid:
• Quicktime: D10, MPEG-2 I-frame, DV25• MXF OP1a D10, DV25, DV50
– Quantel• Clipbox: D10• sQserver MXF OP1a IMX30, DV25, DV50
– Sony e-VTR MXF D10
– Thomson Grass Valley:
• Profile XP, PDR in GXF: DV25, DV50, MPEG2-50I, or MPEG-2 LGOP, MJPEG
• K2 in GXF: MPEG-2 I-Frame and MPEG-2 LGOP (SD & HD), DV25, DV50 and MJPEG in SD
• K2 in MXF OP1a: IMX/D10 (30 and 50), DV25 and DV50
– Generic MXF OP1a : D10 (IMX) and DV25, DV50– Standard MPEG2 TS format
YOUR BROADCAST & IT CONVERGENCE PARTNER
Manager Queue Processing
• Since DIVArchive V5.10.6
• Concept of “priority tier” : value which can be added to a request
priority
• Manager regularly collects the tape-id mounted in drives
• Requests requiring a tape already mounted will see their priority
increased of the “priority tier”
• Only 1st tape is taken into account when object is spanned
YOUR BROADCAST & IT CONVERGENCE PARTNER
Tape Partitionning
Other
…
Set
Set 1 Set 2
Channel A Channel B
…
news
movies
2 months
…
Low
Res
series
Group
YOUR BROADCAST & IT CONVERGENCE PARTNER
• Back up
• Disaster Recovery
• Multiple Robots
• Multiple Sites
Redundancy Possibilities
YOUR BROADCAST & IT CONVERGENCE PARTNER
Disaster Recovery–Multiple Sites
x2
Fiber Channel
Switch
Application
Nearline Tape Storage
Main Robot
x2DIVArchive
Control GUI
Library Control
DIVArchive Manager
Network
Switch
x2
DIVArchive Actor2
Transcoder
DIVArchive Actor1
Backup Manager
Nearline Disk Storage
DIVArchive API
Tape Library
x
Nearline Tape Storage
Backup Robot
Ingest and playout Servers
Video ServerVideo ServerVideo Server
Tape Library
x
x2x2
IP network
Main Site Backup Site
Backup Manager
Actor 3
DIVArchive Actor4
Transcoder
Network
Switch
YOUR BROADCAST & IT CONVERGENCE PARTNER
Tape Library Tape Library
Disaster RecoveryFull Production Mirroring Multiple Sites
YOUR BROADCAST & IT CONVERGENCE PARTNER
Transcoding or Analysis Plug-in
• Based on Transcoding or Analysis engines
• Automatic Transcoding initiated by Media HSM
• Rules can differ according to initial Media name (Group/Array) or object category
• Typical workflows: – Archive high-res material on disk array
– Transcode to MPEG4 or WM9 for browsing
– Make available low res on a browsing server
– Archive low-res material on tape
– Create another high res instance on tape
– Delete instance on disk
YOUR BROADCAST & IT CONVERGENCE PARTNER
• Networking and storage systems must be designed to support advanced file based workflows
• Bandwidth bottlenecks can be easy introduced and cripple the system – or worse!
• Storing your content at excessive bitrates (ie. uncompressed) can be costly and perhaps unnecessary
• Ensure the system can incrementally scale in bandwidth, storage and technology for new business/revenue opportunities ( future HD for ex )
System Bandwidth must
fit the requested Workflow
61 YOUR BROADCAST & IT CONVERGENCE PARTNER
YOUR BROADCAST & IT CONVERGENCE PARTNER
Our methodology ...
• Customized proposalin close relation with customer requests
• Confidentiality
• Respect of Budget & Time Frame
• Testing Design & Performances
• Follow up & Training
• Insurance of investments in the long run
• Specific procedures in order to provide to the customer the foreseen functionalities
YOUR BROADCAST & IT CONVERGENCE PARTNER
• Independent from manufacturers
• In order to cope with different suppliers we have set up a
• Quality and Co-responsability Agreement
• Based on architecture and functinalitiesagreed with customer
Our Methodology