Timecodes & Sync P. Vickers/J. Edwards/A. Watson 1
School of Informatics
CG087 Time-based Multimedia Assets
Time codes and Synchronisation
Synchronising video and audio with time codes such as SMPTE
and SMIL
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 2
CG087 Time-based Multimedia AssetsSchool of Informatics
Time codes
• A time code allows each frame of video to be numbered.
• Numbering frames allows precise identification when editing.
• It also facilitates synchronisation of audio with video
• An 80-bit word is stored on each frame to represent hours, minutes, seconds, frames and other synchronizing information. There is also room for ‘user bit’
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 3
CG087 Time-based Multimedia AssetsSchool of Informatics
Time code example
03:10:45:18
Hours
Minutes
Seconds
Frames
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 4
CG087 Time-based Multimedia AssetsSchool of Informatics
Starting time codes
• 2 ways of starting a time code– Record run– Free run
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 5
CG087 Time-based Multimedia AssetsSchool of Informatics
Record run
• Only stamps a time code when camera is recording.
• Time code set to zero at the start of the day’s operation and a continuous record is produced on each tape covering all takes
• Customary to record the tape number in place of the hour section on the time code– E.g. the first cassette of the day would start
01.00.00.00, and the second cassette would start 02.00.00.00.
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 6
CG087 Time-based Multimedia AssetsSchool of Informatics
Free run
• Time code is set to the actual time of day and when synchronized is set to run continuously.
• Whether the camera is recording or not, the internal clock will continue to operate.
• When the camera is recording, the actual time of day will be recorded on each frame.– Useful for editing day-long events (e.g. sport)
Action can be logged by time as it occurs and can be quickly found by reference to the time code on the recording
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 7
CG087 Time-based Multimedia AssetsSchool of Informatics
Free run: gaps
Shot 1(8 seconds)
Shot 2(5 seconds)
Shot 3(7 seconds)
200 frames 125 frames 175 frames
04:12:45:01
14:44:22:05
15:02:44:03
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 8
CG087 Time-based Multimedia AssetsSchool of Informatics
Free run: gaps
• A change in shot produces a gap in time code corresponding to the elapsed time between recordings
Shot 1(8 seconds)
Shot 2(5 seconds)
Shot 3(7 seconds)
200 frames 125 frames 175 frames
04:12:45:01
14:44:22:05
15:02:44:03
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 9
CG087 Time-based Multimedia AssetsSchool of Informatics
Time code problems
• Missing time code numbers can cause problems with edit controller when it rolls back from intended edit point and is unable to find time code number it expects there (i.e. the time code of the frame to cut on, minus the pre-roll time).
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 10
CG087 Time-based Multimedia AssetsSchool of Informatics
Synchronisation
• To fit a sound track to a video sequence you need to syncrhonise the audio with the video
• When a system is synchronized, all the video and audio decks, computer sequencers, hard disks and so forth will play back exactly together. This allows you to compose layers of sound, secure in the knowledge that once you get something in place, it will stay there. (see http://arts.ucsc.edu/ems/music/equipment/video/smpte/SMPTE.html)
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 11
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE
• This section shamelessly plagiarised from http://arts.ucsc.edu/ems/music/equipment/video/smpte/SMPTE.html
• The Society for Motion Picture and Television Engineers (SMPTE) has defined its own time code
• SMPTE format allows different devices to be synchronised in a master/slave arrangement– Master device must be a video deck– Other devices follow the master– Multiple video decks are ‘genlocked’ to a clock device
• Genlocking is a line-by-line (rather than frame-by-frame) synchronisation system
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 12
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE overview
• SMPTE code recorded on one audio track• Actually a modulated tone that is decoded
into data• 80-bit word for each frame to store
hh:mm:ss:ff + subcode information
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 13
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE gadgets
• SMPTE generator– Creates the time code
• SMPTE reader– Displays the time
• Regenerator– Reads the code and recreates it for recording on
another device
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 14
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE gadgets continued
• MTC generator– Converts SMPTE into MIDI time code which can
be sent over MIDI cables
• SPS converter– Turns SMPTE into song pointer and MIDI clock
data to control sequencers
• FSK– System for recording MIDI song pointer and MIDI
clock data directly on tape. Simpler than SMPTE but less flexible
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 15
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE gadgets continued
• Controller– Tape deck remote control– Uses SMPTE to monitor tape position and can cue
tape to any desired location– More accurate than a mechanical tape counter
• Synchroniser– Controls >=2 tape decks
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 16
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE frame rates
• 24 fps– Film
• 25 fps– European video (PAL, SECAM)
• 30 fps– American & Japanese B&W video
• 29.97 fps– American & Japanese colour video. Can lock 30fps
audio deck to this, but pitch will drop from A=440 to 335.6 Hz
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 17
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE numbering schemes
• Each frame contains hh:mm:ss:ff• SMPTE time code can only increment in
whole frame numbers• How to deal with odd division of frames in
US/Jap colour format (29.97 fps)?• At each new frame (now lasting a little more
than 1/30th of a sec.), the Time Code still increments by a whole 1/30th and drags behind real time
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 18
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE DF and NDF
• In Non Drop Frame mode we slow down the code generator to match the frame rate– This makes a 1-hour programme actually last
59:56.4!– 29.97/30 * 3600 = 3596.4s
• In Drop Frame mode we skip two frame numbers a the start of each minute except those ending in zero (00, 10, 20, 30, 40 ,50)– 54 mins that have 2 frame no.s added = 108– 3600s *29.97 fps = 107892 frame no.s + 108 =
108000 = 3600 * 30
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 19
CG087 Time-based Multimedia AssetsSchool of Informatics
SMPTE and MIDI
• MIDI time code is a specialised SMPTE code that can be transmitted over MIDI cables
• Most sequencers now support SMPTE control
• Look at DOP’s Options|Sync Settings
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 20
CG087 Time-based Multimedia AssetsSchool of Informatics
SMIL
• Synchronized Multimedia Integration Language.
• Pronounce it "smile" !• An XML abstraction for Synchronisation of
Multimedia• Originally proposed around 1998• At present v2.0• Can be used in IE6 and RealPlayer• GRINS and SOJA also example implementations• Synchronisation reflects position and time
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 21
CG087 Time-based Multimedia AssetsSchool of Informatics
A simple SMIL document …… looks familiar ???
<smil> <head> <meta name="copyright" content="Your Name" /> <layout> <!-- layout tags --> </layout> </head> <body> <!-- media and synchronization tags --> </body> </smil>
SIMILAR STRUCTURE TO HTML
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 22
CG087 Time-based Multimedia AssetsSchool of Informatics
Positioning - 101
• Not really very exciting (link region to data)
<smil>
<head>
<layout>
<root-layout width="300" height="200" background-color="white" />
<region id="vim_icon" left="75" top="50" width="32" height="32" /> </layout>
</head>
<body>
<img src=“myimage.gif" alt="The vim icon" region="vim_icon" /> </body> </smil>
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 23
CG087 Time-based Multimedia AssetsSchool of Informatics
Positioning - Relative
<smil> <head> <layout> <root-layout width="300" height="200" background-
color="white" /><region id="vim_icon" left="50%" top="40%" width="32"
height="32" /> </layout> </head> <body> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" /> </body></smil>
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 24
CG087 Time-based Multimedia AssetsSchool of Informatics
OTHER MM TAGS AVAILABLE
Media Tag G2 GRiNS Soja
GIF img OK OK OK
JPEG img OK OK OK
Microsoft Wav audio OK OK -
Sun Audio audio OK OK OK
Sun Audio Zipped audio - - OK
MP3 audio OK - -
Plain text text OK OK OK
Real text textstream OK - -
Real movie video OK - -
AVI video OK OK -
MPEG video OK OK -
MOV video OK - -
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 25
CG087 Time-based Multimedia AssetsSchool of Informatics
The meat of SMIL is timings!!
• This is really what sets it apart from DHTML• Allows the specification of
– Duration – Dur = “3s”– Sequence - <seq> </seq> – Synchronisation – id referencing
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 26
CG087 Time-based Multimedia AssetsSchool of Informatics
Duration
<smil> <head> <layout> <root-layout width="300" height="200" background-
color="white"/> <region id="vim_icon" left="75" top="50" width="32"
height="32"/> </layout> </head> <body> <img src="vim32x32.gif" alt="The vim icon"
region="vim_icon" dur="6s" /> </body> </smil>
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 27
CG087 Time-based Multimedia AssetsSchool of Informatics
Sequence
<smil> <head> <layout> <root-layout width="300" height="200" background-color="white" /> <region id="vim_icon" left="75" top="50" width="32" height="32" /> <region id="soja_icon" left="150" top="50" width="100" height="30" /> </layout> </head> <body> <!-- the sequence starts bellow with the "seq" tag --> <seq> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" dur="6s" /> <img src="madewithsoja.gif" alt="Made with SOJA" region="soja_icon"
dur="4s“ begin="1s" /> </seq> </body> </smil>
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 28
CG087 Time-based Multimedia AssetsSchool of Informatics
Sychronisation
<smil> <head> <layout> <root-layout width="300" height="200" background-color="white" /> <region id="vim_icon" left="75" top="50" width="32" height="32" /><region id="soja_icon" left="150" top="50" width="100" height="30" /></layout> </head> <body> <par> <img src="vim32x32.gif" alt="The vim icon" region="vim_icon" dur="6s" /> <img src="madewithsoja.gif" alt="Made with SOJA" region="soja_icon“
dur="6s" /> </par> </body> </smil>
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 29
CG087 Time-based Multimedia AssetsSchool of Informatics
Synchronisation
<smil> <!-- head (skipped) --> <body> <par> <img src="vim32x32.gif" region="vim_icon" id="vim" begin="4s" /> <img src="madewithsoja.gif" region="soja_icon" begin="id(vim)
(2s)" /></par> </body> </smil>
• synchronisation specified us id(previous sources)(time duration)
Timecodes & Sync P. Vickers/J. Edwards/A. Watson 30
CG087 Time-based Multimedia AssetsSchool of Informatics
SMIL Summary
• FLASH for the open source world !!!• SMIL 2.0 augments the above framework to
incorporate more media handling– More animation– More event handling– Multiple Windows
• Reference : All examples from– http://www.helio.org/products/smil/tutorial/chapter1/index.
html
• Reference: SMIL 2.0– http://www.w3.org/Talks/2001/06CSMIL/slide1-0.html