September 24-25 2003 M4 meeting Delft 1

VUT Brno meeting data collection VUT Brno meeting data collection for M4for M4

Petr Jenderka, Stanislav Sumec, Igor Potůček, Petr Motlíček, Jan Černocký, Pavel Zemčík

Brno University of Technology, Czech Republic

September 24-25 2003 M4 meeting Delft 2

GoalsGoals

• Provide M4 and follow-ups with more data, complementary to ICSI and IDIAP

• Different speakers, different accent (heavily accented CZEnglish)

• Setup of a secondary M4 meeting room: low-cost, mobile, built from off-the-shelve products, easy to install and operate.

• Plans for meeting recordings• Plans for annotations.

September 24-25 2003 M4 meeting Delft 3

The roomsThe rooms

• IDIAP room very valuable but expensive, difficult to assemble/setup and impossible to bring outside IDIAP

Our meeting room:• recordings of up to 4 persons• 4 lapel mikes, no mike arrays• one camera with hyperbolic mirror allowing for almost

360 degree view. • audio/video recording on one notebook.

Need for cheaper, lighter, and “buyable” room

September 24-25 2003 M4 meeting Delft 4

SetupSetup

September 24-25 2003 M4 meeting Delft 5

4 people around a mirror

September 24-25 2003 M4 meeting Delft 6

September 24-25 2003 M4 meeting Delft 7

Original wrapped image

September 24-25 2003 M4 meeting Delft 8

Hardware

• 4 lapel microphones Sennheiser MKE 2 P-C (same as in IDIAP)

• 2 mixers Behringer EURORACK UB1204-PRO providing gain control and phantom power supply (actually 90% of mixers are not used, but they’re cheaper than special mike pre-amps with phantom feeders…)

• two Hi-Fi PC-MCIA sound-cards VX-pocket (we know them from the SpeeCon project)

September 24-25 2003 M4 meeting Delft 9

Hardware (2)

• digital camera SONY CLIP 345

• hyperbolic mirror by NEOVISION (Prague)

• FireWire interface

• The camera provides 2 additional (global) audio channels.

• The cost: 4.2 kEUR w/o the notebook

September 24-25 2003 M4 meeting Delft 10

Recording

• a home-made (S. Sumec) tool, based on Direct-show libraries, under Windows.

• post-processing tool (I. Potúček) for unwrapping video to panoramic format of view.

• All audio and video streams are stored in real time: two audio wav files with four audio channels, one DV-compressed video avi file with video and stereo audio stream.

• Plan for M4: >30 hours of meetings

September 24-25 2003 M4 meeting Delft 11

Unwrapped image

September 24-25 2003 M4 meeting Delft 12

Annotations – major meeting events

• annotations using the „annotation editor“ by Staňa Sumec (audio, video, hotkeys, XML).

• This annotation will be done internally by pre- and post-grads at UPGM.

• Plan: >20 hours

September 24-25 2003 M4 meeting Delft 13

Annotations - speech

• ETCA Transcriber with ICSI MultiChannel extension.

• An annotator hired for speech annotations, just beginning to be operational.

• Plan: >10 hours of speech annotations.

• Both annotations: quality check protocols? (quite an issue in our DB projects SpeechDat-E and SpeeCon!)

September 24-25 2003 M4 meeting Delft 14

Current work and future• Recording, annotations, integration to the Media

Server mmm.idiap.ch• Still quite some issues to solve (position of the

mirror, lighting in the room – what can be done on the resulting video?)

• Extensions – PowerPoint and Acrobat Reader slide/change timing grabber, T0 must be synced with the recording software

• … and research with the data (phoneme recognition, mouth area detection, speaker activity detection) – where does the simplified setup degrade performances?

September 24-25 2003 M4 meeting Delft 15

Questions, comments ?

THE DEMOPlease have a look during the

break :-)