Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
1
The FAME Project• Acronym: Facilitating Agent for Multicultural Exchange
• Partners:
Universität Karlsruhe, INPG Grenoble,UJF Grenoble, ITC-irst Trento,UPC Bacelona, SONY Europe, StuttgartATLAS Barcelona
• Project volume: 5.5 M Euro• Duration: 40 months, started October 2001
• More info: http://www.fame-project.org
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
2
The FAME ProjektFacilitating Agent for Multicultural Exchange
• Volume:• Duration: 40 months (since October 2001)
• Financial Volume: 5,5 Mio. €
• currently approx. 30 scientists
• Partners:Uni Karlsruhe , INPG Grenoble , UJF Grenoble,
UPC Barcelona , ITC-irst Trento ,
Stuttgart, Barcelona
www.fame-project.org
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
3
Project Goals• long term vision: facilitate communication between humans
• reduce the workload on the users of technical equipment• observe humans and their activities in an intelligent room
and serve as a context-aware information butler
• FAME project goal: provide and integrate core technologies (video and speech perception, augmented reality, translation, information retrieval) to show feasibility of the concept
• demonstrate system at fair• scenario 1 (lecture scenario):
one person is giving a talk or lecture or presentation• scenario 2 (meeting scenario):
several people are discussing / working on a common task
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
4
The FAME Showcasesscenario 1 (presentation) scenario 2 (meeting)
• use A/V equipment• intelligent cameraman• presentation tracking• summarisation + archiving• translation, crosslingual IR
• augmented reality• video-based activity tracking • topic spotting• information butler• service: planning of fair visit
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
5
The FAME Demonstrator(at Barcelona Fair „Forum of Cultures“ 2004)
FAME outside view reception by FAME-guy
meeting inside people mention topicsroom gives informationabout spotted topics
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
6
The FAME Demonstrator(at Barcelona Fair „Forum of Cultures“ 2004)
at the phicon wall gestures the projection table
multimodal input on tableoutput also on the wall interactive visit planning
borrow a camera forphotographs of the visit
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
7
The FAME Demonstrator(at Barcelona Fair „Forum of Cultures“ 2004)
Back from the visit in theFAME room: dowload ...
... and look at photos select, print, save photosusing phicon interaction
record testimony intelligent cameraman,presentation tracker
take home photos andinformation about FAME
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
8
• multimodal environment
• context-aware intelligent camera-manautomatically track people and their activities
• augmented reality environmentmove physical icons (phicons) on table/wall,and interact with projection on table/wall
• spontaneous speech recognition (with distant microphones)• translation and crosslingual information retrieval
in European-English, Catalan, and Spanish• dialog and context model
Important Components
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
9
Microphon-Array(Speaker Lokalization)Microphon-Array(Speaker Lokalization)
Smartboard asProjection WallSmartboard asProjection Wall
LivingroomgLivingroomg
IR-Remote ControlIR-Remote Control X-10X-10 IlluminationIllumination
Audio SignalsAudio Signals
Several BeamersSeveral Beamers 4 Cameras4 Cameras TV/VideoTV/Video
LoudspeakersLoudspeakers
MicrophonesMicrophones
Multimodal Environment at UKA
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
10
Augmented Reality Table
• project virtual reality on real table• move around physical icons (multiple users)• interact with projection
• select, move, rotate, resize, delete, change color• write on table, pass notes to others, point to items
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
11
Intelligent Camera Man
• follow speaker while talking and moving around
• detect interaction from audience
• zoom on area of intereste.g. when pointing somewhere or showing something
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
12
LectureSupporter
• track lecture or presentation
• operate FAME room equipmentby speech commands
• automatically switch slides during presentation
• automatically create transcript of lecture
• create summary, translate to other languages
• record and store all lectures in searchable database
• retrieve and browse through previously recorded lectures
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
13
100 Links100 Links
wichtige Wörter
wichtige Wörter
wichtige Wörter
TO THE RECOGNITION OF THECONTINUOUS SPEECH RECOGNITION IN NOISY AND PATTERN RECOGNITION NOT IN SECURING DUE RECOGNITION AND RESPECT
CLASS 32CLASS 14CLASS 57CLASS 6CLASS 70
top n
most frequent40k words
least frequent20k words
60k vocabularyHUB-4 corpus
trigramm- modelP(wn|wn-1,wn-2) 72 classesP(Cn|wn-1,wn-2)· P(wn|Cn)
add to classes
estimate
presentation-slides
important words
tf-idf
100 links±2 contexts
perplexity
scores
Adaptation Overview
20% fewererrors
Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003
14
Welcome in Barcelonain Summer 2004