Smart Meeting Systems

Josh Reilly

Why are Smart Meeting Systems worth studying?

Objectives of a Smart Meeting System

Improves the productivity of a team by automating the:

– Capture of the meeting

– Processing of the meeting for valuable information

– Displaying of that information accurately and effectively to the end user through a client application

Organization of Smart Meeting System Processes

A smart meeting system can be decomposed into three sets of processes

Meeting Capture Meeting Recognition Semantic Processing

Organization of Smart Meeting System Processes

Meeting Capture

Gathering raw inputs from the meeting Video Capture Audio Capture Other Context

Video Capture

Video feeds from: Cameras for the attendees

Could use a single static camera Could use a single camera with pan, tilt, zoom (PTZ)

capabilities Recommend camera view of every contributor's face

Visual Aids Separate camera Digital feed from device

Microsoft Distributed Meetings Project Camera Placement

Microsoft Distributed Meetings ProjectVideo Capture

RingCam Array of 90º Cameras 360º Panoramic view

Audio Capture

Use an array of microphones Placed on the table Placed on the ceiling Worn on the person

Levels need to be controlled so that they are similar levels for each contributor

Microsoft Distributed Meetings Project Audio Capture

RingCam Has an array of

microphones on its base.

Other Context Capture

RFID to track attendees Attendees swipe their RFID cards when they enter

the meeting to add their ID to the list of people attending this meeting

Motion Detectors to track the locations of attendees within the room

Organization of Smart Meeting System Processes

Meeting Recognition

The processing of the raw capture before it is organized into something useful

Steps: Person Identification Attention Detection Activity Recognition Hot Spot Recognition Summarization

Person Identification

Person Identification

is associating sections of video, audio, and the visual aids that were captured from the meeting with the attendee(s) that they belong to

Face Recognition Face Tracking Speech Recognition SSL Beamforming

Person IdentificationFace Recognition

Facial Recognition Identify the person speaking from a list of attendees Eigenface Approach Challenges

Poor Quality Images Poor Room Lighting Continuously changing facial expressions Occlusion

Face RecognitionThe Eigenface Approach

All faces are assumed to be made up of different percentages of different eigenfaces

A set of eigenfaces is a set of very generalized pictures of faces that were generated so that each has a basic ingredient that can be used to make a face Eigenfaces from AT&T

Laboratories Cambridge

Person IdentificationSpeech Recognition

• Speech Recognition

• Match the voice of the person speaking to someone on the list of attendees

• Using Voice recognition in conjunction with face recognition allows for an accurate identification of the speaker

• Sound Source Localization (SSL)

• Used to determine which camera is pointed at the speaker

• Could be used to point PTZ camera

• Beamforming

Person IdentificationWriter Recognition

Writer Recognition When someone writes on the whiteboard, they may

not be in clear view of the cameras Writing recognition algorithms can be used to

identify who wrote what during a meeting

Attention Detection

Attention Detection Attempt to determine who is looking at whom during

a meeting. Provides information used for activity recognition

and hot spot recognition Done using:

Hidden Markov Models (HMM) Sound Source Localization (SSL) Known layout of room

Activity Recognition

Determine what is happening during the meeting Step 1:

Determine what each individual is doing at each point during the meeting

Person Identification, Attention Detection, SSL, Gesture Recognition

Step 2: Take that information to determine what activity the

entire group is engaging in at each point during the meeting

Hot Spot Recognition

Find the important parts of the meeting Using sound queues

Ex: Changes in pitch Using activity recognition

When people are nodding When their focus changes

Summarization

Takes all of the information that the smart meeting system has learned about the meeting and creates a quick overview of the events that took place during that meeting.

This information will be used in the semantic processing stage

Organization of Smart Meeting System Processes

Semantic Processing

Takes the information from the meeting recognition step and makes it usable by the end user.

Meeting Annotation Meeting Indexing Meeting Browsing

Meeting Annotation

Describe the raw data from the meeting from each viewpoint

Attempt to label all meeting segments Implicitly

Automatically Explicitly

By Hand

Meeting AnnotationImplicit

Automated Annotation Assumes that the meeting recognition processes

performed with relatively high efficiency Tags every person in the video Narrates what was happening during the meeting Has not been achieved

Meeting AnnotationExplicit

Annotation By Hand When the recognition processes fail to gather

sufficient correct information about the raw data Users will have to go through the meeting and tag

the people attending as well as indicate what events are happening all through the meeting

Meeting Indexing

Indexing is done at all levels of data from a raw audio feed to the annotations

The best form of indexing to use is the event-based indexing

An index is created every time an event occurs This is the best way for users to find a specific spot

in the meeting when performing a query

Meeting Browsing

The interface that the end user uses to retrieve information from the meetings

Functions: Can browse/search a list of all meetings for a

specific meeting Can browse/search the contents of the chosen

meeting Aided by tools like bookmarks, a meeting outline, and

queries (content, people, camera angles, visual aids, etc...)

Meeting BrowsingMicrosoft Distributed Meetings

Remote Attendee

Use the smart meeting system as the attendee's eyes and ears

Microsoft's PING project Uses a monitor and speaker to display the remote

attendee's voice and audio during the meeting However, the remote attendee is often ignored

Carnegie Mellon University’sMeeting System Architecture

Lacks•Activity Recognition•Hot Spot Recognition•Annotations

University of California, San DiegoAVIARY System Architecture

• 2 PCs• 4 Static Cameras• 4 PTZ Cameras• No SSL

RicohPortable Meeting Recorder

RicohPortable Meeting Recorder

Doughnut Camera

RicohPortable Meeting Recorder

Meeting Browser

Technology Limitations

Speech recognition and facial recognition algorithms are not yet as efficient as they should be in order for a smart meeting system to perform accurately

Workspace Limitations

Cameras and microphones can block view, distract, or intimidate attendees during the meeting

Security and Privacy needs to be addressed

References

[1] Zhiwen Yu and Yuichi Nakamura. 2010. Smart meeting systems: A survey of state-of-the-art and open issues. ACM Comput. Surv. 42, 2, Article 8 (March 2010), 20 pages. DOI=10.1145/1667062.1667065 http://doi.acm.org/10.1145/1667062.1667065

[2] Ross Cutler , Yong Rui , Anoop Gupta , Jj Cadiz , Ivan Tashev , Li-wei He , Alex Colburn , Zhengyou Zhang , Zicheng Liu , Steve Silverberg. (2002). Distributed Meetings. A Meeting Capture and Broadcasting System. 10 pages. http://research.microsoft.com/en-us/um/people/yongrui/ps/mm02.pdf

[3] Harold Fox. 2004. The eFacilitator: A Meeting Capture Application and Infrastructure. 89 pages. http://hdl.handle.net/1721.1/17672

[4] Yong Rui, Eric Rudolph, Li-wei He, Rico Malvar, Michael Cohen, Ivan Tashev. 2006. Ping: A Group-To-Individual Distributed meeting System. 4 pages. http://research.microsoft.com/apps/pubs/default.aspx?id=76779

[5] Dar-Shyang Lee, Berna Erol, Jamey Graham, Jonathan Hull, Norihiko Murata. 2011. Portable Meeting Recorder. 10 pages. http://rii.ricoh.com/sites/default/files/Portable_Meeting_Recorder.pdf