SMART SPACES TESTBED

Marty Herman, ChiefVince Stanford, Computer Specialist

INFORMATION ACCESS AND USER INTERFACES DIVISION

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

Smart Space

• Embedded with computers, information appliances, multi-modal sensors

• Allows people to work efficiently through unprecedented access to information and help from computers

• Work can be done collaboratively or individually

Characteristics of Smart Spaces

Situation AwarenessIdentify and perceive users and their actions and goalsUnderstand and anticipate user needsUses speech and natural language understanding, computer vision, multi-modal fusion

Information AccessFacilitate interaction with information-rich sourcesAllow rich, natural forms of interaction between humans and computersProvide multi-media (e.g., text, voice, images, video) information

extraction

Information presentationProvide extensive capabilities for presenting information in an optimal and integrated mannerProvide immersive presentation capabilities

Characteristics of Smart Spaces(cont.)

CollaborationProvide for collaborative working environmentIncrease the rate of information interchange among usersPermit mobile workers to interact and participate in smart space activities

Mobile computingProvide for use of mobile devices and receivers, allowing ubiquitous “anywhere, anytime” accessProvide for discovery and integration of mobile devices with smart space infrastructure

Record keepingProvide improved memory of activities and deliberations for later useRecord and summarize dialogue, activities, and events

Create Smart-Space Integration Testbed at NIST

• Showcase future of Smart Space technologies• Integrate component technologies for demos• Develop and apply measurement and testing approaches• Develop and test prototype standards (e.g., interface

standards)• Develop test collections to support evaluations• Infrastructure for industry and academia to work hand-in-hand

with NIST• Facilitate adoption by industry

NIST Smart Spaces Testbed Scenario:Smart Meeting Room

• Room understands and guides meeting participants• Senses who is talking to whom and where they are located• Realizes when information is requested and outputs information that

seems to be useful• Engages in dialogue with participants to get more information• Room connected to Internet and digital libraries/databases• Personal information from palm/pocket computers are integrated

into room’s information infrastructure• Information displayed on information appliances and wall displays• Room aids in collaboration both within the room and with field

personnel

•Situation awareness•Speech, natural language input•Computer vision input•Integration of handheld computers into room•Retrieval & visualization of information•Distributed collaboration

Smart Meeting Room

Applications:

• Business meeting room• Medical consultation meeting room• Training and education facility• Military command center• Crisis management command center

Role of NIST

Smart Spaces is an emerging research area. Many of its component technologies require research to advance the state of the art. NIST can provide the following:

• measurement methods• testing and evaluation approaches• benchmark tests• reference materials (test data and test protocols)

Role of NIST(cont.)

Many technology suppliers will be required to make the vision of Smart Spaces a commercial reality.

To encourage commercial innovation, NIST can work with industry to provide open interface standards for interoperability.

Near Term Smart Space

• Experimental Collaborative interface using:– Microphone array, with beam fan, for acoustic signal

acquisition– Acoustic source location and tracking– Continuous speech recognition (CSR) using single

channel reduced from array– Speaker identification combined with CSR for

collaborative groups– Shared high resolution visual interface with information

visualization

Near Term Smart Space(cont.)

• Experimental Collaborative interface using:– Video camera array to identify, locate and track

individuals in the smart space– Video teleconferencing– Initial test case will use several functions:

• Broadcast news transcription• Real time spoken query processing• Information retrieval using BN transcripts

Current Status...• Lab space allocated; construction begins:

– Initial smart space November 23– Acoustically conditioned space 1Q99

• Computing, display, sensing infrastructure:– Dual Processor SGI Octane

• 640Mb Ram• thirty-two channels A/D• Dual camera inputs

– 1280x1024 large screen projector– Parallel Linux cluster:

• 112Gb disk array• Seven 400Mhz P-II compute nodes

– Gigabit Ethernet switch in room– Microphones:

• Desktop• Wall mounted array(s)

– Cameras:• Three computer controlled• One digital fixed mount

– 384Kb video teleconferencing– Wireless connection/Palm Pilot

• External cooperative research efforts:– Microphone array/acoustic source locator

from Rutgers CAIP; Jim Flanagan P.I.– BBN and IBM offering research and real

time speech recognition– Investigating possible use of MIT speaker

identification– Investigating possible use of U of Md visual

localization and gesture recognition• Internal cooperative efforts:

– Information retrieval (PRISE)– Spoken document retrieval with spoken

query (SPIDERS)– VRML smart space representation– Information visualization– Face recognition technology

Desired Long Term Capabilities/Technologies

• Accurate, robust speech recognition for transcription and command• Anticipatory Web and database query based on speech• Speech understanding:

– Spoken dialog abstraction– Selective recording of meeting minutes

• Data stream segmentation/annotation• Face/expression recognition• Persistence of meeting memory, automatic links to related meetings• Sensor fusion (acoustic/visual/other)• Smart white boards• Image understanding and person recognition• Participant sensing, task identification, and adaptive response

Metrics and Metrics Research Needed

• End-to-end metrics needed for multi-stage processes, e.g.: Full SDR– Sensing– Recognition– Understanding– Response

• Training and test reference data sets needed for:– Component recognition tasks, e.g.: CSR, speaker ID, gesture

recognition – Data reduction, e.g.: array processing and source location– Information extraction and summary

Metrics and Metrics Research Needed (cont.)

• Initial Smart Space is needed as test bed to bootstrap future improvements

• Command/Crisis Management may offer measurable, reproducible tasks

• Need well defined tasks with solution quality, time, and labor level measurements

Interoperability Issues are Severe• Rapid handheld appliance discovery and connection to IP

infrastructure, e.g.: IP masquerading or NAT • Dense wireless device deployment:

– IR

– RF• Multi-resolution devices and displays• Device protocol discovery, automatic translation among

heterogeneous protocols used by multiple devices• Multimedia output into the space and acoustic input• Security and accessibility to and by infrastructure