ISSCC 2002 / SESSION 1 / PLENARY / 1.3

1.3

_ _

Ambient Intelligence: the Next Paradigm for Consumer Electronics: How Will it Affect Silicon?

Fred Boekhorst

Philips Research, Eindhoven, The Netherlands

Ambient Intelligence refers to an environment where the user experience is what matters. People want to have fun, feel free, enjoy life, feel secure, be in control and be productive. This experience is not linked to one particular device but is realized by a network of devices present in the environment that will provide us these experiences in an intelligent way.

Ambient Intelligence IS introduced as a new paradigm in consumer electronics. Some of the technologies required for Ambient Intelligence are covered, with a focus on IC consequences. Ongoing work in three fields is described: ubiqui- tous radio, intuitive user/system interfaces and three-dimensional visual displays. These three fields highlight the diverse nature of Ambient Intelligence technologies as well as the resulting requirements for ICs. Considerations with respect to an introduction timeline of Ambient Intelligence are given.

The changing landscape of consumer electronics

Ambient Intelligence, the concept Challenges for silicon Topics - Ubiquitous radio concept - Easy access to audio content - Towards an enhanced visual experience

A likely timeline of introduction Conclusions

clevision in the sixties

See Digest page 28

Television now The inside story

n

Ambient Intelligence, the concept

An environment that is sensitive, adaptive and responsive to the presence of people or objects An environment where technology is embedded, hidden in the background An environment that augments activities through smart, non-explicit assistance An environment that will preserve security, privacy and trustworthiness while utilizing information when needed and appropriate.

I __ I ----_5_ -=_c_i__-_ -==--- -.--=-

Ambient Intelligence Domains & Drivers

Domains: - Home [focus of this presentation] - Car - Office

Drivers - Ubiquitous communications - Distributed computing - Intelligent interfaces

.

50 years of semiconductor industry Trend = + 17% I Y e a r

__

0. I

0.01

1950 60 70 80 90 (CY) SOUKC: SI& WSTS

*.----,---- -- w - - m

Process technology drives progress n n

LAN, WAN HDD DVD

Internet Digital TV

1 - Solid-state Process

pProcessor DSP ASlCs

'pchnology I

a LCD

Wearables

Plastic electronics

CCD -I .- -- I---- - ______ - - - ~

Home connected to the world

Continued on Page 370

0 lEEE / lSSCC 2002 VISUALS SUPPLEMENT 17

ISSCC 2002 / PAPER 1.3

Continued from Rape 17

First generation IT- svstems

_==_-=__=_I__F_i_ *-=**--

Second generation IT- systems

Third generation IT- systems

Interfaces to computing machinery

The Evolution of Computers and Networks

The Evolution of User Interfaces

Room size Mainframe

Mini-computer Desktop PC

Notebook PC PDA Networked

appliance Personalized

VU1 anticipatory GUI multi-modal

u1 High-level languages

Assembly and machine code

370 , ISSCC 2002 VISUALS SUPPLEMENT / 0 IEEE

_ - ~ _ _ ~- r”*y----- -_--vi =--=- __I_- I, - -r - - - I,- Challenges for Silicon

A multitude of environment / system interfaces

Large variation in computational complexity - 1 KOPS [control] to 1 TOPS [synthetic video] Large variation in power requirements - < 100 uW [no battery] to > 1 W [mains operated]

Ubiquity presumes ultra low cost ~ < 10 ct [tagging]

Design time & effort must be limited

- Bitrates from 1 bids to SGbit/s

i 3

t I 1 Wireless Data Transfer Market 1

ZigBee

1 The aim of ZigBee is to bring about a low cost, s I low bit rate, low power, short range RF data 1 1 link aimed at consumer products in and around

~ the home as well as industrial sensor 1 ! applications. k--.----- -=-==--_LI F__ -%-‘-Sw-- -_* IF-

h ZigBee characteristics ii

lOkbps - 115.2kbps data throughput 10-75m coverage range (home/garden) Support for up to 255 nodes Support for up to1 00 co-located networks Support for 4 time-critical devices 0.5-2 year battery life (2 x AA) Up to 5ms-’ permitted mobility Module cost: $1.5-$2.5 in 2003/4

_ l l _ l l l _ l _ ~ _ _ I_ - __ - -- _I -------$--.-=A ---- _j__li_i___

- Actual message flow

I Aooarent message flow I _ _ _ _

Between Radios and Tags ...

I -

Funclionulity

Active <radio’ tag

---------

Ultimate Miniaturization!

Existing ZigBee based active tag Ubiauitous radio ?

. . . . .

0 IEEE / ISSCC 2002 VISUALS SUPPLEMENT 371

_ I “i?l - ~ _ - _1_- _ _ I IC objectives:

Small size (- few mm*)

Low power design - lOOpW * Integrated antenna -I- m

No external frequency reference

No external ‘glue’ components

ated battery (f power coupli

Power optimised protocol

Easy Access

Problem statement - Storage capacity doubling every two years - Compression technologies rapidly improving j

I High quality audio coding now possible @ 40Kb/s 3000 songs on a DVD ; 60000 songs on a HDD [100GB] 1 1

So, you can store anything you want.. .....

But will you ever find it back ?

Easy Access

\ ii

1 1 1 Necessary ingredients ’ ’ I I/

What is Easy Access ?

- Innovative user interface options that support different user strategies for handling large collections of media content in CE devices.

- Knowledge on the user- and technology related issues for implementing multi-modal and personalised user interfaces.

- Methods and tools for the specification and validation of multi-modal user interface solutions.

........... -.-.___ ~

D 1 Multimodal interaction

......................... I.-.-.-. - --

I Query by Humming

of a song while the system seeks the

Query by Humming is a novel way to access music by producing a ‘hook-line’

Humm,ng

i 1 11 I song containing that melody. 11

I I For those occasions in which: - you have only some knowledge of the melody - you don’t want to browse, hoping you will recognize the title - you want to find songs with similar ‘hook-lines’

__.I_ -

. __II__ .... l_ll_l ____ . ...... ................................................

Signal Processing steps Singing

inner re resentation J

Concept demonstrator

Multi modal & personalized internet audio jukebox: - Control through touch and

- Feedback through graphics phoneme based speech recognition

and audio (with text-to-speech) - Access to “unknown” content through Query by Humming - Personalized access based on Speaker Identification and

Automatic Collaborative Filtering

Processing steps being cascaded: - Pitch extraction, event detection, note

quantization, melody encoding, approximate pattern matching

Complexity mainly determined by approximate pattern matching: - 0 (n.m) where

n = number of notes sung - m= number of notes of the melody in the database

___ - - -_ I x - ~

Computational load [2]

Realistic system:

Approximate pattern matching takes 3 GOPS - N=12, M=300,20000 songs, T<0.5 second

372 ISSCC 2002 VISUALS SUPPLEMENT / 0 IEEE

~ 1 1 ~- ___ il I ~ _ _ II___ _II

Towards an enhanced visual experience

Comnt generation

A 3D TV chain

C0mpRssiOZl & N Displaying,

Transmission

2D => 3D MPEG2 Stereo cameras MPEG4 Camera + sensors ... ...

3D LCD shutter-glasses Dual-CRT PTV ...

Different ways of representing 3D

t 3D models

U

z e.

L (also useful for

2

m VR; gaming)

no control adjustment of interactivity; depth impression editing

Control by viewer -*-.=-- ---*- =I?--- nr "P*. ~

Key requirements Arbitrary scene content Fully automatic

Depth estimation (2.5D) from 2D video

Original video Segmentation Depth maps

1 Assumption: color and depth

discontinuities coincide *Segment-based motion estimation .Motion to deDth conversion

Courtesy: Marc Pollefeys, KU Leuven - __

Combining depth maps to a 3D model

0 IEEE / ISSCC 2002 VISUALS SUPPLEMENT 373

~ ~ Computational load

Most demanding processing steps: - Segment-based motion estimation [5 GOPS] - Rendering:

Multi-view auto-stereoscopic display, 3000*2000, allowing multiple SD [720*576] images Nearest neighbor interpolation: 4 GOPS Bi-lineair interpolation: 20 GOPS

--_ - ---_ - i_i_a-____L=-.L

Embedded Systems: Design objective Computational efficiency [Roza]

. 1 0 5 0 2 5 0 13 007

100 - Feature size [ ~ m ]

Design space spanned by different types of cores I _ __ __ __l_ll _lllllll_l_ ~ ______ __ ___ - - - __ - I

Types of Embedded Cores weakly-programmable (parametrizable)

application specific and programmable functions (FSMD) e.g. video functions

cores (ASIP) e.g. Tensilica, AlRT VLIW

programmable DSP cores: e.g. R.E.A.L.,

programmable general purpose CPU cores

Palm, Oak ...

e.g. ARM, MIPS, TM ...

+ embedded memories

= a heterogeneous multiprocessor _____i__I_ ---- architecture _-

Current platforms: DVP example

domain specific

5 busses and programmable

____ _l___-.._...___l.l_l..l.l. I ____I_

DVP example: Viper (PNX85007

Communication in a Multiprocessor Architecture communication

Networks-on-Silicon Communication centric separation of concerns

Multi-processor Platform based design architectures Reuse of IP

Embedded processors HLS + SW-compilers ILP, VLlW

computation

Networks-on-Silicon

3 router E Multistage network = graph construct with routers & links From global to segmented interconnect Similarity with telecom

consuming design iterations ~

_____XI--

Networks-on-Silicon: OS1 Protocols

374 ISSCC 2002 VISUALS SUPPLEMENT / 0 IEEE I"

888CC 2002 / PAPER 2.1

Continued from Rage 19 - A likely timeline of introduction

Not a “Big Bang” but rather a gradual blending

Ubiquitous communications will further develop

Intelligent interfaces will be enabled step-by-step

Natural- and synthetic image processing is very

into daily life

to cover the whole spectrum ;-)

through exploitation of Moore’s law

compute-cycle demanding and needs progress in novel architectures and algorithms. A new paradigm for on-chip communications is needed.

~~~~- -*--?____ 3- - -z--- -I__ _--a --- Conclusions

Ambient Intelligence has been presented Challenges for IC design are many-fold: - Ultra low cost [ubiquity] - Ultra low power [5 year battery life; off the

- Ultra high-speed [lo0 GOPS .... I TOPS] - Novel architectural approaches on silicon needed

- Car, office, home

consumer electronics means business

environment]

Applications in various domains

Convergence of computing, communications and

, _ - P I u

Standard FT-CCD (3) _. first step completed: B2 and B3 fon’

1 2 3 4

- + + - - + + - - + + - - + + - - + + - - =_=-I - _Fu _____-- D___ l l ~ l l _ _ - -- - - _ _ _ _ _ _ ~ __r _I_c_

Standard FT-CCD (4) second step: B2, B3 and B4 ‘on’

7

- + + - - + + - - + + - - + + - - + + -

Standard FT-CCD ( 5 ) second step completed: B3 and B4 ‘on’

1 2 4

- + + - - + + - - + + - - + + - - + + - _I.. -4 - --w_ = --a n -- __I _---

Standard FT-CCD (6) third step completed. B1 ‘on’, B3 ‘off

1 2 3 4

0 IEEE / ISSCC 2002 VISUALS SUPPLEMENT 375