Introduction to MPEG-V (M di C t t & C t l)(Media Context & Control)

Oct. 31. 2009Sanghyun Joo (joos@etri.re.kr)Sanghyun Joo (joos@etri.re.kr)

Electronics and Telecommunications Research Institute

1

Problem issue

2

Transition of Media Consumptions

SMSD (Single Media Single Device)

A/V

media Single Renderer

A/V

SMMD (Single Media Multiple Devices)

MDSM (Multiple Devices Single Media)

Metaverse (roadmap report ’07)

4

MPEG-V Process

84th (‘08.04

85th (‘08.07 Hannover)

Call for

86th (’08.10 Busan)N10081WD 1.0 N10294 87th (’09 02 Lausanne)(

Archamps) Proposal

N9898Requirements

N10294WD 2.0N10082

CE

N10474

87th ( 09.02 Lausanne)

N10477 N10475 N10476

MPEG-V

83rd (‘08.01Antalya)

N9659Draft

R i t

0WD 1.0

0WD 1.0

0 5WD 3.0

0 6WD 1.0

Part 1 Part 2 Part 3 Part 4N10616 N10617 N10618 N10619 88th (’09 04 M i)

Part 1 Part 2 Part 3 Part 4

y ) Requirements N10616WD 2.0

N10617WD 2.0

N10618WD 2.0

N10619WD 2.0

88th (’09.04 Maui)

ISO/IEC 23005

82nd (’07.10 Shenzhen)

Committee Draft(CD)

Call for Requirements

89th (’09.07 London)

Fi l C itt D ft(FCD) 90th (’09 10 Xi )

ISO/IEC 23005

M14900Proposal on RoSE

Exploration

Study of FCD

N9426Framework for Representation of Sensory Effects I f ti

Final Committee Draft(FCD)

Final Draft International Standard (FDIS)

90th (’09.10 Xian)

91st (’10.01 Kyoto)

5

Information (contexts and objectives)

Final Draft International Standard (FDIS)

International Standard (IS)

RoSE System Architecture

Scope of Standardization (normative)

6

MPEG-V Project

█ 23005: MPEG-V (Media Context and Control)

Part 1. Architecture

Part 2. Control InformationPart 2. Control Information

Part 3. Sensory Information

Part 4. Virtual World Object Characteristics

Part 5. Data Formats for Interaction Device

Part 6. Reference Software

Part 1: Architecture

8

Instantiation 1: Instantiation 1: RoSERoSE

9

Instantiation 2: Exchanges between real and virtual worldsworlds

10

Instantiation 2: Exchanges between real and virtual worldsworlds

2.1 Full motion control and navigation of avatar/object

i h l i iwith multi-input sources

2.2 Virtual Travel

2.3 Serious gaming for Ambient Assisted Living

2.4 Virtual Traces of Real Places

11

Instantiation 3: Exchanges between virtual worldsInstantiation 3: Exchanges between virtual worlds

3.1 Avatar Appearance3.2 Social presence3 3 G D i i ki i th t t f S ti l Pl i3.3 Group Decision-making in the context of Spatial Planning3.4 Consumer Collaboration in Product Design Processes

along the Supply Chain3 5 Virtual Objects3.5 Virtual Objects

Figure 8 — Geode of industry and sciences museum overviewFigure 8 — Geode of industry and sciences museum overview

12

Instantiation 4: Control of avatars and other virtual Instantiation 4: Control of avatars and other virtual objects by real signalsobjects by real signals

4.1 Internet Haptic Service - YouTube, Online Chatting 4.2 Next Generation Classroom – Sensation Book 4.3 Immersive Broadcasting – Home Shopping, Fishing Channels4.4 Entertainment – Game (Second Life, Star Craft), Movie Theater4.5 Virtual Simulation for Training – Military Task, Medical training

Figure 8 — Geode of industry and sciences museum overview

13

Part 2: Control InformationPart 2: Control Information

Normative referencesISO/IECIS 15938-12, Information technology -- Multimedia content description gy pinterface -- Part 5: Multimedia description schemeISO/IECIS 21000-7, Information technology -- Multimedia framework (MPEG-21) -- Part 7: Digital item adaptation

Device CapabilitiesSensory Device capability base type

User Sensory PreferencesSensoryPreference base type

Sensory Device capability base attributesIndividual device capability typesSensor capability base type

User sensory preference base attributesUser’s sensory preference types for individual sensory effects

Sensor capability base attributesIndividual sensor capability types

Device Commands Sensed InformationDevice CommandsDevice command base typeDevice command base attributesIndividual Device Command Types

Sensed Information Sensed information base typeSensed information base attributesIndividual sensor types

14

Individual Device Command Types Individual sensor types

Part 2: Control InformationPart 2: Control Information

Normative referencesISO/IECIS 15938-12, Information technology -- Multimedia content description , gy pinterface -- Part 5: Multimedia description schemeISO/IECIS 21000-7, Information technology -- Multimedia framework (MPEG-21) -- Part 7: Digital item adaptation

Device CapabilitiesSensory Device capability base type

User Sensory PreferencesSensoryPreference base type

Sensory Device capability base attributesIndividual device capability typesSensor capability base type

User sensory preference base attributesUser’s sensory preference types for individual sensory effects

Sensor capability base attributesIndividual sensor capability types

Device Commands Sensed InformationDevice CommandsDevice command base typeDevice command base attributesIndividual Device Command Types

Sensed Information Sensed information base typeSensed information base attributesIndividual sensor types

15

Individual Device Command Types Individual sensor types

Part 2: Control InformationPart 2: Control Information

Individual Device Capability types

Individual sensor capability typestypes

Light capability typeFlash capability typeHeating capability type

typesLight sensor capability typeAmbient noise sensor capability typeTemperature sensor capability typeHeating capability type

Cooling capability typeWind capability typeVibration capability type

Temperature sensor capability typeHumidity sensor capability typeLength sensor capability typeAtmospheric Pressure sensor capability typeVibration capability type

Scent capability typeFog capability typeSprayer capability type

Atmospheric Pressure sensor capability typePosition sensor capability typeVelocity sensor capability typeAcceleration sensor capability typep y p y yp

Color correction capability typeTactile capability typeKinesthetic capability type

p y ypOrientation sensor capability typeAngular velocity sensor capability typeAngular acceleration sensor capability type

RigidBodyMotion capability type Force sensor capability typeTorque sensor capability typePressure sensor capability type

16

Motion sensor capability typeIntelligent camera capability type

Part 2: Control InformationPart 2: Control Information

User’s sensory preference types for individual sensory effects

Individual Device Command Typesfor individual sensory effects

Light preference  typeFlash preference typeHeating preference type

TypesLight typeFlash typeHeating typeHeating preference type

Cooling preference typeWind preference typeVibration preference type

Heating typeCooling typeWind typeVibration typeVibration preference type

Scent preference typeFog preference typeSpraying preference type

Vibration typeScent typeFog typeSpraying typep y g p yp

Color correction preference typeTactile preference typeKinesthetic preference type

p y g ypColor correction typeTactile typeKinesthetic type

RigidBodyMotion preference type RigidBodyMotion type

17

Part 2: Control InformationPart 2: Control Information

Individual sensor typesLight sensor typeLight sensor typeAmbient noise sensor typeTemperature sensor typeHumidity sensor typey ypLength sensor typeAtmospheric Pressure sensor typePosition sensor typeVelocity sensor typeAcceleration sensor typeOrientation sensor typeAngular velocity sensor typeAngular acceleration sensor typeForce sensor typeTTorque sensor typePressure sensor typeMotion sensor typeIntelligent camera type

18

Intelligent camera type

Part 3: Sensory InformationPart 3: Sensory Information

Normative referencesISO/IEC 15938-3, Information technology — Multimedia content description gy pinterface — Part 3: VisualISO/IEC 15938-5, Information technology — Multimedia content description interface — Part 5: Multimedia description schemesISO/IEC 21000 7 Information technology Multimedia framework (MPEG 21)

Sensory effects description 

ISO/IEC 21000-7, Information technology — Multimedia framework (MPEG-21) —Part 7: Digital Item Adaptation

Sensory effect vocabularylanguage (SEDL)DeclarationsGroupOfEffects

Light, colored light, flash lightTemperatureWindGroupOfEffects

EffectReferenceEffect

VibrationWater sprayerScentFogShadowColor correctionKi th ti

19

KinestheticTactile

Part 4: Virtual World Object CharacteristicsPart 4: Virtual World Object Characteristics

❐ AppearanceContains the high level description of the appearance and may refer a media containing the exact geometry and texturea media containing the exact geometry and texture

❐ Haptics PropertiesContains the high level description of the haptics properties

❐ AnimationContains the description of a set of animation sequences that the avatar is able to perform and may refer to several medias containing the exact (geometric transformations) animation parameters

❐ Communication SkillsContains a set of descriptors providing information on the different modalities an avatar is able to communicate

❐ PersonalityContains a set of descriptors defining the personality of the avatar

❐ ControlContains a set of descriptors defining possible place-holders for

20

Contains a set of descriptors defining possible place holders for sensors on body skeleton and face feature points

Part 5: Data Formats for Interaction DevicePart 5: Data Formats for Interaction Device

- New part for General uses of Part 2 & 3

- To be discussed in an ad hoc group meetingTo be discussed in an ad hoc group meeting

Part 6: Reference SWPart 6: Reference SW

- Current Status: WD 2.0 of ISO/IEC 23005-6

- Contains only for Part 3 Sensory InformationContains only for Part 3 Sensory Information

- Add Reference SW for Part 2 and Part 4

- Consist of the module to parse and validate the tools- Consist of the module to parse and validate the tools

contained in the individual Part (2/3/4) of the standard.

Category

Ref. SW Availability

ResponsibilityCategory Responsibility91st(WD 3.0)

92nd(CD)

93th(FCD)

94th(Study)

95th(FDIS)

Part 2 ETRI ETRI ETRI ETRI ETRI B S ChoiPart 2 ETRI ETRI ETRI ETRI ETRI B.S. Choi

Part 4 Samsung Samsung Samsung Samsung Samsung J.J. Han

Outputs of MPEG-V at 90th Xian meeting

X 23005-1 Media context and control - Architecture10981 DoC on ISO/IEC CD 23005-1 Architecture10982 Text of ISO/IEC FCD 23005-1 Architecture10982 Text of ISO/IEC FCD 23005-1 Architecture

X 23005-2 Media context and control – Control Information10983 DoC on ISO/IEC CD 23005-2 Control Information10984 Text of ISO/IEC FCD 23005-2 Control Information10985 WD of ISO/IEC 23005-2 Control Information

X 23005-3 Media context and control – Sensory Information10986 DoC on ISO/IEC CD 23005-3 Sensory Information10987 Text of ISO/IEC FCD 23005-3 Sensory Information

X 23005-4 Media context and control – Virtual World Object Characteristics

10988 DoC on ISO/IEC CD 23005-4 Virtual World Object Characteristics10989 Text of ISO/IEC FCD 23005-4 Virtual World Object Characteristics

X 23005-5 Data Formats for Interaction DevicesReq est for s bdi ision of ISO/IEC FCD 23005 5 Data Formats for Interaction De i10990 Request for subdivision of ISO/IEC FCD 23005-5 Data Formats for Interaction Device

10991 Text of ISO/IEC FCD 23005-5 Data Formats for Interaction DeviceX 23005-6 Media context and control – Reference SoftwareX 23005 6 Media context and control Reference Software

10992 Workplan of Reference Software regarding ISO/IEC 23005-2

Thank you.y

2424