Enabling integrated cross-media entertainment services over heterogeneous networks
Introduction:Reasoning, business aspects, involved processes
Dr. Kostas KarpouzisDr. Kostas KarpouzisImage, Video and Multimedia Systems Lab, National Technical University of AthensImage, Video and Multimedia Systems Lab, National Technical University of AthensInstitute of Communication and Computer SystemsInstitute of Communication and Computer Systems
The current scene…
Digital Content A valuable asset nowadays Associated with a variety of emerging and novel distributed
multimedia services Big revenue bringers: entertainment services
Convergence of media and technologies From single, distinct ones to multi-disciplinary ones
Allowing advanced service provisioning and consumption
The current scene…
The plethora of terminal devices available for networking access lead to immense market opportunities for content and service providers
The emergence of advanced network infrastructures enables fast, efficient and reliable end-to-end transmission of huge multimedia content
Media providers faced with two choices: select the most appropriate device for their content and tailor
the consumer experience and business models to take advantage of this environment
invest in multiple devices, access networks and content formats
the current trend… enables transparent access to services, thus more subscribers
Why entertainment..?
Entertainment and leisure market is an attractive commercial target for many business groups Data providers, telecommunication companies,
broadcasters, online publishers, device producers It further allows increasing revenue streams
A never-ending consumer interest Online entertainment and leisure market is a
small percentage of the overall it is expected to become the major boost of this
business sector
The underlying concepts…
Cross-media world: what does it mean? Publish/consume content to a variety of terminals
fixed or mobile, accessing heterogeneous networks Actually refers to:
Create once, Publish everywhere (COPE paradigm) At content/service providers, mediators, network operators
Universal access to content (UMA paradigm) From content/service providers to end-users
Heterogeneous networks Diverse core and access networks
Satellite, wireless, mobile, cable, ISDN, … Different QoS management
User-oriented Services driven by users’ real needs and preferences
So…
How can the same content be transmitted over different channels and consumed at various terminals without costly re-processing and re-production under specific QoS requirements dictated both by content/service providers and end-users?
The business implications
Content/service providers, network operators, content/service mediators increase market competitiveness and advantage establish new user-centered business opportunities
definition of new business models increase subscribers open up new revenue streams be technologically advanced and thus competitive
Consumers Access to any type of entertainment service wherever they
are, using any type of terminal Services tailored to their needs and preferences
The underlying value chain
Digital Rights Management
Content / ServiceCreation
Content/ Service
Mediation & Management
ContentDistribution& Delivery
PersonalizedConsumption
The technological requirements
Enable Create once, publish everywhere Universal multimedia access Adapting technology to people & people being the
centre i.e., primarily
Context awareness, content adaptation scalable coding, automatic transcoding use of standardized metadata descriptions & profiling
(MPEG-7, MPEG-21) Personalization
The technological requirements
Other requirements involved At the service provider side
Service management, discovery, integration Smart, self-adaptive content authoring & aggregation Context modeling
At the mediation/network layer Monitoring End-to-end QoS provisioning
At the end user layer interactivity mobility transparent access to services
In all, digital rights management and protection
Abstract end-to-end system design
PC PDA Mobile Digital TV
Personalized
Consumption
Content Distribution & Delivery
Re-production WorkflowContent
Aggregation
Service Managemen
t
Identification
Registration
Discovery
Integration
Cross Media
Provisioning
Context Awareness
Content Adaptation
Context Modeling
Personalization
User/Terminal Profile
QoS Managemen
t
Content and ServiceMediation
Content description / representation -
metadata
Content Authorin
g
Scalable coding
Content and ServiceCreation
Dig
ital R
ights
Managem
ent
Service/Content
Mediators
Content Providers
Service Providers
Content Transcoding
Users
Transmission
Enabling integrated cross-media entertainment services over heterogeneous networks
The concepts…:Context Awareness, Content Adaptation
Context Awareness
Context includes any information that characterizes an entity’s situation
Entity: person, place, object relevant to an interaction between a user and an application
is a function of time and environment environment is a function of users, services, resources and other entities in
environment Context-aware system
when using contexts to provide relevant information or services to the user Relevance depends on user’s task
Context-aware applications must detect, interpret and respond to contexts
Contextualization enables efficient content adaptable services and systems important component of the ubiquitous/pervasive paradigm aids in understanding the environment sufficiently
Context Awareness
Context-aware computing involves collection of context information dynamic program behavior dictated by knowledge environment
This is enabled by: Use of profiles
Environment (network, terminal, service) User (preferences, location, …) represented in standardised format (MPEG-7, MPEG-21)
since a common representation format is required to allow for interoperability in a distributed world, over heterogeneous
networks and a variety of terminals Monitoring mechanisms gathering context information Decision engines that use context information as input to adapt
content based on environment status Usually processed at service mediation and adaptation layers, but
gathered stored and interpreted at different parts of the system such as end-user terminal devices or access networks
So… a necessary step before content adaptation
Context Awareness
A context profile should be Structured
Aids in effectively filtering relevant information Interchangeable
Among different system components Mechanisms for transferring sub-trees (and not entire profiles) desirable
To compensate for big network delays Composable/decomposable
To allow maintenance in a distributed way To grasp changes in environment
Uniform Eases interpretation
Extensible To allow for future requirements/advances
Standardized To allow interoperable exchange among different entities in a distributed
environment
Context Awareness
Context-aware infrastructure Physical and logical sensors (software entity, network
components, software agents) used to collect context information related to presence, location, identity and profile of users and services/content
Typical context use Locate services and users Call up services according to user behavior Provide information for service composition Facilitate ad hoc communication mechanism between users Adaptation of QoS to changes in environment as a result of user
and service mobility Passive
context gathering and representation Active
smart context information delivery
Content Adaptation
Why adapt contents? Most contents for viewing are for the larger screens Creating multiple versions a burden Even if you don’t mind, there are just too many
possible devices Different users want different things Having one, original version is easier to manage
Content adaptation is about generating any content version from one single original version
Create once, Publish everywhere (COPE) Universal Multimedia Access (UMA)
Content Adaptation
Pre-adaptation Keeping just the original version (any other version is runtime-
generated) could be slow Pre-adaptation
to create all possible versions (at creation/production phase), and do static “selection” at runtime, or
to create just a few essential versions, and do dynamic adaptation – hence the “balance”
Dynamic adaptation On-the-fly adaptation Where it happens
Either at end systems (end-to-end service) Or at intermediate network nodes (active service)
At specially designed mediators
Content Adaptation
Major processes involvedAccess to and management of context
information and content/service metadataDecision engine to negotiate further actionsAdaptation by
Layered coding (scalable bitstreams)Transcoding
Content Adaptation
A Content AdaptationSystem
Mediation Layer
Content Adaptation
Use of context profileInformation and contentInstance metadatato decide upon transcodingstrategies
Enabling integrated cross-media entertainment services over heterogeneous networks
The concepts…:
Scalable Coding/Transcoding
Transcoding introduced
Video transcoding: converting a previously compressed video signal into another onePossibly different bit rate, frame rate, frame
size or compression standardInitially proposed for compressed video
bit rate scalingSun et al., 1996
Transcoding introduced (2)
Also used for:Video frame size conversion
spatial resolutionVideo frame rate conversion
temporal resolutionBit rate adaptationMultipoint video combiningError resilience
Transcoding for heterogeneous nets
Diversity of channel capacitiesChannel capacity of the outgoing
channel may be less than that of the incoming channelOr may change over time
Distribution to users with different connections target transmission channel conditions are
generally unknown during encoding
Major transcoder architectures
Cascaded Pixel Domain TranscoderKeesman et al., 1996Close-loop transcoder connects a standard
decoder and a standard encoder togetherMost straightforward method
Major transcoder architectures (2)
Open-Loop TranscoderSun et al., 1996 Input video bitstream is first partially
decoded to the DCT coefficient levelBit rate is scaled down by cutting higher
frequency coefficients or by requantizing all coefficients with a larger quantization step size
Major transcoder architectures (3)
DCT domain transcoder (DDT)Zhu et al., 1999 requantization of the DCT coefficientserror is stored in a buffer and is fed back to
the requantizer to correct the requantization error introduced in previous frames
Simplifies architecture of OLT by reusing motion vectors
Merges two motion compensation loops in the CPDT into one
Video rate control
Accurate and dynamic control of the output bit rate according to the channel bandwidth
Video rate control (2)
Two different steps:Frame-Layer Rate Control: allocate target
bits for each frame according to image complexities, buffer occupancy, or a given channel bit rate
Macroblock-Layer Bit Allocation: derive the actual quantization parameter for each picture Macroblock and make the number of produced bits meet the bit target
Frame-Layer Rate Control
Bit budget for every frame has to be determined considering the channel bit rate and buffer occupancy
Assumption for constant bit rate (CBR) Internet transmission cannot provide a
guaranteed constant bit rate channelwireless channels: high bit error rate and
variable effective channel bit rate
Frame-Layer Rate Control (2)
Theoretically, compression and error protection can be performed separately and sequentiallyShannon’s separation theorem
In practical video communication systems source coding and channel coding are combined together
Wireless video communication
Wire line channels: signal strength is relatively constant reception errors mainly due to additive noise
Wireless channels: reception errors mostly due to time-varying signal strengthmulti-path propagation from local scatterserror bursts
Wireless video transmission
Delay constraintsChannel errors hurt perceptual quality of
video at decoderEvident in standard coders
variable length codingpredictive coding compressione.g., MPEG or H.263
Challenge transmission error
At the encoder side: layered codingmultiple description codingerror resilience entropy coding
Increase robustness of video stream against channel errors
Challenge transmission error (2)
At the decoder side: Error concealment techniques recover lost
information without relying on information from the encoder
Channel coding techniques FEC codes may cause unnecessary overhead and
bandwidth waste when channel is in good state ARQ error control requires retransmission only
during periods of poor channel conditions
Abstract architecture
Delay = Processing Delay + Transmission Delay + Buffer Delay
Adaptive transcoding
Frame layer rate control based on video content frame typesvideo source coding ratescene changes
Frame types and source video
MPEG-1, -2 and H.26x use motion compensation
Motion compensation reduces temporal redundancy between successive frames
MPEG-1 and MPEG-2
In MPEG, a GOP (Group of Pictures) contains one I-frame (intra) and several P- or B-frames (inter) in a certain pattern
I-frame: no motion compensation motion compensation of following frames depends
on I-frame quality P-frames use the previous I- B-frames use both the previous and
successive I- or P-frames as references for motion compensation
H.263
Focus on low bit rateMore popular for wireless applications
Fixed quantization parameterNear constant visual objective qualityVarying bit rate
scene content different frame types
Constant rate quality degradation
H.263 (2)
Source bit rate can be recorded during encoding and saved as side informationunified quantization parameter in same type
of frames generated frame size indicates scene variations and motion activities
Thus, video content can be measured by the source bit rateUsed to calculate bit budget for every frame
Scene changes
Scene changes represent distinctive differences between adjacent video framesRapid motion of moving objectsChanges to different visual content
Information obtained from previously coded frames is no longer useful
Scene changes (2)
When scene changes happen, the first frame after scene change should be transcoded in high quality to prevent quality degradation after scene change
Scene change detectionMost MBs in the anchor frame will be
encoded as INTRA blocks INTRA mode MBs percentage can be used
to detect scene changes
Adaptive frame layer rate control
Relaxed requirement for end-to-end delay
Allowed initial startup delaydecoder buffer smoothes delay jitters
H.263 determines frame budget by buffer occupancy and channel bandwidthnearly constant bit budget similar end-to-
end delay for each frame
Adaptive frame layer rate control (2)
Adaptive methods consider scene changes, frame types and source coding ratee.g., I-frames are transcoded as I-frames,
with unified quantization parameter used for every MB, while the bit budget for B-frames will be half of that for P-frames
Scene change detected anchor frame is transcoded as an I-frame
A practical scenario
Offline encoding Initial presentation is prepared according to
pre-defined scenariosUses pre- created static content
Online transcoding In charge of transmission of initial
presentationGeneration, encoding and transmission of
real-time interactions
Enabling integrated cross-media entertainment services over heterogeneous networks
The technologies…:Metadata & Profiling (environment, user, terminal)
The need for standardised metadata (MPEG-7)
Today people cannot easily create, find, edit, share, and reuse media
Computers don’t understand media content Media is opaque and data rich We lack structured representations
Without content representation (metadata), manipulating digital media will remain like word-processing with bitmaps
Need standardized metadata framework Designed for video and rich media data Human and machine readable and writable Standardized and scalable Integrated into media capture, archiving, editing, distribution, and
reuse
MPEG-7
MPEG-7 Describing the multimedia content data that supports some
degree of interpretation of the information’s meaning, which can be passed onto, or accessed by, a device or a computer code
Motivation Create standardized multimedia description framework Enable content-based access to and processing of
multimedia information on the basis of descriptions of multimedia content and structure (metadata)
Support range of abstraction levels for metadata from low-level signal characteristics to high-level semantic information
MPEG-7 Query Examples
Play a few notes on a keyboard and retrieve a list of musical pieces similar to the required tune, or images matching the notes in a certain way, e.g., in terms of emotions
Draw a few lines on a screen and find a set of images containing similar graphics, logos, ideograms,...
Define objects, including color patches or textures and retrieve examples among which you select the interesting objects to compose your design
On a given set of multimedia objects, describe movements and relations between objects and so search for animations fulfilling the described temporal and spatial relations
Describe actions and get a list of scenarios containing such actions
Using an excerpt of Pavarotti’s voice, obtaining a list of Pavarotti’s records, video clips where Pavarotti is singing and photographic material portraying Pavarotti
MPEG-7 Sample Application Areas
Architecture, real estate, and interior design (e.g., searching for ideas)
Broadcast media selection (e.g., radio channel, TV channel)
Cultural services (history museums, art galleries, etc.)
Digital libraries (e.g., image catalogue, musical dictionary,
bio-medical imaging catalogues, film, video and radio archives)
E-Commerce (e.g., personalized advertising, on-line
catalogues, directories of e-shops) Education
(e.g., repositories of multimedia courses, multimedia search for support material)
Home Entertainment (e.g., systems for the management of
personal multimedia collections, including manipulation of content, e.g. home video editing, searching a game, karaoke)
Investigation services (e.g., human characteristics recognition,
forensics) Journalism
(e.g. searching speeches of a certain politician using his name, his voice or his face)
Multimedia directory services (e.g. yellow pages, Tourist information,
Geographical information systems) Multimedia editing
(e.g., personalized electronic news service, media authoring)
Remote sensing (e.g., cartography, ecology, natural
resources management) Shopping
(e.g., searching for clothes that you like) Social
(e.g. dating services) Surveillance
(e.g., traffic control, surface transportation, non-destructive testing in hostile environments)
MPEG-7 Scope
To define the description framework
Scope of MPEGScope of MPEG--77Scope of MPEGScope of MPEG--77
Descriptionconsumption
DescriptionconsumptionDescriptionDescriptionDescription
generation
Descriptiongeneration
MPEG-7 Metadata Framework
Data “multimedia information that will be described using MPEG-7, regardless
of storage, coding, display, transmission, medium, or technology.” Feature
“a distinctive characteristic of the data [that] signifies something to somebody.”
Descriptor “A representation of a Feature. A Descriptor defines the syntax and the
semantics of the Feature representation.” Description Scheme
“The structure and semantics of the relationships between its components, which may be both Descriptors and Description Schemes.”
Description Definition Language (XML Schema) “A language that allows the creation of new Description Schemes, and,
possibly, new Descriptors. It also allows the extension and modification of existing Description Schemes.”
MPEG-7 Framework
TagsTags
<scene id=1><time> ....<camera>..
<annotation</scene>
InstantiationInstantiation
TagsTags
<scene id=1><time> ....<camera>..
<annotation</scene>
InstantiationInstantiation
Descriptors:Descriptors:(Syntax & semantic(Syntax & semanticof feature representation)of feature representation)
D7
D2
D5
D6D4
D1
D9
D8
D10
101011 0
Encoding&
Delivery
101011 0
Encoding&
Delivery
D3
LanguageDescription Definition extensionextension
DefinitionDefinitionLanguage
Description Definition extensionextension
DefinitionDefinition
Description SchemesDescription Schemes
D1
D3D2
D5D4D6
DS2
DS3
DS1
DS4StructuringStructuring
Description SchemesDescription Schemes
D1
D3D2
D5D4D6
DS2
DS3
DS1
DS4
Description SchemesDescription Schemes
D1
D3D2
D5D4D6
DS2
DS3
DS1
DS4StructuringStructuring
MPEG-7 Standard Parts
1. MPEG-7 Systems The binary format for encoding MPEG-7 descriptions and the terminal architecture
2. MPEG-7 Description Definition Language The language for defining the syntax of the MPEG-7 Description Tools and for
defining new Description Schemes3. MPEG-7 Visual
The Description Tools dealing with (only) Visual descriptions4. MPEG-7 Audio
The Description Tools dealing with (only) Audio descriptions5. MPEG-7 Multimedia Description Schemes
The Description Tools dealing with generic features and multimedia descriptions 6. MPEG-7 Reference Software
A software implementation of relevant parts of the MPEG-7 Standard with normative status
7. MPEG-7 Conformance Testing Guidelines and procedures for testing conformance of MPEG-7 implementations
8. MPEG-7 Extraction and Use of Descriptions Informative material (in the form of a Technical Report) about the extraction and use
of some of the Description Tools (under development)
MPEG-7 Description Tools
MPEG-7 Top Level Hierarchy
MPEG-7 Still Image Description
Referencing Temporal Media
MPEG-7 Video Segments Example
MPEG-7 Segment Relationship Graph
MPEG-7 Conceptual Description
MPEG-7 Summaries
MPEG-7 Collections
MPEG-7 Application Framework
MPEG-7 Applications Today
IBM MPEG-7 Annotation Tool Assists in annotating video sequences with MPEG-
7 metadata Ricoh MPEG-7 MovieTool
A tool for creating video content descriptions conforming to MPEG-7 syntax interactively
Canon MPEG-7 Speech Recognition engine Web site allows you to create an MPEG-7 Audio
“SpokenContent” description file from an audio file in “wav” format
IBM MPEG-7 Annotation Tool
IBM MPEG-7 Annotation Tool
The IBM MPEG-7 Annotation Tool assists in annotating video sequences with MPEG-7 metadata Each shot in the video sequence can be annotated with static
scene descriptions, key object descriptions, event descriptions, and other lexicon sets
The annotated descriptions are associated with each video shot and are stored as MPEG-7 descriptions in an XML file
Can also open MPEG-7 files in order to display the annotations for the corresponding video sequence
Customized lexicons can be created, saved, downloaded, and updated
Ricoh MovieTool Creates an MPEG-7 description by
loading video data Provides visual clues to aid the
user in creating the structure of the video
Automatically reflects the structure in the MPEG-7 descriptions
Visually shows the relationship between the structure and MPEG-7 descriptions
Presents candidate tags to help choose appropriate MPEG-7 tags
Checks the validation of the MPEG-7 descriptions in accordance with MPEG-7 schema
Can describe all metadata defined in MPEG-7
Is able to reflect any future changes and extensions made to MPEG-7 schema
Canon MPEG-7 ASR Tool
MPEG-7 Resources
http://www.chiariglione.org/mpeg/index.htm
http://www.mpegif.org/mpegif/index.php
http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0471486787.html
The need for standardized profiles (MPEG-21)
Dictated by the emerging demands on Universal multimedia access Ubiquitous and pervasive computing Heterogeneity of networks and end-user terminals Use of content and services anywhere, at any time
To allow for uniform, exchangeable and interoperable definition of context profiles
To allow for efficient negotiation mechanisms for context retrieval and content adaptation anywhere in the content production, transmission and consumption chain
MPEG-21
A normative open framework for multimedia delivery and consumption for use by all the players in the delivery and consumption chain
Based on two essential concepts: the definition of a fundamental unit of distribution and
transaction (the Digital Item) Digital Items: the “what” of the Multimedia Framework
a video collection, a music album the concept of Users interacting with Digital Items
Users: the “who” of the Multimedia Framework
Goal define the technology needed to support Users to exchange,
access, consume, trade and otherwise manipulate Digital Items in an efficient, transparent and interoperable way
MPEG-21 scope
User A User BTransaction/Use/Relationship
Digital ItemAuthorization/Value Exchange
Term inals &Networks
ContentManagem ent
and usage
I ntellectualProperty
m anagem entand
Protection
Digital ItemIdentification
and Description
MultimediaContent
Representation
Digital I temDeclaration
Exam ples:•U nique Identifiers•C ontent D escriptors
Exam ples:•Encryption•Authentication•W aterm arking
Exam ples:•R esource Abstraction•R esource M gt. (Q oS)
Exam ples:•“C onta iner”•“Item ”•“R esource”
Exam ples:•S torage M gt.•C ontent Personalization
Exam ples:•N atura l and Synthetic•Scalab ility
Event Reporting
MPEG-21 standard parts
MPEG-21 Digital Item Declaration To describe a set of abstract terms and concepts to form a useful model for
defining Digital Items MPEG-21 Digital Item Identification
To uniquely identify Digital Items and link with related information such as descriptive metadata
MPEG-21 Intellectual Property Management and Protection To define an interoperable framework for Intellectual Property Management
and Protection MPEG-21Rights Expression Language
To define machine-readable language that can declare rights and permissions using the terms of the Rights Data Dictionary
MPEG-21 Rights Data Dictionary MPEG-21 Digital Item Adaptation
To achieve interoperable transparent access to (distributed) advanced multimedia content by shielding Users from network and terminal installation, management and implementation issues
Definition of environmental profiles
MPEG-21 DIA
Conceptual architecture a Digital Item is subject to a resource adaptation engine and a description
adaptation engine producing the adapted Digital Item
Within scope Descriptions and format-independent mechanisms in terms of resource
adaptation, descriptor adaptation, and/or Quality of Service management Not adaptation engines
Digital ItemAdapted
Digital Item
DIA Tools
Digital ItemAdaptation Engine
ResourceAdaptation Engine
DescriptionAdaptation Engine
Scope of standardization
Digital ItemAdapted
Digital Item
DIA Tools
Digital ItemAdaptation Engine
ResourceAdaptation Engine
DescriptionAdaptation Engine
Scope of standardization
Scope of standardization
MPEG-21 DIA Tools
Three major categories
Usage Environment Description Tools
• User Characteristics• Terminal Capabilities• Network Characteristics• Natural Environment Characteristics
Digital Item Resource Adaptation Tools
• Bitstream Syntax Description• Terminal and Network QoS• Bitstream Syntax Description Link• Metadata Adaptability
Digital Item Declaration Adaptation Tools
• Session Mobility• DIA Configuration
Usage Environment Description Tools
• User Characteristics• Terminal Capabilities• Network Characteristics• Natural Environment Characteristics
Digital Item Resource Adaptation Tools
• Bitstream Syntax Description• Terminal and Network QoS• Bitstream Syntax Description Link• Metadata Adaptability
Digital Item Declaration Adaptation Tools
• Session Mobility• DIA Configuration
MPEG-21 DIA Tools & Other MPEG-21 Parts
MPEG-21 DID
Digital ItemAdaptation Engine
MPEG-21 IPMP/REL
Resource
MPEG-21 DII
ResourceAdaptation Engine
DescriptionAdaptation Engine
Descriptor
MPEG-21 DIA Tools
MPEG-21 DID’
MPEG-21 IPMP/REL’
MPEG-21 DII’
MPEG-21 DID
MPEG-21 IPMP/REL
MPEG-21 DII
MPEG-21 DIA Tools
Usage Environment Description ToolsDigital Item Resource Adaptation Tools
Digital Item Declaration Adaptation Tools
DI-1 DI-3
DI-2
Descriptor
Resource’
Descriptor’
MPEG-21 DIA Tools
MPEG-21 DID
Digital ItemAdaptation Engine
MPEG-21 IPMP/REL
Resource
MPEG-21 DII
ResourceAdaptation Engine
DescriptionAdaptation Engine
Descriptor
MPEG-21 DIA Tools
MPEG-21 DID’
MPEG-21 IPMP/REL’
MPEG-21 DII’
MPEG-21 DID
MPEG-21 IPMP/REL
MPEG-21 DII
MPEG-21 DIA Tools
Usage Environment Description ToolsDigital Item Resource Adaptation Tools
Digital Item Declaration Adaptation Tools
DI-1 DI-3
DI-2
Descriptor
Resource’
Descriptor’
MPEG-21 DIA Tools
Enabling integrated cross-media entertainment services over heterogeneous networks
The concepts…:Personalization
Why personalize?
Multitude of concurrent programsDuring the upcoming Olympic Games, more
than 5 events may be broadcast during prime-time hours
Athletics offers many events in parallelDuring weekends, many sports events and
movies are broadcast in parallel
Why personalize? (2)
Diversity in presentation optionsPeripheral information, e.g. sports statistics,
critics’ reviews for movies, other results, etc.Visual enhancements, e.g. arrows pointing
distance to goal line, lines showing WR time, etc.
Different viewing options, e.g. multiple viewing angles, virtual replays, etc.
So, personalization is…
Choosing what to view and whenChoice of a particular event or specific
portions, e.g. the final 20 minsChoice of a particular movie or movies
belonging to a specific genreChoice of live or recorded viewing
and…
Choosing how to view the programChoice of specific enhancementsChoice of available interaction, e.g. betting
opportunities during a football matchAlerts when other programs of interest are
available
Is that all?
Adapting TV to individual viewers is a topic in itselfO’Sullivan et al., 2004
Watching TV, especially sports and movies, tends to be a social activity
Strategies for combining individual user models to adapt to groupsSocial Choice Theory?
Group theory
In contrast to the use of PCs, television viewing is largely a family or social activity (Barwise and Ehrenberg, 1988)
Unfortunately, statistics do not include data on the average number of people watching TV together and who watches with whom
Culturally dependent?
Group theory (2)
Groups can be heterogeneousage, gender, and personality may influence
program choiceswatching TV together is a major activity
shared between parents and childrenyoung people like to watch TV with friends
Challenge: to adapt to a group of viewers, in such a way that each individual enjoys the broadcast
Strategies for Combining User Models
Modeling of user preferencesDirectly, via explicit profile definition Indirectly, via observation and inference
Group modelingMUSICFX, McCarty and Anagnost, 1998POLYLENS, O’Conner et al., 2001 INTRIGUE, Ardissono et al., 2002
Group modeling
MUSICFX is used in a fitness center to select background music to suit a group of people working out at any given time
POLYLENS recommends movies based on group tastes inferred from ratings and social filtering
INTRIGUE recommends places to visit for tourist groups taking into account characteristics of subgroups within that group, e.g. children and disabled
However…
Limited actual evaluation no indication of how effective these group
modeling strategies really areApplication domains differ from TV
viewingpeople usually see one movie per eveningmusic stations can play foreverno need to select a balanced groupgroup of
items
Social choice
Also called group decision makinggroup decision makingDeciding what is best for a group, given the
opinions of individualsStudied extensively in economics, politics,
sociology, and even mathConstruction of a social welfare functionsocial welfare function
Pattanaik, 1971Taylor, 1995
Social choice (2)
Meta-Search Results from multiple search engines are combined
into one list - rank aggregation (Dwork et al., 2001)
Database middleware Objects with numerical values for multiple fields
have to be ordered (Fagin et al., 2003)
Collaborative Filtering Preferences of a group of individuals have to be
aggregated to predict a third party preference (Wallace et al., 2003)
A hands-on scenario
Choose a set of items available in an EPG e.g. video clips, TV series, movies, sports events,
etc.
Group of viewers Receiver knows who the viewers are Has preference ratings for each of them
Problem: which items to show, given time for a certain number but not all of them?
Assumptions are fine, but…
How will the receiver know who is watching? IR card or login procedure registers viewers Probabilistic mechanism: known probability of a
viewer watching at a particular time of day
How will user preferences be determined? Social, and content-based filtering can be used Inferences from group viewing actions when
user watches alone? An individual’s ratings may depend on the group
they are in
Assumptions are fine, but… (2)
How to deal with differences in rating tendencies? Some people either “really hate” or “really love” a
program Others may never be very positive and never really
negative
How to deal with uncertainty? How to deal with changing groups?
Assumptions are fine, but… (3)
How to deal with users feeling strongly about their preferences? All users are equal? Or some are “more equal than others”?
How to present recommendations to the group? Receiver decides which items to show? Receiver gives viewers a choice? List of recommendations?
Example strategies
Plurality VotingEach voter votes for their most preferred
alternativeSequence of alternatives repetitive
method
Example strategies (2)
Utilitarian StrategyUtility values express expected happinessAdditive/Average strategy (Masthoff, 2002)
Weighted form in multi-agent systemsMultiplicative
Individual viewers might always lose out, because their opinion is a minority view
In a small group, individual opinions will have a large impact on the average/product
Example strategies (3)
Borda CountBorda, 1781 (!)Points awarded according to position
preference listbottom alternative gets zero points, the next
one up one point, etc.Multiple alternatives with the same rating?
Example strategies (4)
Copeland RuleForm of majority voting - orders alternatives
according to Copeland indexNumber of times an alternative beats other
alternatives minus the number of times it loses to other alternatives
Approval VotingVoters allowed to vote for as many
alternatives as they wishPromotes election of moderate alternatives
Example strategies (5)
Least Misery StrategyNew list of ratings with minimum of
individual ratingsHigher ranked items get selected soonerA group is as happy as its least happy
memberMinority opinions dictate the group
If everybody wants to see something, but one person does not like it, then it will never be seen.
Example strategies (6)
Most Pleasure StrategyNew list of ratings with maximum of
individual ratingsHigher ranked items get selected sooner
Average Without Misery StrategyNew list of ratings with average of individual
ratingsSkip items that score below a certain
threshold for individuals
Example strategies (7)
Fairness StrategySelect top items from all individualsEqual rating take into account others’
opinions It’s not so bad to watch something you hate,
as long as you get to watch the things you really love as well
Example strategies (8)
Most Respected Person StrategyAlso called “Dictatorship”Use ratings of most respected personSimilar ratings use others’ ratingsTV remote control is often operated by the
oldest male presentAdults may dominate children, visitors may have
more influence than inhabitantsBirthdays or illness can influence who is ‘the
most respected’ person
The issue of satisfaction
A major difference between strategies is the emphasis placed on individual satisfactionavoidance of miserysatisfaction of group majority
Another major difference: use only relative item positions in preference lists, or also strengths of these preferences?
A basic satisfaction function
Input: a set of ratings for any sequence of clips
Output: a real numbere.g. summation of an individual’s ratings of
concerned clipshow do these ratings compare to those of
unselected clips? normalizationLinearity, misery, order, group solidarity?
The holy grail
Is there a strategy that keeps everyone happy?
Multiplicative Utilitarian seems the most promising strategyLength of clips may be importantDiscussion on the basis of the clipsSpecify and definitely include favorite clips
Presenting a Sequence
The receiver will also need to decide in which order to show itemsGroup List RankingChanging OrderChanging Ratings
Open questions
Do people adjust ratings?Having watched the first item influences
ratings for the other items?Mood? Thematic relatedness?Viewer’s satisfaction Interaction between these factors?