Music Technology Research: Scientific and Artistic Challenges · 2015. 7. 28. · (Lesaffre et...

1

Cosmo CaixaBarcelona, Nov. 16th 2007

Music Technology Research: Scientific and Artistic Challenges

Xavier SerraMusic Technology Group (http://mtg.upf.edu)

Universitat Pompeu Fabra, Barcelona

Cosmo CaixaBarcelona, Nov.16th 2007

Introduction

2


Music &Technology: Long time friendship

Interior view of a pipe organ (Dom Bedos, 1776)


Music communication framework

Symbolic representation

Temporal controls

Source sound

Sound field

PerceptionCognition

Listener

Composer

Performer

InstrumentRoom

“Musical” Knowledge base

“Physical” Knowledge base (Moorer, 1990)

3


Music technology research disciplines

Music

Psychology

PhysicsEngineering

MusicTechnology

TheoryPerformance CompositionComputer

Science

ArtificialIntelligence Psychomusicology

Cognition

Psychoacoustics

Acoustics

Device design

Digitalsignal

processing

Digitalhardware

Programming

(Moorer, 1990)


Research trends

4


Sound Synthesis

Current trends:

Audio content processing

Physical modeling based on measurements

Sound-gesture modelingTraditional approach to sound synthesis


Sound synthesis trend: Content processing

Vocaloid(Yamaha & MTG-UPF, 2005)

5


Sound synthesis trend: Physical modeling from measurements

Measuring impulse responses from a violin

Various sensors on a violin


Sound synthesis trend: Sound-gesture modeling

Finger position

Playing state estimator

String

Bow- bridge distance

Bow pressing force

Bow acceleration

Bow velocity

Bow position

Audio

Performed scores

Violin synthesisPachelbel Canon

6


Sound Description and Understanding

(Lesaffre et alt., 2003)

Current trends:

Search and retrieval

Combining bottom-up and top-down approaches


Sound description trend: Search and retrieval

Harmonic analysis (Gomez, 2006)

MusicSurfer

7


Sound description trend: Bottom-up and top-down

Music recommendation system (Celma, 2006)


Sound/Music Interaction

Mathews with the Radio Drum

Current trends:

Understanding performance expression

New musical interfaces

8


Sound/music interaction trend: Understanding performance

High-qualityrecordings

analyze

encode

extract

obtain

Analyze

Machinerepresentation

Structure of pieces

SymbolicdescriptionExpressive

aspects ofrecordings

Machinelearning

Models

Synthesizedscore

Automatic expression


Sound/music interaction trend: New interfaces

Reactable (Jordà, 2006)

9


Scientific challenges


Challenge: Understanding sounding objects

String vibration Membrane vibration

10


Challenge: Understanding perception

Diagram of inner ear


Challenge: Understanding cognition

(Peretz and Coltheart, 2003)

11


Musical challenges


Challenge: Bridging the semantic gap

Signalfeatures

ContentObjects

HumanKnowledge

semanticgap

Audio(music recordings)

Text(lyrics, editorial text,

press releases, …)

Image(video clips, CD covers,

printed scores, …)

frequency

duration

spectrum

intensity

pitchtimbreloudness time

melody

harmony

rhythmsource

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes

texturescontrastsverbs

sentences

articles

numbers

semanticfeatures

Music information plane

12


Signalfeatures

ContentObjects

HumanKnowledge

semanticgap






frequency

duration

spectrum

intensity


melody

harmony

rhythmsource

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes


sentences

articles

numbers

semanticfeatures


Signalprocessing

Machinelearning

Webmining

MusictheoryStatistical

modeling



Signalfeatures

ContentObjects

HumanKnowledge

semanticgap






frequency

duration

spectrum

intensity


melody

harmony

rhythmsource

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes


sentences

articles

numbers

semanticfeatures


Signalprocessing

Machinelearning

Webmining

MusictheoryStatistical

modeling

Computationalneuroscience

Multimodalprocessing

Musiccognition

Ontologies

Reasoningrules

Computationalmusicology

Textunderstanding


13


Challenge: Understanding music making

(Leman, 2007)


Challenge: Understanding communication

Score A

Score B

Score C

Score D

Performer A

Performer B

Performer C

Performer D

Instrument A

Instrument B

Instrument C

Instrument D

Audience indiv. 1

COMPOSITION

Audience indiv. 2

Audience indiv. 3

Compositional channel: musical message + role in the performance (solo, accompanist, etc…)

Sonic channel

Instrumental channel: sound-producing and modifying movements / actions, haptic feedback

Visual channel

AUDIENCE

PERFORMANCE

14


Challenge: Understanding social interaction

http://freesound.iua.upf.edu

Cosmo CaixaBarcelona, Nov. 16th 2007

Music Technology Research: Scientific and Artistic Challenges

Xavier SerraMusic Technology Group (http://mtg.upf.edu)

Universitat Pompeu Fabra, Barcelona

15


Thanks !!!

Date post:	18-Aug-2021
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

Music Technology Research: Scientific and Artistic Challenges · 2015. 7. 28. · (Lesaffre et...

Documents