Self-Assemblying Hypernetworks for Cognitive Learning of Linguistic Memory Int. Conf. on Cognitive...

Self-Assemblying Hypernetworks for Self-Assemblying Hypernetworks for Cognitive Cognitive

Learning of Linguistic MemoryLearning of Linguistic Memory

Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6-8, 2008, Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6-8, 2008, Sheraton Hotel, Cairo, EgyptSheraton Hotel, Cairo, Egypt

Byoung-Tak Zhang and Chan-Hoon Park

Biointelligence LaboratorySchool of Computer Science and Engineering

Cognitive Science, Brain Science, and Bioinformatics ProgramsSeoul National University

Seoul 151-744, Korea

[email protected]://bi.snu.ac.kr/

Talk Outline

A Language Game Learning the Linguistic Memory

The Hypernetwork Model of Language

Sentence Recall Experiments Extension to Multimodal Memory Game

(Language + Vision) Conclusion

© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

3

The Language Game PlatformThe Language Game Platform


4

A Language GameA Language Game

? still ? believe ? did this. I still can't believe you did

this.

We ? ? a lot ? gifts. We don't have a lot of gifts.


5

Text Corpus: TV Drama SeriesText Corpus: TV Drama Series

Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City

289,468 Sentences

(Training Data)

700 Sentences with Blanks(Test Data)

I don't know what happened.Take a look at this.…

What ? ? ? here.? have ? visit the ? room.

…


6

Step 1: Learning a Linguistic Step 1: Learning a Linguistic MemoryMemory

Display

Color

Text

News

Monitor

Computer

Network

Price

Computer network is rapidly increased. The price of computer network installing is very cheap. The price of monitor display is on decreasing. Nowadays, color monitor display is so common, the price is not so high. This is a system adopting text mode color display. This is an animation news networks. ...

Price

Computer Network

Computer Price

Computer Display

Computer Monitor Display

News Network

Text News

Monitor Display

Monitor Price

Color Monitor

Color Monitor

Color Display

Text Display

Color

Display

Computer Network Price

Price

Color

Text

k = 2k = 2k = 3k = 4 …

HypernetworkMemory

© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/7

Step 2: Recalling from the Step 2: Recalling from the MemoryMemory

He is my best friend

best friendmy bestis myHe is

He is a ? friend

He is a strong boy

Strong friend likes pretty girl

He is is a a strong strong boy

Strong friend friend likes likes strong pretty girl

Strong friend

a strong

best friend

He is a strong friend

X7

X6

X5

X8

X1

X2

X3

X4

Recall

Self-assembly

Storage


8x8 x9

x12

x1x2

x3

x4

x5

x6

x7x10

x11

x13

x14

x15

x1 =1

x2 =0

x3 =0

x4 =1

x5 =0

x6 =0

x7 =0

x8 =0

x9 =0

x10 =1

x11 =0

x12 =1

x13 =0

x14 =0

x15 =0

y

= 1

x1 =0

x2 =1

x3 =1

x4 =0

x5 =0

x6 =0

x7 =0

x8 =0

x9 =1

x10 =0

x11 =0

x12 =0

x13 =0

x14 =1

x15 =0

y

= 0

x1 =0

x2 =0

x3 =1

x4 =0

x5 =0

x6 =1

x7 =0

x8 =1

x9 =0

x10 =0

x11 =0

x12 =0

x13 =1

x14 =0

x15 =0

y

=14 sentences (with labels)

x4 x10 y=1x1

x4 x12 y=1x1

x10 x12 y=1x4

x3 x9 y=0x2

x3 x14 y=0x2

x9 x14 y=0x3

x6 x8 y=1x3

x6 x13 y=1x3

x8 x13 y=1x6

1

2

3

1

2

3

x1 =0

x2 =0

x3 =0

x4 =0

x5 =0

x6 =0

x7 =0

x8 =1

x9 =0

x10 =0

x11 =1

x12 =0

x13 =0

x14 =0

x15 =1

y

=14

x11 x15 y=0x84

Round 1Round 2Round 3


9

x1x2

x3

x4

x5

x6

x7

x8 x9

x10

x11

x12

x13

x14

x15

The Hypernetwork Memory The Hypernetwork Memory

)( 2121...21

21

21...21

321

321321

21

2121

321

321321

21

2121

2 ,...,,

)()()()(

2 ,...,,

)()()()(

,,

)()()()3(

,

)()()2(

)()(

,,

)()()()3(

,

)()()2()(

...)(

1exp)Z(

isfunction partition thewhere

,...)(

1exp

)Z(

1

...6

1

2

1exp

)Z(

1

)];(exp[)Z(

1 )|(

ondistributiy probabilit The

...6

1

2

1 );(

rkhypernetwo theofenergy The

m kkiiikiii

k

kiiikiii

iiiiiiiiii

iiiiiiiiii

K

k iii

mmmk

K

k iii

nnnk

iii

nnn

ii

nn

nn

iii

nnn

ii

nnn

xxxwkc

W

xxxwkcW

xxxwxxwW

WEW

WP

xxxwxxwWE

x

xx

x

Nn

K

iii

i

D

WWWW

SkXSSS

xxxX

WSXH

I

1)(

)()3()2(

,...,,

}{

:set Training

),...,,(

|| , ,

)(

),,(

as defined isrk hypernetwo The

21

x

[Zhang, DNA12-2006]


10

Molecular Self-Assembly of HypernetworksMolecular Self-Assembly of Hypernetworks

xi xj y

X7

X6

X5

X8

X1

X2

X3

X4

Hypernetwork Representation

x1 x3 Class

x1 x2 x4 Classx2 x3 Class

x1 x4 Class

x1 x3 Class

x1 x3 Class

x1 x2 x4 Class

x1 x2 x4 Class

x2 x3 x4 Class

x2 x3 x4 Class

x2 x3 x4 Class

x2 x3 Class

x2 x3 Class

x1 x4 Class

x1 x4 Class

x1 Class

x2 Class

x1 x2 Class

x1 x3 Class

x1 xn Class…

x1 Class

x1 Class

x2 Class

x1 x2 Class

x1 x2 Class

x1 x3 Class

x1 x3 Class

x1 x3 Class

x1 xn Class…

x2 Class

x2 Class

x1 x3 Class

x1 x3 Class

Molecular Encoding

DNA Computing


11

Experimental SetupExperimental Setup

The order (k) of an hyperedge Range: 2~4 Fixed order for each experiment

The method of creating hyperedges from training data Sliding window method Sequential sampling from the first word

The number of blanks (question marks) in test data Range: 1~4 Maximum: k - 1


12

Learning Behavior Analysis (1/3)Learning Behavior Analysis (1/3)

The performance monotonically increases as the learning corpus grows. The low-order memory performs best for the one-missing-word problem.


13


The medium-order (k=3) memory performs best for the two-missing-words problem.


14


The high-order (k=4) memory performs best for the three-missing-words problem.


15

The Language Game: ResultsThe Language Game: Results

Why ? you ? come ? down ? Why are you go come on down here

? think ? I ? met ? somewhere before I think but I am met him somewhere before

? appreciate it if ? call her by ? ? I appreciate it if you call her by the way

I'm standing ? the ? ? ? cafeteria I'm standing in the one of the cafeteria

Would you ? to meet ? ? Tuesday ? Would you nice to meet you in Tuesday and

? gonna ? upstairs ? ? a shower I'm gonna go upstairs and take a shower

? have ? visit the ? room I have to visit the ladies' room

We ? ? a lot ? gifts We don't have a lot of gifts

? ? don't need your ? If I don't need your help

? ? ? decision to make a decision

? still ? believe ? did this I still can't believe you did this

What ? ? ? here What are you doing here

? you ? first ? of medical school Are you go first day of medical school

? ? a dream about ? In ? I had a dream about you in Copenhagen

Extension to Multimodal Memory Game


ImageImage SoundSound TextText

But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.


Memorizing Images to Retrieve Texts

Memorizing Images to Retrieve Texts

Memorizing Textsto Retrieve Images

Memorizing Textsto Retrieve Images

Hint Hint TextText








Image Image HintHint

Image Generation GameImage Generation GameImage Generation GameImage Generation GameText Generation GameText Generation GameText Generation GameText Generation Game

Machine LearnerMachine Learner

How can it be done?

How can it be done?

Scene1 He is best friend

Scene2 She is strong boy

Scene3 friend likes pretty girl

Scene1 a1 a2 a3 a4

Scene2 b1 b2 b3 b4

Scene3 c1 c2 c3 c4

TextText ImageImage

Text: N sequential samplesImage: N random samples

He is a1 a3 a4

best friend a1 a3 a2

Multimodal HNMultimodal HN

She is b1 b3 b5

is strong b2 b3 b1

He is best friendText QueryText Query

He is a1 a3 a4


is best b2 b3 b1

MatchingMatching

He is best friendText QueryText Query

He is a1 a3 a4


is best b2 b3 b1

MatchingMatching

Generating an Image

Mapa1=2b1=1

b2=1a3=2b3=1

a4=2 voting Map a1 b2 a3 a4

Map a1 b2 a3 a4

Scene1 a1 a2 a3 a4

Scene2 b1 b2 b3 b4

Scene3 c1 c2 c3 c4

ImageImage

Hamming Distance

Scene1 a1 a2 a3 a4Map a1 b2 a3 a4


18

Image-to-Text Crossmodal RecallImage-to-Text Crossmodal RecallText

Learningby Viewing

Image- Where am I giving birth- You guys really don't know anything- So when you guys get in there- I know it's been really hard for you- …

User TextCorpus

Question:

Answer:

Where am I giving birth

Where ? I giving ?


19

Text-to-Image Crossmodal RecallText-to-Image Crossmodal RecallText

Learningby Viewing

Image Corpus User

Question:

Answer:

Image

You've been there

- Where am I giving birth- You guys really don't know anything- So when you guys get in there- I know it's been really hard for you- …


20

The Multimedia (Movie) CorpusThe Multimedia (Movie) Corpus

Dataset: 2 dramas Images and the corresponding scripts Titles

Friends, Prison Break

Training data: 2,808 images and scripts Image size: 80 x 60 = 4800 pixels Vocabulary: 2,579 words

Where am I giving birth

I know it's been really hard for you

So when you guys get in there


21

Experimental SetupExperimental Setup

The order (k) of memory units Text: k = 2, 3, 4 Image: k = 10, …, 340

Constructing hyperedges from training data Text: Sequential sampling from a random position Image: Random sampling from 4,800 pixel positions

The number of repetitive samples from an image-text pair N = 150, …, 300

AnswerQuery

I don't know what happened

There's a kitty in my guitar case

Maybe there's something I can do to make sure I get pregnant

Maybe there's something there's something I … I get pregnant

There's a a kitty in … in my guitar case

I don't know don't know what know what happened

Matching &Completion

Image-to-Text Recall ExamplesImage-to-Text Recall Examples


Query Matching &Completion

I don't know what happened

Take a look at this

There's a kitty in my guitar case

Maybe there's something I can do to make sure I get pregnant

Answer

Text-to-Image Recall ExamplesText-to-Image Recall Examples



24

Image to Text (Recall Rate)Image to Text (Recall Rate)

Note: In the tolerant recall, the generated sentence is evaluated correct if the number of mismatches is within the specified tolerance level (here two words).


25

Text to Image (Recall Rate)Text to Image (Recall Rate)

Note: The retrieved image is evaluated correct if its hamming distance to the target image is the smallest.


26

ConclusionConclusion Hypernetworks are a random graph model employing higher-order edges

and allowing for a more natural representation for learning higher-order interactions.

We introduce a linguistic memory model based on a self-organizing hypernetwork inspired by mental chemistry.

The hypernetwork stores the sentences in random fragments and recalls a sentence by self-assemblying them given a partial, query sentence.

Applied to a sentence corpus of 290K sentences, we obtain a recall performance of 90-100%, depending on the difficulty of the task.

Cognitive plausibility: “Multiple representations of partially overlapping micromodules which are

partially active simultaneously” [Fuster, 2003] Neural microcircuits [Grillner et al, 2006] Cognitive schema or cognitive code [Tse et al., 2007]

Data Acquisition and Experimentation Text: Ha-Young Jang Image: Min-Oh Heo Sun Kim Joo-Kyoung Kim Ho-Sik Seok Kwonil Kim Sang-Yoon LeeSupported by - National Research Lab Program of Min. of Sci. & Tech. (2002-2007)

- Next Generation Tech. Program of Min. of Ind. & Comm. (2000-2010)

- BK21-IT Program of Min. of Education (2006-2009)

- SK Telecom (2007-2008)

More Information at - http://bi.snu.ac.kr/ Research MMG (to be open soon)

Acknowledgements


29

The Hypernetwork Model of The Hypernetwork Model of LearningLearning

Nn

K

iii

i

D

WWWW

SkXSSS

xxxX

WSXH

I

1)(

)()3()2(

,...,,

}{

:set Training

),...,,(

|| , ,

)(

),,(

as defined isrk hypernetwo The

21

x

)( 2121...21

21

21...21

321

321321

21

2121

321

321321

21

2121

2 ,...,,

)()()()(

2 ,...,,

)()()()(

,,

)()()()3(

,

)()()2(

)()(

,,

)()()()3(

,

)()()2()(

...)(

1exp)Z(

isfunction partition thewhere

,...)(

1exp

)Z(

1

...6

1

2

1exp

)Z(

1

)];(exp[)Z(

1 )|(

ondistributiy probabilit The

...6

1

2

1 );(

rkhypernetwo theofenergy The

m kkiiikiii

k

kiiikiii

iiiiiiiiii

iiiiiiiiii

K

k iii

mmmk

K

k iii

nnnk

iii

nnn

ii

nn

nn

iii

nnn

ii

nnn

xxxwkc

W

xxxwkcW

xxxwxxwW

WEW

WP

xxxwxxwWE

x

xx

x

[Zhang, 2006]


30

Deriving the Learning RuleDeriving the Learning Rule

N

n

K

k iii

nnnk

N

n

Kn

Nn

WZxxxwkc

WWWP

WP

k

kiiikiii1 2 ,...,,

)()()()(

1

)()3()2()(

1)(

)(ln...)(

1exp

),...,,|(ln

)|}({ln

21

21...21

x

x

)|}({ln 1)(

)(

...21

WPw

Nns

siii

x

N

n

Nn

WP

WP

1

(n)

1)(

)|(

)|}({

x

x


31

Derivation of the Learning RuleDerivation of the Learning Rule

xx

x

x

x

x

)|(......

...1

...

where

......

......

)(ln...)(

1exp

)(ln...)(

1exp

)|}({ln

2121

2121

2121

2121

...2121

21...21

...21

21

21...21

...21

...21

)|(

1

)()()(

)|(

1)|(

)()()(

1)(

2 ,...,,

)()()()()(

1 2 ,...,,

)()()()()(

1)(

)(

WPxxxxxx

xxxN

xxx

xxxxxxN

xxxxxx

WZw

xxxwkcw

WZxxxwkcw

WPw

siiis

siiis

ss

ssiii

siiik

kiiikiii

siii

k

kiiikiii

siii

siii

WPiii

N

n

nnn

Dataiii

WPiiiDataiii

N

nWPiii

nnn

N

ns

K

k iii

nnnks

N

n

K

k iii

nnnks

Nns

Molecular Self-AssemblyMolecular Self-Assembly


33

Encoding a Hypernetwork with Encoding a Hypernetwork with DNADNA

z1 :

z2 :

z3 :

z4 :

b)

x1

x2

x3

x4

x5

y

0

1

where

z1 : (x1=0, x2=1, x3=0, y=1)z2 : (x1=0, x2=0, x3=1, x4=0, x5=0, y=0)z3 : (x2=1, x4=1, y=1)z4 : (x2=1, x3=0, x4=1, y=0)

a)

AAAACCAATTGGAAGGCCATGCGG

AAAACCAATTCCAAGGGGCCTTCCCCAACCATGCCC

AATTGGCCTTGGATGCGG

AATTGGAAGGCCCCTTGGATGCCC

GG

AAAA

AATT

AAGG

CCTT

CCAA

ATGC

CC

Collection of (labeled) hyperedges

Library of DNA molecules corresponding to (a)


34

DNA Molecular ComputingDNA Molecular Computing

Self-assembly

Heat

Cool

Polymer

Repeat

Self-replication

Molecular recognitionNanostructure

Date post:	20-Jan-2016
Category:	Documents
Upload:	lionel-dalton
View:	216 times
Download:	0 times

Self-Assemblying Hypernetworks for Cognitive Learning of Linguistic Memory Int. Conf. on Cognitive...

Documents