Cognitive Learning and the Multimodal Memory Game: Cognitive Learning and the Multimodal Memory Game: Toward HumanToward Human--Level Machine LearningLevel Machine LearningToward HumanToward Human Level Machine LearningLevel Machine Learning
2008 IEEE World Congress on Computational Intelligence (WCCI 2008)2008 IEEE World Congress on Computational Intelligence (WCCI 2008)2008 IEEE World Congress on Computational Intelligence (WCCI 2008)2008 IEEE World Congress on Computational Intelligence (WCCI 2008)Cognitive Architectures: Towards HumanCognitive Architectures: Towards Human--Level Intelligence SessionLevel Intelligence Session
June 5, 2008, Hong KongJune 5, 2008, Hong Kong
Byoung-Tak Zhang
Biointelligence LaboratorySchool of Computer Science and Engineering
Cognitive Science, Brain Science, and Bioinformatics ProgramsSeoul National UniversitySeoul National University
Seoul 151-744, Korea
[email protected]@http://bi.snu.ac.kr/
T lk O tliT lk O tliTalk OutlineTalk OutlineHuman-level machine learning is a prerequisite to achieving human-level machine intelligence.♦ Differences of behaviors in humans and machines
What principles are underlying the cognitive learning and memoryin humans?♦ A proposal for three principles♦ A proposal for three principles
What tasks are challenging enough to study human-level machine learning?♦ A proposal for the multimodal memory game (MMG)♦ A proposal for the multimodal memory game (MMG)
Some illustrative results♦ Linguistic memory♦ Language vision translation♦ Language-vision translation
Future directions♦ Toward human-level machine intelligence
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
2
Cognitive LearningCognitive Learning
H d M hiH d M hiHumans and MachinesHumans and Machines
Humans are ♦ creative
Humans are ♦ imprecise, ♦ creative,
♦ compliant, ♦ attentive to change
p ,♦ sloppy, ♦ distractable, ♦ attentive to change,
♦ resourceful, and ♦multipurpose
,♦ emotional, and ♦ illogical♦multipurpose
To achieve human level intelligence these
♦ illogical
To achieve human-level intelligence theseproperties should be taken into account.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
4
T d HT d H L l I lli L l I lli Toward HumanToward Human--Level Intelligence Level Intelligence
Human intelligence develops situated in a multimodal environment [Gibbs, 2005].The human mind makes use ofThe human mind makes use of multiple representations and problem-solving strategies [Fuster, 2003]. The brain consists of functionalThe brain consists of functional modules which are localized in subcortical areas but work togetheron the whole-brain scale [Grillner et al., 2006].al., 2006]. Humans can integrate the multiple tasks into a coherent solution [Jones, 2004]. Humans are versatile and come upHumans are versatile and come up with many new ideas and solutions to a given problem [Minsky, 2006].
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
5
Learning and Memory as a Substrate for Learning and Memory as a Substrate for Learning and Memory as a Substrate for Learning and Memory as a Substrate for IntelligenceIntelligence
It is our memory that enables us to value everything elsewe possess Lacking memory we would have no abilityIt is our memory that enables us to value everything elsewe possess Lacking memory we would have no abilitywe possess. Lacking memory, we would have no abilityto be concerned about our hearts, achievements, lovedones, and incomes. Our brain has an amazing capacity to
we possess. Lacking memory, we would have no abilityto be concerned about our hearts, achievements, lovedones, and incomes. Our brain has an amazing capacity too es, a d co es. Ou b a as a a a g capac ty tointegrate the combined effects of our past experiencestogether with our present experiences in creating our
o es, a d co es. Ou b a as a a a g capac ty tointegrate the combined effects of our past experiencestogether with our present experiences in creating ourthought and actions. This is all possible by the memoryand the memories are formed by the learning process.thought and actions. This is all possible by the memoryand the memories are formed by the learning process.
McGaugh, J. L. Memory & Emotion: The Making of Lasting Memories, 2003.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
6
P i i l f L i E l Id P i i l f L i E l Id Principles of Learning: Early Ideas Principles of Learning: Early Ideas
Aristotle: Three Laws of Association [Crowder, 1976]♦ Similarity♦ C t t♦ Contrast♦ Contiguity
James Mill (1773-1836): Strength Criteria ofJames Mill (1773 1836): Strength Criteria of Association♦ Permanence♦ C t i t♦ Certainty♦ Spontaneity♦ “Mental Compounding”p g
John Stuart Mill (1806-1873)♦ “Mental Chemistry”
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
7
P i i l f L i M d CP i i l f L i M d CPrinciples of Learning: Modern ConceptsPrinciples of Learning: Modern Concepts
Types of learning: Accretion, tuning, restructuring (e.g., R lh & NRumelhart & Norman, 1976)Encoding specificity
i i l (T l i 1970’ )principle (Tulving, 1970’s)Cellular and molecular basis of learning and
(K d l t lmemory (Kandel et al., 1990’s) Conceptual blend and chemical scramble (e gchemical scramble (e.g., Feldman, 2006)
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
8
M th d f M hi L iMethods of Machine Learning
Symbolic Learning♦ Version Space Learning
Probabilistic Learning♦ Bayesian Networks
H l h l M hi♦ Case-Based LearningNeural Learning♦ M ltil P t
♦ Helmholtz Machines♦ Latent Variable Models♦ Generative Topographic ♦ Multilayer Perceptrons
♦ Self-Organizing Maps ♦ Support Vector Machines
p g pMapping
Other Machine Learning Methodspp
Evolutionary Learning♦ Evolution Strategies
Methods♦ Decision Trees♦ Reinforcement Learning
♦ Evolutionary Programming♦ Genetic Algorithms♦ Genetic Programming
♦ Boosting Algorithms♦ Kernel Methods♦ Independent Component♦ Genetic Programming ♦ Independent Component
Analysis
Three Fundamental Principles of Cognitive Three Fundamental Principles of Cognitive Three Fundamental Principles of Cognitive Three Fundamental Principles of Cognitive Learning: Our ProposalLearning: Our Proposal
Continuity. Learning is a continuous, lifelong process. “The experiences of each immediately past moment are memories that merge with currentmemories that merge with current momentary experiences to create the impression of seamless continuity in our lives” [McGaugh, 2003]Glocality. “Perception is dependent on context” and it is important to maintain both global and local, i.e. glocal representations [Peterson andglocal, representations [Peterson and Rhodes, 2003]Compositionality. “The brain activates existing metaphorical g pstructures to form a conceptual blend, consisting of all the metaphors linked together” [Feldman, 2006]
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
10
Research Platform for Cognitive Research Platform for Cognitive Research Platform for Cognitive Research Platform for Cognitive LearningLearning
Toward HumanToward Human--Level Machine Learning: Level Machine Learning: Multimodal Memory Game (MMG)Multimodal Memory Game (MMG)
But, I'm getting married tomorrowBut, I'm getting married tomorrowBut, I'm getting married tomorrowBut, I'm getting married tomorrowBut, I'm getting married tomorrowBut, I'm getting married tomorrowBut, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.
But, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.But, I'm getting married tomorrowW ll b IBut, I'm getting married tomorrowW ll b IBut, I'm getting married tomorrowW ll b IBut, I'm getting married tomorrowW ll b I
But, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.
But, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.
But, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.
But, I m getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.g g g pAre you thinking about me?But if you are, call me tonight.
g g g pAre you thinking about me?But if you are, call me tonight.
Well, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?
Well, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?
Well, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?
Well, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?
g g g pAre you thinking about me?But if you are, call me tonight.
g g g pAre you thinking about me?But if you are, call me tonight.
g g g pAre you thinking about me?But if you are, call me tonight.
g g g pAre you thinking about me?But if you are, call me tonight.
Are you thinking about me?But if you are, call me tonight.Are you thinking about me?But if you are, call me tonight.Are you thinking about me?But if you are, call me tonight.Are you thinking about me?But if you are, call me tonight.
Image Sound Text
Text HintHint Image
© 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Image-to-Text Generator Text-to-Image GeneratorMachine Learner
T G i G (f I )T G i G (f I )Text Generation Game (from Image)Text Generation Game (from Image)
Image SoundSound Text
L iLearningby ViewingT
I2T T2IGameManager
Text HintT
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
13
I G i G (f T )I G i G (f T )Image Generation Game (from Text)Image Generation Game (from Text)
TextImage SoundSound
L iLearningby ViewingI
I2T T2IGameManager
Hint ImageI
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
14
Some Experimental ResultsSome Experimental Results
Th E iTh E iThree ExperimentsThree Experiments
Sentence Generation♦ Learn: a linguistic recall memory from a sentence corpus♦ Gi ti l t t♦ Given: a partial or corrupt sentence♦ Generate: a complete sentence
Image-to-Text Translationg♦ Learn: an image-text joint model from an image-text pair corpus♦ Given: an image (scene)♦ G t t t (di l f th )♦ Generate: a text (dialogue of the scene)
Text-to-Image Translation♦ Learn: an image-text joint model from an image-text pair corpusg j g p p♦ Given: a text (dialogue)♦ Generate: an image (scene of the dialogue)
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
16
E i 1 L i Li i i ME i 1 L i Li i i MExperiment 1: Learning Linguistic MemoryExperiment 1: Learning Linguistic Memory
Dataset: scripts from dramas♦ Friends♦ House♦ 24♦ Grey Anatomy♦ Grey Anatomy ♦ Gilmore Girls ♦ Sex and the City
Training data: 289,468 sentences Test data: 700 sentences with bl kblanksVocabulary size: 34,219 words
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
17
S t C l ti R ltS t C l ti R ltSentence Completion ResultsSentence Completion Results
? gonna ? upstairs ? ? a shower I'm gonna go upstairs and take a shower
We ? ? a lot ? giftsWe don't have a lot of gifts
d d? have ? visit the ? roomI have to visit the ladies' room
? ? don't need your ?if I don't need your help
? still ? believe ? did thisI still can't believe you did this
? ? a dream about ? In ?I had a dream about you in Copenhagen
? i i if ? ll h b ? ?
? ? ? decisionto make a decision
I still can't believe you did this
What ? ? ? hereWhat are you doing here
? ? fi ? f di l h l
I had a dream about you in Copenhagen
? appreciate it if ? call her by ? ? I appreciate it if you call her by the way
I'm standing ? the ? ? ? cafeteria I' t di i th f th f t i
Would you ? to meet ? ? Tuesday ? W ld i t t i T d d
? you ? first ? of medical schoolAre you go first day of medical school
Why ? you ? come ? down ? Why are you go come on down here
? think ? I ? met ? somewhere beforeI think but I am met him somewhere before
I'm standing in the one of the cafeteriaWould you nice to meet you in Tuesday and
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
18
Experiments 2 & 3: Crossmodal Experiments 2 & 3: Crossmodal Experiments 2 & 3: Crossmodal Experiments 2 & 3: Crossmodal TranslationTranslation
D t t d di
The order (k) of hyperedge♦ Text: Order 2~4
Dataset: scenes and corresponding scripts from two dramas♦ Friends
♦ Image: Order 10~340The method of creating hyperedges from training data
♦ Prison Break
Training data: 2,808 scenes and scripts
g♦ Text: Sequential sampling from a
randomly selected position♦ Image: Random sampling in 4,800 p
Scene (image) size: 80 x 60 = 4800 binary pixelsV b l i 2 579 d
g p gpixel positions
Number of samples from an image-text pair
Vocabulary size: 2,579 words
Where am I giving birth
p♦ From 150 to 300
Where am I giving birth
I know it's been really hard for you
So when you guys get in there
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
19
So when you guys get in there
ImageImage--toto--Text Translation ResultsText Translation Results
Matching &
ImageImage toto Text Translation ResultsText Translation Results
AnswerQuery
I don't know
gCompletion
I don't know what happeneddon't know whatknow what happened
There's a kitty in my guitar case
There's aa kitty in
case…in my guitar case
Maybe there's something I can do to make sure I get pregnant
Maybe there's somethingthere's something I
…pregnantI get pregnant
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
TextText--toto--Image Translation ResultsImage Translation Results
Matching &
TextText toto Image Translation ResultsImage Translation Results
Query Matching &Completion Answer
I don't know what happened
Take a look at this
There's a kitty in my guitar case
Maybe there's something I can d t k I t tdo to make sure I get pregnant
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Hypernetwork Architecture for Hypernetwork Architecture for Hypernetwork Architecture for Hypernetwork Architecture for Cognitive LearningCognitive Learningg gg g
x1=1
x2=0
x3=0
x4=1
x5=0
x6=0
x7 =0
x8=0
x9=0
x10=1
x11=0
x12=1
x13=0
x14=0
x15=0
y= 1
x1=0
x2=1
x3=1
x4=0
x5=0
x6=0
x7 =0
x8=0
x9=1
x10=0
x11=0
x12=0
x13=0
x14=1
x15=0
y= 0
1
2
Learningx1=0
x2=0
x3=1
x4=0
x5=0
x6=1
x7 =0
x8=1
x9=0
x10=0
x11=0
x12=0
x13=1
x14=0
x15=0
y=1
4 Data Items3
x1=0
x2=0
x3=0
x4=0
x5=0
x6=0
x7 =0
x8=1
x9=0
x10=0
x11=1
x12=0
x13=0
x14=0
x15=1
y=14
x1x2 x15
x4 x10 y=1x1Round 1Round 2Round 3
x3 x144 10 y1
x4 x12 y=1x1
x10 x12 y=1x4
1
x12
x4 x13x3 x9 y=0x2
x3 x14 y=0x22x12
x5x9 x14 y=0x3
x6 x8 y=1x3
x6 x11x6 x13 y=1x3
x8 x13 y=1x6
3
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
23x8 x9
x7x10x11 x15 y=0x84
k f D A l lHypernetwork of DNA Molecules
[Zhang, DNA-2006]
Hypernetwork as Chemical Associative Hypernetwork as Chemical Associative Hypernetwork as Chemical Associative Hypernetwork as Chemical Associative MemoryMemory
WSXH ),,( asdefined isrk hypernetwoThe
= ∑∑ −−−= )()()()3()()()2()( ...61
21);(
rkhypernetwo theofenergy The
iiiiiiiiii
nnnnnn xxxwxxwWE x
K
iii
i SkXSSSxxxX I
)()3()2(
,...,,
|| , , )( 21
=⊆=
=
∑
∑∑
−=
321321321
212121
)()(
,,,
)];(exp[1)|(
ondistributiy probabilit The62
)(iiiiiiiiii
nn
iiiii
WEWP xx β
Nn
K
D
WWWW
1)(
)()3()2(
}{ :set Training
),...,,(
x=
=
∑∑ ⎥⎦
⎤⎢⎣
⎡++=
321321321
212121
,,
)()()()3(
,
)()()2( ...61
21exp
)Z(1
)];(exp[)Z(
)|(
iiiiiiiiiiiii
nnn
ii
nn xxxwxxwW
WEW
WP xx β
1}{
∑ ∑ ⎥⎦
⎤⎢⎣
⎡=
⎦⎣
= 2121...21
32121
2 ,...,,
)()()()(
,,,
,...)(
1exp)Z(
1
)(
kkiiikiii
K
k iii
nnnk
iiiii
xxxwkcW
∑ ⎥⎦
⎤⎢⎣
⎡∑ ∑==)( 21
21...212 ,...,,
)()()()( ...)(
1exp)Z(
isfunction partition thewhere
m kkiiikiii
K
k iii
mmmk xxxwkc
Wx
[Zhang, DNA-2006]
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
25For more details: Zhang, B.-T., IEEE Computational IntelligenceMagazine, August 2008 (in press)
I t T t (R ll R t )I t T t (R ll R t )Image to Text (Recall Rate)Image to Text (Recall Rate)
0 9
1
Recall Rate
0.7
0.8
0.9
0 4
0.5
0.6
Rat
e
0.2
0.3
0.4
Perfect Recall
0
0.1
10 40 70 100 130 160 190 220 250 280 310 340
Tolerant Recall
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
26Image Order
T t t I (R ll R t )T t t I (R ll R t )Text to Image (Recall Rate)Text to Image (Recall Rate)
1
Recall Rate
0.7
0.8
0.9
0.4
0.5
0.6
Rat
e
0.1
0.2
0.3
0
0.1
2 3 4 5 6
Text Order
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
27
Text Order
Toward HumanToward Human--Level IntelligenceLevel Intelligence
F Mi d M l l d B kF Mi d M l l d B kFrom Mind to Molecules and BackFrom Mind to Molecules and BackMind
BrainBrain
Cell
Molecule∞ memory
Molecule1011 cells
>103 molecules
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
29
P di f C t ti l I t lliP di f C t ti l I t lliParadigms for Computational IntelligenceParadigms for Computational Intelligence
Symbolism Connectionism Dynamicism Hyperinter-actionism
Metaphor symbolt
neuralt dynamical system biomolecular
tp system system y y systemMechanism logical electrical mechanical chemical
Description syntactic functional behavioral relational
Representation localist distributed continuous collective
Organization structural connectionist differential combinatorial
Adaptation substitution tuning rate change self-assembly
Processing sequential parallel dynamical massively parallel
Structure procedure network equation hypergraphStructure procedure network equation hypergraph
Mathematics logic, formallanguage
linear algebra,statistics geometry, calculus graph theory,
probabilistic logicS /ti f l ti l t l ti t l
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
30
Space/time formal spatial temporal spatiotemporal
[Zhang, IEEE Comp. Intel. Mag., August 2008]
S d C l iS d C l iSummary and ConclusionSummary and ConclusionWe argue that understanding and implementing the principles of cognitive learning and memory is a prerequisite to achieving human-level intelligence.gSuggested three principles as the most fundamental to cognitive learning. ♦ Continuity, glocality, compositionality
Proposed the multimodal memory game (MMG) as a research platform forProposed the multimodal memory game (MMG) as a research platform for studying the architectures and algorithms for cognitive learning.Presented the hypernetwork model as a cognitive architecture for learning i MMG i tin an MMG environment.Showed some experimental results to illustrate the usefulness of the platform.♦ Linguistic recall memory or sentence completion♦ Language-vision crossmodal translation tasks
Future work can extend the experimental setups in various dimensions, such
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
31
p p ,as corpus size, kinds of modality, and learning strategies.
D i ti f th L i R lDerivation of the Learning Rule
⎪⎫⎪⎧ ⎤⎡
∂∂ x )|}({ln
...21
1)(
)( WPw
siii
N K
Nns
∑ ∑ ∑
∑ ∑ ∑
⎪⎬⎫⎪
⎨⎧ ∂
⎥⎤
⎢⎡∂
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
−⎥⎦
⎤⎢⎣
⎡
∂∂
== =
)(l1
)(ln...)(
1exp 21
21...21...21
)()()()(
1 2 ,...,,
)()()()()(
WZ
WZxxxwkcw
kkiiikiii
siii
N Knnnk
N
n
K
k iii
nnnks
{ }∑
∑ ∑ ∑
−=
⎪⎭
⎪⎬
⎪⎩
⎪⎨ ∂
∂−⎥
⎦⎢⎣∂
∂=
= =
x......
)(ln...)(
1exp
2121
...212121...21
...21
)|(
)()()(
1)(
2 ,...,,
)()()()()(
xxxxxx
WZw
xxxwkcw
ssiii
siiikkiiikiii
siii
N
WPiiinnn
ns
k iii
nnnks
{ }{ }−=
=
x
x
where
...... 2121
21
)|(
1)|(
xxxxxxNss
s
WPiiiDataiii
nWP
[ ]
[ ]∑
∑
=
==
x )|(
...1...2121
1
)()()(
WPxxxxxx
xxxN
xxxsiiis
N
n
nnn
Dataiii
[ ]∑=x
xx )|(......
2121 )|(WPxxxxxx
siiis WPiii
C l i f O Mi i W d (1/3)C l i f O Mi i W d (1/3)Completion of One Missing Word (1/3)Completion of One Missing Word (1/3)
700
800Sentences with One Missing Words Completion
500
600
200
300
400
0
100
200Order 2Order 3Order 4
40K 80K 120K 160K 200K 240K 280K 290K
The number of completion increases while the number of training
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
34sentences become larger.
C l i f T Mi i W d (2/3)C l i f T Mi i W d (2/3)Completion of Two Missing Words (2/3)Completion of Two Missing Words (2/3)
700
800Sentences with Two Missing Words Completion
500
600
200
300
400
0
100
200Order 2Order 3Order 4
40K 80K 120K 160K 200K 240K 280K 290K
The number of completions increases until the number of missing
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
35words equals the order – 1. (ex) Orders 3 and 4
C l i f Th Mi i W d (3/3)C l i f Th Mi i W d (3/3)Completion of Three Missing Words (3/3)Completion of Three Missing Words (3/3)
700
800Sentences with Three Missing Words Completion
500
600
200
300
400
0
100
200Order 2Order 3Order 4
40K 80K 120K 160K 200K 240K 280K 290K
The number of completions rapidly decreases if the number of
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
36missing words becomes larger than the order. (ex) Orders 2 and 3
Multimodal Memory Game as a Platform Multimodal Memory Game as a Platform Multimodal Memory Game as a Platform Multimodal Memory Game as a Platform for Cognitive Machine Learningfor Cognitive Machine Learning
Image SoundSound Text
LearningI2T Learningby Viewing T2I
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
37
Ill t ti R ltIll t ti R ltIllustrative ResultsIllustrative Results
Query Completion Classification
who are youwho are you
? are youwho ? you
what are youwho are you
FriendsFriends
who are ? who are you Friends
you need to wear it
? need to wear it you ? to wear ityou need ? wear it
I need to wear ityou want to wear ityou need to wear it
242424you need ? wear it
you need to ? ityou need to wear ?
you need to wear ityou need to do ityou need to wear a
24House24
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
38
I t T t T l tiI t T t T l tiI
Image to Text TranslationImage to Text TranslationImage Text
- Where am I giving birth- You guys really don't know
thiL i anything- So when you guys get in there
- I know it's been really hardf
Learningby Viewing
for you- …
Question:Where ? I giving ?
User Answer: TextUser
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
39Where am I giving birth Corpus
T t t I T l tiT t t I T l tiI
Text to Image TranslationText to Image Translation
Text- Where am I giving birth- You guys really don't know
thiL i
Image
anything- So when you guys get in there
- I know it's been really hardf
Learningby Viewing
for you- …
Q tiQuestion:You've been there
UserImage Corpus
Answer:
UserImage Corpus
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
40