Convolutional Neural Networks for Language · Convolutional Neural Networks for Language Features...

transcript

CS6956:DeepLearningforNLP

ConvolutionalNeuralNetworksforLanguage

Featuresfromtext

Example:Sentimentclassification

Thegoal:Isthesentimentofasentencepositive,negativeorneutral?

Thefilmisfunandishosttosometrulyexcellentsequences

Approach:TrainamulticlassclassifierWhatfeatures?

Featuresfromtext

Thefilmis funandishosttosome trulyexcellentsequences

Approach:TrainamulticlassclassifierWhatfeatures?Somewordsandngrams areinformative,whilesomearenot

Featuresfromtext

Thefilmis funandishosttosome trulyexcellentsequences

Approach:TrainamulticlassclassifierWhatfeatures?Somewordsandngrams areinformative,whilesomearenot

Weneedto:1. Identifyinformativelocalinformation2. Aggregateitintoafixedsizevectorrepresentation

ConvolutionalNeuralNetworks

Designedto1. Identifylocalpredictorsinalargerinput

2. Poolthemtogethertocreateafeaturerepresentation

3. Andpossiblyrepeatthisinahierarchicalfashion

IntheNLPcontext,ithelpsidentifypredictivengrams foratask

Overview

• ConvolutionalNeuralNetworks:Abriefhistory

• ThetwooperationsinaCNN– Convolution– Pooling

• Convolution+Poolingasabuildingblock

• CNNsinNLP

• RecurrentnetworksvsConvolutionalnetworks

Overview

• CNNsinNLP

ConvolutionalNeuralNetworks:Briefhistory

• HubelandWiesel,1950s/60s:Mammalianvisualcortexcontainneuronsthatrespondtosmallregionsandspecificpatternsinthevisualfield

• Fukushima1980,Neocognitron:DirectlyinspiredbyHubel,Wiesel– Keyidea:localityoffeaturesinthevisualcortexisimportant,integratethemlocallyand

propagatethemtofurtherlayers– Twooperations:convolutionallayerthatreactstospecificpatternsandadown-sampling

layerthataggregatesinformation

• LeCun 1989-today,ConvolutionalNeuralNetwork:Asupervisedversion– Relatedtoconvolutionkernelsincomputervision– Verysuccessfulonhandwritingrecognitionandothercomputervisiontasks

• Hasbecomebetteroverrecentyearswithmoredata,computation– Krizhevsky etal2012:ObjectdetectionwithImageNet– Thedefactofeatureextractorforcomputervision

Firstaroseinthecontextofvision

NobelPrizeinPhysiologyorMedicine,1981

DavidH.Hubel Torsten Wiesel

propagatethemtofurtherlayers– Twooperations

1. convolutionallayerthatreactstospecificpatternsand,2. adown-samplinglayerthataggregatesinformation

• LeCun 1989-today,ConvolutionalNeuralNetwork:Asupervisedversion– Relatedtoconvolutionkernelsincomputervision– Successwithhandwritingrecognitionandothercomputervisiontasks

• Hasbecomebetteroverrecentyearswithmoredata,computation– Krizhevsky etal2012:ObjectdetectionwithImageNet– Thedefactofeatureextractorforcomputervision

• IntroducedtoNLPbyCollobert etal,2011– Usedasafeatureextractionsystemforsemanticrolelabeling

• Sincethenseveralotherapplicationssuchassentimentanalysis,questionclassification,etc– Kalchbrener etal2014,Kim2014

CNNterminology

• Filter– Afunctionthattransformsininputmatrix/vectorintoascalarfeature– Afilterisalearnedfeaturedetector

• Channel– Incomputervision,colorimageshavered,blueandgreenchannels– Ingeneral,achannelrepresentsamediumthatcapturesinformation

aboutaninputindependentofotherchannels• Forexample,differentkindsofwordembeddings couldbedifferentchannels• Channelscouldthemselvesbeproducedbypreviousconvolutionallayers

• Receptivefield– Theregionoftheinputthatafiltercurrentlyfocuseson

Showsitscomputervisionsandsignalprocessingorigins

CNNterminology

• Filter– Afunctionthattransformsininputmatrix/vectorintoascalarfeature– Afilterisalearnedfeaturedetector(alsocalledafeaturemap)

CNNterminology

• Channel– Incomputervision,colorimageshavered,blueandgreenchannels– Ingeneral,achannelrepresentsa“viewoftheinput”thatcaptures

informationaboutaninputindependentofotherchannels• Forexample,differentkindsofwordembeddings couldbedifferentchannels• Channelscouldthemselvesbeproducedbypreviousconvolutionallayers

Overview

• CNNsinNLP

Whatisaconvolution?

Let’sseethisusinganexampleforvectors.

Wewillgeneralizethistomatricesandbeyond,butthegeneralidearemainsthesame.

Whatisaconvolution?

Anexampleusingvectors

2 3 1 3 2 1Avector𝐱

Whatisaconvolution?

2 3 1 3 2 1

Avector𝐱

Filter 𝐟 ofsize𝑛

Here,thefiltersizeis3

Whatisaconvolution?

2 3 1 3 2 1

Avector𝐱

Filter𝐟 ofsize𝑛

Theoutput isalsoavector

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Whatisaconvolution?

2 3 1 3 2 1

Avector𝐱

Theoutput isalsoavector

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Thefiltermovesacrossthevector.

Ateachposition,theoutputisthedotproductofthefilterwithasliceofthevectorofthatsize.

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

Paddingatthebeginning

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7Theoutput isalsoavector

Paddingatthebeginning

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7 9Theoutput isalsoavector

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7 9 8Theoutput isalsoavector

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7 9 8 9Theoutput isalsoavector

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7 9 8 9 8Theoutput isalsoavector

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

7 9 8 9 8 4Theoutput isalsoavector

Paddingattheend

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

Whatisaconvolution?

2 3 1 3 2 1

output( =*𝑓, ⋅ 𝑥(/ 01 2,

Avector𝐱

Thefiltermovesacrossthevector.

Ateachposition,theoutputisthedotproductofthefilterwithasliceofthevectorofthatsize.

Whatisaconvolution?

Thesameideaappliestomatricesaswell

Aninputmatrix Afilter

Thefiltermovesacrossthematrix.

Ateachposition,theoutputisthedotproductofthefilterwithasliceofthematrix ofthatsize.

Whatisaconvolution?

Aninputmatrix Afilter Theresultofconvolution

Ateachposition,theoutputisthedotproductofthefilterwithasliceofthematrixofthatsize.

Whatisaconvolution?

Andsoon…Thefiltermovesacrossthematrix.

Whatisaconvolution?

Andsoon…Thefiltermovesacrossthematrix.

Whatisaconvolution?

Overview

• CNNsinNLP

Pooling:Anaggregationoperation

• Aconvolutionproducesavector/matrixthatcapturespropertiesofeachwindow

• Poolingcombinesthisinformationtoproduceadown-sampledversionvector/matrix– Typicallyusingthemaximumortheaveragevaluewithinawindow

• Intuition– Afilterisafeaturedetectorthatdiscovershowwelleachwindow

matchesafeatureofinterest– Themostimportantfeaturesshouldberecognizedregardlessoftheir

location– Answer:Pooltheinformationfromdifferentwindowstogether

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Thepoolingoperationcanbeappliedusingawindowaswell

Example1:Maxpoolingwithwindowsize3

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Whatispooling?

2 3 1 3 2 1

Avector𝐱

9 9 9 8

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Example2:Averagepoolingwithwindowsize3

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Whatispooling?

2 3 1 3 2 1

Avector𝐱

8 8.6 8.3

Whatispooling?

2 3 1 3 2 1

Avector𝐱

8 8.6 8.3 7

Whatispooling?

2 3 1 3 2 1

Avector𝐱

Example3:Maxpoolingwithwindowsize=lengthofthevector

Whatispooling?

2 3 1 3 2 1

Avector𝐱

ImportantnoteTherearenolearnedparametersforthepoolingoperation.Itisadeterministicoperation.

Typicalkindsofpooling

• Maxpooling– Takethemaximumvalueoftheresultsoftheconvolution

• Averagepooling– Usesaveragetopoolinsteadofmax

• K-maxpooling– TakethetopKvalues(forafixedk)– Generalizationofmaxpooling

Overview

• CNNsinNLP

Convolution+Pooling=onelayer

• Input:amatrix.Convolutionwilloperateoverwindowsofthismatrix.

Thiscouldbeextendedtogeneraltensorsaswell

• Thewindowsizedefinesthereceptivefield– Wewillrefertothewindowasx5

• Afilterisdefinedbysomeparameters(thatwillbelearned)– Ingeneral,amatrixu ofthesameshapeasathewindowandabiasb

• Convolution:Iterateoverallwindowsandapplythefilter– Typicallyhasanon-linearity(e.g.ReLU)

𝑝( = 𝑔(𝑢 ⋅ 𝑥( + 𝑏)

• Pooling:Aggregatethe𝑝(’sintoadown-sampledversion,sometimesasinglenumber

𝑝( = 𝑔(𝑢 ⋅ 𝑥( + 𝑏)

• Pooling:Aggregatethe𝑝(’sintoadown-sampledversion,sometimesasinglenumber

• Typically,therearemanyfilters,eachofwhicharepooledindependently

Hyperparameters

• Filtersizes:Howbigshouldthefilterbe?– Typically,3x3,5x5,etc

• Stride:howdoesthefiltermovealongtheinput?– Itcouldskipsomesteps,ornot.

• Howmanyfiltersshouldthebe?

• Padding:Shouldtherebepaddingornot?Ifso,shouldthepaddingbezerosorrandom?

• Howbigshouldthepoolingwindowbe?

• Whatkindofpooling:Average,Max,L2norm?

Example:LeNet

Anexamplenetworkusesthesebuildingblock

LeNet-5wasproposedbyLeCun 1998forhandwritingrecognitionHadseverallevelsofconvolution-pooling

Overview

• CNNsinNLP

ConvolutionalNeuralNetworksinNLP

• Goal:Torepresentasequenceofwordsasafeaturevector

• Approach:– Representthesequenceofwordsbysequence(s)ofembeddings– Convolvewithseveralfilters– Poolacrossthesequencetogetafeaturevectorofafixeddimensionality

Iatecaketoday

Supposewewanttoclassifythissentence:

Goal:Torepresentasequenceofwordsasafeaturevector

Wordembeddings

padding

Wordembeddings

padding

Applyafilter

Wordembeddings

padding

Wordembeddings

padding

Wordembeddings

padding

Wordembeddings

padding

Wordembeddings

padding

Convolutionwithonefilter

Wordembeddings

padding

Convolutionwithonefilter

Poolingacrossthesentence(oftenmax

pooling)togetonefeature

Wordembeddings

padding

Convolutionwithmanyfilters

Poolingacrossthesentence(oftenmax

pooling)getsafeaturevector

Therecanbeseveralfilters(sometimescalledkernels,orfeaturemaps)

Convolution+poolingexample

1. Eachwordisembeddedintoa2dvector,thewindowconcatenatesthem

2. A6x3filterwithatanh non-linearity

3. Maxpoolingovereachdimensiontoproducea3dimensionalvector

Examplesofconvolution+pooling

82FigurefromGoldberg2017

Thinkofconvolutionsasfeatureextractors

Anarrowconvolution(i.e.withoutanypadding)inthevectorconcatenationnotation

Awideconvolution(i.e.withpadding)inthevectorstackingnotation

Overview

• CNNsinNLP

Featuresfromtext

• Ifwewanttoclassifytext,weneedtorepresenttheminsomefeaturespace

• Wehave(atleast)twowaystogetfeaturesfromtextusinganeuralnetwork:– RecurrentNeuralNetworks– ConvolutionalNeuralNetworks

RNNsvsCNNs

• RNNsmodelnon-Markoviandependencies– Canlookat(effectively)infinitewindowsaroundatargetword– Cancapturesequentialpatternsinsuchwindows

RNNsvsCNNs

• CNNscaptureinformativengrams– Alsogappy n-grams– Butalsoaccountforlocalorderingpatterns

RNNsvsCNNs

• CNNscaptureinformativengrams– Alsogappy n-grams– Butalsoaccountforlocalorderingpatterns

• Howdotheycompare?– Botharetrainedend-to-endwithataskloss– RNNs(specifically,BiRNNs)aremorepopulartoday…

• … butthiscanchange– CNNsallowformoreparallelism,andsomaybebettersuitedforcertain

hardware/softwareimprovements

RNNsandCNNsasbuildingblocks

ThinkofthemasLegobricksforconstructinglargerarchitectures

BotharecomputationgraphsMixandmatchwithothercomputationgraphstocreatelargerneuralnetworks

Generaltoolsthatcanbeusedwithotherideasthatwehaveseenandwillsee

Eg:contextualembeddings,attention,etc.

Convolutional Neural Networks for Language · Convolutional Neural Networks for Language Features...

Documents