CS388:NaturalLanguageProcessingLecture1:Introduc9on

GregDurre<

Administrivia

‣ Coursewebsite:h<p://www.cs.utexas.edu/~gdurre</courses/sp2021/cs388.shtml

‣ Piazza:linkonthecoursewebsite

‣ Myofficehours:Officehours:Tuesday1pm-2pm,Wednesday3:30pm-4:30pm

‣ TA:XiYe.SeecoursewebsiteforOHs

‣ Lecture:TuesdaysandThursdays9:30am-10:45am

‣ Note:myOHstodayare12:30pm-1:30pm

‣ Gradescope:youshould’vego<enanemail

CourseRequirements

‣ 391LMachineLearning(orequivalent)

‣ 311or311HDiscreteMathforComputerScience(orequivalent)

‣ Addi9onalpriorexposuretoprobability,linearalgebra,op9miza9on,linguis9cs,andNLPusefulbutnotrequired

‣ Pythonexperience

‣ Mini1isoutnow(dueJanuary28),pleaselookatitsoon

‣ Ifthisseemslikeit’llbechallengingforyou,comeandtalktome(thisissmaller-scalethantheprojects,whicharesmaller-scalethanthefinalproject)

What’sthegoalofNLP?‣ Beabletosolveproblemsthatrequiredeepunderstandingoftext

‣ Example:dialoguesystemsSiri,what’syourfavoritekindof

movie?

Ilikesuperheromovies!

What’scomeoutrecently?

TheAvengers

Ques9onAnsweringWhenwasAbrahamLincolnborn?

February12,1809

Name Birthday

Lincoln,Abraham 2/12/1809Washington,George 2/22/1732

Adams,John 10/30/1735

Theparkhasatotaloffivevisitorcenters

five

HowmanyvisitorscentersarethereinRockyMountainNa9onalPark?

maptoBirthdayfield

MachineTransla9on

中共中央政治局7⽉30⽇召开会议，会议分析研究当前经济形势，部署下半年经济⼯作。

ThePoli9calBureauoftheCPCCentralCommi<eeheldamee9ngonJuly30toanalyzeandstudythecurrenteconomicsitua9onandplaneconomicworkinthesecondhalfoftheyear.

People’sDaily,August10,2020

ThePoli9calBureauoftheCPCCentralCommi<ee July30 holdamee9ng

Translate

Automa9cSummariza9on

…

…

OneofNewAmerica’swriterspostedastatementcri9calofGoogle.EricSchmidt,Google’sCEO,wasdispleased.

Thewriterandhisteamweredismissed.

providemissingcontext

paraphrasetoprovideclarity

compresstext

NLPAnalysisPipeline

Syntac9cparses

Coreferenceresolu9on

En9tydisambigua9on

Discourseanalysis

Summarize

Extractinforma9on

Answerques9ons

Iden9fysen9ment

‣ NLPisaboutbuildingthesepieces!Translate

TextAnalysis ApplicaDonsText AnnotaDons

‣ Allofthesecomponentsaremodeledwithsta9s9calapproachestrainedwithmachinelearning

Howdowerepresentlanguage?Labels

Sequences/tags

Trees

Text

themoviewasgood +Beyoncéhadoneofthebestvideosofall6me subjecDve

TomCruisestarsinthenewMissionImpossiblefilmPERSON WORK_OF_ART

Ieatcakewithicing

PPNP

SNP

VP

VBZ NNflightstoMiami

λx.flight(x)∧dest(x)=Miami

Howdoweusetheserepresenta9ons?

Labels

SequencesTrees

TextAnalysisText

‣ Mainques9on:Whatrepresenta9onsdoweneedforlanguage?Whatdowewanttoknowaboutit?

‣ Boilsdownto:whatambigui9esdoweneedtoresolve?

…

ApplicaDons

Treetransducers(formachinetransla9on)

Extractsyntac9cfeatures

Tree-structuredneuralnetworks

end-to-endmodels …

Whyislanguagehard?(andhowcanwehandlethat?)

LanguageisAmbiguous!

‣ HectorLevesque(2011):“Winogradschemachallenge”(namedaoerTerryWinograd,thecreatorofSHRDLU)

Thecitycouncilrefusedthedemonstratorsapermitbecausethey______violence

‣ >5datasetsinthelasttwoyearsexaminingthisproblemandcommonsensereasoning

‣ Referen9alambiguity

Thecitycouncilrefusedthedemonstratorsapermitbecausetheyadvocatedviolence

Thecitycouncilrefusedthedemonstratorsapermitbecausetheyfearedviolence

LanguageisAmbiguous!

examplecredit:DanKlein

‣ Syntac9candseman9cambigui9es:parsingneededtoresolvethese,butneedcontexttofigureoutwhichparseiscorrect

TeacherStrikesIdleKids

BanonNudeDancingonGovernor’sDesk

IraqiHeadSeeksArms

N N V NN V ADJ N

NP

NP

PP

PP PPN

NP

PN

body/posi9on

body/weapon

LanguageisReallyAmbiguous!

‣ Therearen’tjustoneortwopossibili9eswhichareresolvedpragma9cally

‣ Combinatoriallymanypossibili9es,manyyouwon’tevenregisterasambigui9es,butsystemss9llhavetoresolvethem

Itisreallyniceout

ilfaitvraimentbeau It’sreallyniceTheweatherisbeau9fulItisreallybeau9fuloutsideHemakestrulybeau9fulItfactactuallyhandsome

‣ Lotsofdata!

slidecredit:DanKlein

Whatdoweneedtounderstandlanguage? Whatdoweneedtounderstandlanguage?

‣ Worldknowledge:haveaccesstoinforma9onbeyondthetrainingdata

DOJgreenlightsDisney-Foxmerger

metaphor;“approves”

DepartmentofJus6ce

‣ Whatisagreenlight?Howdoweunderstandwhat“greenligh9ng”does?

‣ Needcommonsenseknowledge

‣ Grounding:learnwhatfundamentalconceptsactuallymeaninadata-drivenway

McMahanandStone(2015)Gollandetal.(2010)

Whatdoweneedtounderstandlanguage?

‣ Linguis9cstructure

‣ …butcomputersprobablywon’tunderstandlanguagethesamewayhumansdo

‣ However,linguis9cstellsuswhatphenomenaweneedtobeabletodealwithandgivesushintsabouthowlanguageworks

CenteringTheoryGroszetal.(1995)

Whatdoweneedtounderstandlanguage?

Whattechniquesdoweuse?(tocombinedata,knowledge,linguis9cs,etc.)

PretrainingUnsup:topicmodels,grammarinduc9on

Collinsvs.Charniakparsers

Abriefhistoryof(modern)NLP

1980 1990 2000 2010 2020

earlieststatMTworkatIBM

Largelyrule-based,expertsystems

Penntreebank

NP VP

S

Ratnaparkhitagger

NNP VBZ

Sup:SVMs,CRFs,NER,Sen9ment

Semi-sup,structuredpredic9on

Neural

Supervisedvs.Unsupervised

‣ Supervisedtechniquesworkwellonveryli<ledata(evenneuralnetworks)

annota9on(twohours!)

unsupervisedlearning

“LearningaPart-of-SpeechTaggerfromTwoHoursofAnnota9on”Garre<eandBaldridge(2013)

be<ersystem!

‣ Fullyunsupervisedtechniqueshavefallenoutoffavor

Petersetal.(2018),Devlinetal.(2019)

Pretraining

‣ Languagemodeling:predictthenextwordinatext P (wi|w1, . . . , wi�1)

P(w|Iwanttogoto)= 0.01Hawai’i

P(w|theac9ngwashorrible,Ithinkthemoviewas)=

‣ Modelunderstandssomesen9ment?

0.005LA0.0001class

‣ Trainaneuralnetworktodolanguagemodelingonmassiveunlabeledtext,fine-tuneittodo{tagging,sen9ment,ques9onanswering,…}

:usethismodelforotherpurposes

0.1bad0.001good

Interpretability

Wallace,Gardner,SinghInterpretabilityTutorialatEMNLP2020

‣ Whenwehavecomplexmodels,howdoweunderstandtheirdecisions?

Interpretability‣ Whenwehavecomplexmodels,howdoweunderstandtheirdecisions?

‣ “A<ribu9on”:understandwhatpartsoftheinputcontributetoapredic9on

‣ WhywasitclassAinsteadofclassB?

‣ Whatisthe“counterfactual”scenarioweareconsidering(thefoil)?

IdrankteabecauseIdon’tlikecoffeeIdrankteabecauseIwasthirsty(JacoviandGoldberg,2020))

‣ Datasetbiases:doesourdatahaveflawsthatpreventthemodelfromdoingtherightthing?

‣ Probing:whatrepresenta9onsgetlearnedindeepmodels?

Wherearewe?

‣ NLPconsistsof:analyzingandbuildingrepresenta9onsfortext,solvingproblemsinvolvingtext

‣ Theseproblemsarehardbecauselanguageisambiguous,requiresdrawingondata,knowledge,andlinguis9cstosolve

‣ Knowingwhichtechniquesuserequiresunderstandingdatasetsize,problemcomplexity,andalotoftricks!

‣ NLPencompassesallofthesethings

NLPvs.Computa9onalLinguis9cs

‣ NLP:buildsystemsthatdealwithlanguagedata

‣ CL:usecomputa9onaltoolstostudylanguage

Hamiltonetal.(2016)

NLPvs.Computa9onalLinguis9cs

‣ Computa9onaltoolsforotherpurposes:literarytheory,poli9calscience…

Bamman,O’Connor,Smith(2013)

Outline

MLandstructuredpredic9onforNLP{

Neuralnets {

Outline:Syntax+Seman9cs Outline:Applica9ons

Ethics

https://toxicdegeneration.allenai.org/

‣ E.g.,“toxicdegenera9on”:systemscangenerate{racist,sexist,…}content

‣Wewilltouchonethicalissuesthroughoutthecourse

CourseGoals

‣ CoverfundamentalmachinelearningtechniquesusedinNLP

‣ Makeyoua“producer”ratherthana“consumer”ofNLPtools

‣ CovermodernNLPproblemsencounteredintheliterature:whataretheac9veresearchtopicsin2021?

‣ Thefourassignmentsshouldteachyouwhatyouneedtoknowtounderstandnearlyanysystemintheliterature(e.g.:state-of-the-artNERsystem=project1+mini2+BERT,basicMTsystem=project2)

‣ Understandhowtolookatlanguagedataandapproachlinguis9cphenomena

Assignments

‣ Twominis(10%each),twoprojects(20%each)‣ Implementa9on-oriented,withanopen-endedcomponenttoeach

‣ Mini1(classifica9on)isoutNOW

‣ 1weekforminis,~2weeksperproject,5“slipdays”forautoma9cextensions

‣ Grading:‣ Minis:largelygradedbasedoncodeperformance

‣ Projects:gradedonamixofcodeperformance,writeup,extension

Theseprojectsrequireunderstandingoftheconcepts,abilitytowriteperformantcode,andabilitytothinkabouthowtodebugcomplexsystems.Theyarechallenging,sostartearly!

Assignments

‣ Finalproject(40%)

‣ Groupsof2preferred,1ispossible‣ (Brief!)proposaltobeapprovedbymebythemidpointofthesemester

‣ Wri<eninthestyleandtoneofanACLpaper

A climate conducive to learning and creating knowledge is the right of every person in our community. Bias, harassment and discrimination of any sort have no place here. If you notice an incident that causes concern, please contact the Campus Climate Response Team: diversity.utexas.edu/ccrt

The College of Natural Sciences is steadfastly committed to enriching and transformative educational and research experiences for every member of our community. Find more resources to support a diverse, equitable and welcoming community within Texas Science and share your experiences at cns.utexas.edu/diversity

Conduct Survey(onInstapoll)1. Name

2. Fillin:Iama[CS/____][PhD/masters/undergrad]inyear[12345+]3. Writeonereasonyouwanttotakethisclassoronethingyouwanttogetoutofit

4. Oneinteres9ngfactaboutyourself,orwhatyouliketodoinyourspare9me