Natural Language Processing - Computer Science and...

Wang: Survey of Artificial Intelligence

NLP, p. 1

Natural Language Processing

Agents that communicate

• Producing language is an action, called speech act

• Why communicate between agents?

• Inform

Example: It is warm outside.

• Query

Example: Did you see John this morning?

• Answer

Example: Yes, I saw John in the printer room.

• Request or command

Examples: Please help me carry the TV (I could usesome help carrying the TV)

Carry the TV to DL 366


NLP, p. 2

Cont.

• Promise or offer

Examples: I'll return the midterm on Friday I'll tip you well if you bring the dish now

• Acknowledge requests and offers

Example: OK, I'll try my best with the chef

• Share feelings and experiences

Example: I don't like his cooking


NLP, p. 3

Fundamentals of language

• Natural language vs. formal language

English vs. LISP

• Formal language is defined as a set of strings

- A string is a sequence of symbols

- Symbols are divided into terminal and nonterminalsymbols

- For English, terminal symbols include words, about400,000 of them

• Phrase structure

Sentence (S), noun phrase (NP), verb phrase (VP)

• Rewrite rules

S → NP VP


NLP, p. 4

Chomsky's four grammar categories

(From simple to complex)

• Regular grammar

- Equivalent in expressive power to finite-state automata- Sample rule: S → a S, S → b- Sample language: a*b*

• Context-free grammar

- Equivalent to pushdown automata- Sample rule: S → a S b- Sample language: an bn

• Context-sensitive grammar

- Equivalent to linear bounded automata- RHS must be no shorter than LHS- Sample rule: A B → B A- Sample language: an bn cn

• Recursively enumerable grammar

- Equivalent to Turing machines- No restriction on rewrite rules- Sample rule: A B → C- Sample language: any


NLP, p. 5

Component steps of communication

S: speaker; H: hearer; P: proposition; W: words

• For the speaker

- Intention : S wants H to believe P

- Generation: S chooses the words W

- Synthesis: S utters the words W

• For the hearer

- Perception: H hears W'Generally W' = W, but not always

- Analysis H infers that W' has possible meanings P1, ..., Pn

- Disambiguation: H infers that S intends to convey Pi

Ideally Pi = P, but misinterpretation is possible

- Incorporation: H decides to believe in Pi

H may reject Pi if it, among other things, isinconsistent with what H already believes


NLP, p. 6

Wumpus World

• A computer game with the following rules

- Agent's task: find the gold in the cave and climb out of it- The beast Wumpus eats anyone in its room,

plus trapping pits- Agent and wumpus can move to adjacent room,

horizontal or vertical- Stench smell next to the wumpus- Breeze next to a pit- Gold glittering in its room- Boundary wall bumps- One arrow to shoot wumpus in facing direction- Killed wumpus screams- Score the time taken to get the gold, while not killed


NLP, p. 7

An Illustration in the Wumpus World

Seven steps involved in communication for the examplesentence: "The wumpus is dead."


NLP, p. 8

Telepathic vs. LanguageCommunication

• Telepathic communication using Tell and Ask

KB: knowledge base

• Advantage: very efficient

• Disadvantage: inconsistent and vulnerable

• Communicating using formal language


NLP, p. 9

Grammar for Describing the WumpusWorld

• This subset of English is called language E0

• The lexicon of E0

Noun → stench | breeze | glitter | nothing | wumpus| pit | pits | gold | east | ...

Verb → is | see | smell | shoot | feel | stinks | go| grab | carry | kill | turn | ...

Adjective → right | left | east | south | back | smelly | ... Adverb → here | there | nearby | ahead | right | left

| east | south | back | ... Pronoun → me | you | I | it | ...

Name → John | Mary | Boston | Aristotle | ... Article → the | a | anPreposition → to | in | on | near | ...Conjunction → and | or | but | ...

Digits → 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9


NLP, p. 10

Language E0, cont.

• The grammar of E0

Rewrite rules Examples

S → NP VP I + feel a breeze

| S Conjunction S I feel a breeze + and+ I smell a wumpus

NP → Pronoun I

| Noun pits

| Article Noun the + wumpus

| Digit Digit 3 4 | NP PP the wumpus + to the east | NP RelClause the wumpus + that is smelly

VP → Verb stinks

| VP NP feel + a breeze

| VP Adjective is + smelly

| VP PP turn + to the east | VP Adverb go + ahead

PP → Preposition NP to + the eastRelClause→ that VP that + is smelly


NLP, p. 11

Components of NLP(Natural language processing)

• Syntactic Analysis (Parsing)

- Recover phrase structure from sentences

• Semantic Interpretation

- Extract meaning from sentences

• Pragmatic Interpretation

- Incorporating the current situation

• Disambiguation

- Chooses the best interpretation if more than one is found


NLP, p. 12

Syntactical Analysis

• A simple bottom-up parsing algorithm for context-freegrammar

• Form a forest list containing a sequence of words

• Find a rewrite rule whose RHS matches a subsequenceof forest

• Replace the subsequence by the LHS of the rule

• If forest contains the starting node (S) of the grammar,exit with success; else, go to Step 2.

• A parsing example for "The wumpus is dead."

The forest list Subsequence RuleThe wumpus is dead The Article → theArticle wumpus is dead wumpus Noun → wumpusArticle Noun is dead Article Noun NP → Article NounNP is dead is Verb → isNP Verb dead dead Adjective → deadNP Verb Adjective Verb VP → VerbNP VP Adjective VP Adjective VP → VP AdjectiveNP VP NP VP S → NP VPS


NLP, p. 13

Definite Clause Grammer (DCG)

• Problems with context-free syntactic analysis

• Purely syntactic, with no meaning associated• Grammar is context-free, while natural language is

context-sensitive

• Proposed solution uses the power of first-orderpredicate logic

• Rewrite rules can be expressed in first-order logic

Rules First-order logic

S → NP VP NP s VP s S Append s s( ) ( ) ( ( , ))1 2 1 2∧ ⇒Noun → stench... ( " " ...) ( )s stench Noun s= ∨ ⇒

• A grammar can be written in logic, called logic grammar

• Definite clause logic grammar uses only definiteclauses

• All definite clauses have the form A A C1 2 1∧ ∧ ⇒...

- one consequent, and zero or more antecedents


NLP, p. 14

DCG, cont.

• To avoid being too verbose, we use the followingspecial DCG notation for rewrite rules

• Rule X → Y Z ... translatesas Y s Z s X Append s s( ) ( ) ... ( ( , ,...))1 2 1 2∧ ∧ ⇒

• Rule X → word translates X(["word"])• Rule X → Y | Z | ... translates

as ′ ∨ ′ ∨ ⇒Y s Z s X s( ) ( ) ... ( ), where Y' is the logictranslation of the DCG expression Y (it may not be asingle nonterminal symbol)

• Extensions

• Nonterminal symbols can be augmented with extraarguments (such as sem for semantics)

• A variable can appear on the RHS of a DCG rule• Logical tests can appear on the RHS in braces

• An example for describing numbers in DCG

DCG Rules First-order logic

Digit(sem) → sem {0 ≤ sem ≤ 9} (s=[sem]) ⇒ Digit(sem,s)

Number(sem) → Digit(sem) Digit(sem,s) ⇒ Number(sem,s)

Number(sem) → Number(sem1,s1) ∧ Digit(sem2,s2)Number(sem1) Digit(sem2) ∧ sem = 10x sem1+ sem2 ⇒{ sem = 10x sem1+ sem2} Number(sem, Append(s1,s2))


NLP, p. 15

Augmenting a Grammar

• Overgeneration problem

Examples: Me sees glitter Go me the gold rather than Give me the gold

• The grammar of E1 to represent noun cases(subjective vs. objective)

S → NP(Subj) VP | ...NP(case)→ Pronoun(case) | Noun | Article Noun | ...

VP → VP NP(Obj) | ...PP → Preposition NP(Obj)

Pronoun(Subj)→ I | you | he | she | ...Pronoun(Obj) → me | you | him | her | ...


NLP, p. 16

Augmenting a Grammar, cont.

• Verb subcategorization

• It states which verbs can be followed by which othercategories

• Provided as a subcategorization list

• Some examples

Verb Subcats Example verb phrase

give [NP, PP] give the gold in 3 3 to me[NP, NP] give me the gold

smell [NP] smell a wumpus[Adjective] smell awful[PP] smell like a wumpus

is [Adjective] is smelly[PP] is in 2 2[NP] is a pit

died [] died

believe [S] believe the smelly wumpusin 2 2 is dead


NLP, p. 17

Augmenting a Grammar, cont.

• Verb subcategorization can be implemented byparameterizing VP into VP(subcat)

• Generative capacity of augmented grammars

• In general, it goes beyond context-free grammar

• Example: the context-sensitive language an bn cn

S(n) → A(n) B(n) C(n)

A(0) → ε A(n+1) → a A(n)

B(0) → ε B(n+1) → b B(n)

C(0) → ε C(n+1) → c C(n)


NLP, p. 18

Semantic Interpretation

Produce logical expression

• Compositional semantics

• The semantics of any phrase is a function of the semanticsof the parts of the phase

• Does not depend on the context of the phrase• Similar to context-free

• NL does not satisfy compositional semantics

• Divide NL semantics into two parts

• Semantic interpretation, that satisfies compositionalsemantics

• Disambiguation that handles multiple interpretations,produced by compositional semantic interpretation


NLP, p. 19

Semantics of "John loves Mary"

• λ-expression (lambda) is used as placeholder infunctions and predicates

• Ex: "are from the same state but different cities"λx,y state(x)=state(y) ∧ city(x)≠city(y)

• Arguments to an λ-expression plug into placeholders toyield a standard term or sentence

[λx,y state(x)=state(y) ∧ city(x)≠city(y)](John,Mary)

yields

state(John)=state(Mary) ∧ city(John)≠city(Mary)

• Same usage as in LISP

• VP "loves Mary" expressed as a λ-expression with

• λx Loves(x,Mary)

• NP "John" expressed as object to VP with relationalsemantics

• S(rel(obj)) → NP(obj) VP(rel)


NLP, p. 20

"John loves Mary", cont.

• Together we have

• (λx Loves(x,Mary))(John) or equivalently

Loves(John,Mary)

• Overall semantics expressed as DCG augmentations

S(rel(obj)) → NP(obj) VP(rel)VP(rel(obj)) → Verb(rel) NP(obj)NP(obj) → Name(obj)

Name(John) → JohnName(Mary) → MaryVerb(λx λy Loves(x,y)) → loves

• A parse tree with semantic intepretation


NLP, p. 21

Semantics of E1

• A more complex sentence "Every agent smells awumpus"

• Its semantics:

∀ ⇒ ∃ ∧∃ ∈ ∧

a Agent a w Wumpus w

e e Perceive a w Nose During Now e

( ) ( )

( , , ) ( , )

• NP: NP a Agent a( ( ) ...)∀ ⇒

• VP: VP w Wumpus w

e e Perceive w Nose During Now e

( ( )

(..., , ) ( , ))

∃ ∧∃ ∈ ∧

• Hard to do plug-in due to mutual references

• Introduce quasi-logical form bridging syntax andsemantics

• Include first-order predicate logic, lambda expressions,and a quantified term

Eg: [ ( )]∀a Agent a used as a logical term

• Quantified terms can be used as arguments to predicates

• "Every agent smells a wumpus" expressed in quasi-logic

∃ ∈ ∀ ∃∧

e e Perceive a Agent a w Wumpus w Nose

During Now e

(

([ ( )],[ ( )], )

( , ))


NLP, p. 22

Semantics of E1, cont.

• One can summarize basic steps involved in obtainingquasi-logical form for semantic interpretation

• E.g., the semantic parse tree for the sentence "Every agentsmells a wumpus"


NLP, p. 23

Semantics of E1, cont.• Grammar E2, which is E1 with semantics, is below

S(rel(obj)) → NP(obj) VP(rel)S(conj(sem1,sem2)) → S(sem1) Conjunction(conj) S(sem2)

NP(sem) → Pronoun(sem)NP(sem) → Name(sem)NP([q x sem(x)]) → Article(q) Noun(sem)NP([q x obj∧rel(x)]) → NP([q x obj]) PP(rel)NP([q x obj∧rel(x)]) → NP([q x obj]) RelClause(rel)NP([sem1,sem2]) → Digit(sem1) Digit(sem2)

VP rules for subcategorizationVP(sem) → Verb(sem)VP(rel(obj)) → VP(rel) NP(obj)VP(sem1(sem2)) → VP(sem1) Adjective(sem2)VP(sem1(sem2)) → VP(sem1) PP(sem2)

VP rules for adjuncts (such as time and place for verbs)VP(λx sem1(x)∧sem2(EVENT-VAR(sem1))) →

VP(sem1) PP(sem2)

VP(λx sem1(x)∧sem2(EVENT-VAR(sem1))) → VP(sem1) Adverb(sem2)

RelClause(sem) → that VP(sem)

PP(λx rel(x,obj)) → Preposition(rel) NP(obj)

• Further combination with case and subcategorization


NLP, p. 24

Convert Quasi-logical Formto Logical Form

• ∀xP x( ) within a quasi-logical form QLF becomes∀ ⇒xP x QLF( )

• The term within QLF is replaced by x

• ∃xP x( ) within QLF becomes ∃ ∧xP x QLF( )

• Example: "John loves everyone"

• ∃ ∈ ∀e e Loves John p Person p ( ,[ ( )]) converts to∀ ⇒p Person p Loves John p ( ) ( , )


NLP, p. 25

Pragmatic Interpretation

• Indexicals: phrases that refer to the current situation

• Example: "I was in Dayton yesterday""I" and "yesterday" are indexicals, whose interpretationsdepend on who uttered the sentence and when

• Use Skolem functions in place of indexicals, which canlater be filled in through situational inference about aspeech act, such as information about speaker and time ofthe action

• Anaphora: phrases that refer to objects mentionedpreviously

• Example: "Mary was sick. She went to a hospital."-Who is she?

• Anaphoric inference requires processing of previoussentences and thorough understanding

• A more difficult example: "After John proposed toMary, they found a priest and got married. For thehoneymoon, they went to Hawaii".- Who are they? What is the honeymoon?

• Some consider finding intentions of a query as part ofpragmatic interpretation

• Example: "Do you know the time?"


NLP, p. 26

Disambiguation

• Most utterances are ambiguous

• A few examples from newspaper headlines

Squad helps dog bite victimRed-hot star to wed astronomerHelicopter powered by human fliesOnce-sagging cloth diaper industry saved by full dumps

• Strong context dependence

He is taking care of this street

- for police- for mail carrier- for gang member

• Sources of ambiguity

• Lexical ambiguity: multiple meanings for one word• Syntactic ambiguity: multiple parse trees for one

sentence. Ex: Ed went to school with a friend• Semantic ambiguity. Ex: He is the printer man• Referential ambiguity. Pronouns, etc.• Pragmatic ambiguity. Ex: I'll meet you next Thursday• Vagueness. Ex: The dish is spicy

• Ambiguity is not necessarily bad


NLP, p. 27

Disambiguation, cont.

• Disambiguation can be formulated as diagnosticinference

• It requires a combination of four models

• A world model: the prior probability for a fact• A mental model: the probability that the speaker

intends to communicate this fact, given that it occurs• A language model: the probability of choosing a

certain string of words, given speaker's intention• An acoustic model: the probability of choosing a

certain sequence of phones, given words

• It works much like in speech recognition


NLP, p. 28

Applications of NLP

• Machine translation

• Human-computer dialogue and interface

• ELIZA (Weizenbaum, 1965): keywords-based NLdialogue system, with little understanding

• LUNAR (Woods, 1973): answering NL questions, aboutlunar rock and soil samples

• CHAT (Pereira, 1983): answering NL queries aboutgeography. An excerpt:

Q: Which countries are bordered by two seas?A: Egypt, Iran, Israel, Saudi Arabia and TurkeyQ: What are the countries from which a river flows

into the Black sea?A: Romania, Soviet UnionQ: What is the total area of countries south of the

equator and not in Australasia?A: 10,228,000 square milesQ: What is the ocean that borders African countries

and that borders Asian countries?A: Indian Ocean


NLP, p. 29

Applications of NLP, cont.

• PEGASUS (Zue, et al., 1994): speech and NLunderstanding system for on-line air travel planning. Anexample:

Traveller: I want to go from Boston to San FranciscoPEGASUS: What date will you be travelling on?Traveller: October 20th

. . . . . .Traveller further provided the following information:,

nonstop, cheapest fare, returning on Sunday, etc.,through the dialogue

PEGASUS provided a confirmed reservation,satisfying all the above demands

• Information Retrieval

• Database access using natural language input• Content-based document retrieval (e.g.: find all stories

about Bill Clinton)• Web-based search


NLP, p. 30

Natural Language Processing

Summary

• Big successes in small domains

• Big market for NLU products

• NLP is AI-hard

• Huge amount of knowledge about the world is required

• A good model for the listener is required

• Demand for communication efficiency leads to largecontextual and social effects

Date post:	19-May-2018
Category:	Documents
Upload:	phamnga
View:	214 times
Download:	0 times

Natural Language Processing - Computer Science and...

Documents