Analyze

Extract

Match

Transform

Information

Revealed

Connect

Analyze

Extract

Match

Transform

Information

Revealed

Connect

Overview

• Very briefly introduce Basis

• Motivate the move from Strings to Things

• Review two enabling technologies:

– Entity Extraction: finding names in text (and classifying them)

– Entity Resolution: connecting names together and to things

• Give you three examples of things you can do:

– Entity-based search, illustrating:

• How entities and enriched typing can empower searchers

• How human corrections might be used to improve accuracy over time

– Get additional high quality enrichments from knowledge sources

– Recognize anomalies/outliers, by establishing rich norms

4

Introduction: Basis Technology

5

Introduction: Gregor Stewart

6

Facebingler cares, should you?

Facebingler cares, should you?

8

Entity Extraction: What is it?

9

Entity Extraction: How is it done?

10

Probabilistic Extractor

Supervised Model

Unsupervised Model

Deterministic Extractor

Exact Match (Gazetteer)

Pattern Match (Regex)

En

tity

Red

act

or

JoiningInputText

Filtering

Adjudication

TaggedText

Domain

Text

Annotated

Text

User Defined

Lists

User DefinedPatterns

Entity Resolution: What is it? (1)

Entity Resolution: What is it? (2)

Alberto

Alberto

AlbertoAlberto

Alberto Amos Fernandez…

Alberto M.Fernandez…

Alberto Fernandez…

Alberto Fernandiz…

AlbertFernandez…

Alberto

Alberto

AlbertoAlberto

Alberto Fernandez…

… Chief of Cabinet… Argentina… …Prof of Criminal Law…

Alberto Fernandez…

… born Sept 7, 1984… cycling… Madrid

Alberto Fernandez…

… born in Cuba… US Ambassador

Alburto Fernandez…

Alberto

Alberto Fernandezde la Puebla…

Alberto

Ratio ofPoliticians to Sportsmen?

2:1

Alberto Fernandez… Sportsmen?

YES

Nickname“El Galleta?”

?

But it’s not just text (1)

But it’s not just text (2)

?

Entity Resolution: How is it Done? (1)

Entity Resolution: How is it Done? (2)

16

Entity Resolution: How is it Done? (3)

17

Entity Resolution: How is it Done? (4)

18

Resolution EngineCandidate Selection

Entity Index

Entity Mentio

n+

Context

Link or Ghost

Ranking

Knowledge Base

Learned

Seeded

!

A (Convenient) Fiction…

• In a nearby place, not so long ago… the CIA was asked by the President to assess the likelihood that the Syrian opposition would use chemical weapons by mid-2014.

• As part of building that analysis, and because there are Al-Qaeda elements in the Syrian opposition, Alice the analyst was asked to: characterize Al-Qaeda’s attitude to using chemical weapons against Middle Eastern governments.

19

20

From: Ayman Al-Zawahiri (?)To: “Hafiz Sultan”

Dear Brother, We need guidance from you on the issue of using chlorine gas technology. It was reported that the brothers in Iraq have used it, but this was implicitly denied in a

statement issued by the Islamic State of Iraq.

The brothers where Mahmud is have the potential to use chlorine gas on the forces of the apostates, Jalal Talabani and Mas'ud Barzani, and have already considered using it.

However, I informed them that matters as serious as this require centralized [coordination] and permission from the senior [al-Qa'ida] leadership, because the gas could be difficult to control and might harm some people, which could tarnish our image, alienate people from

us, and so on.”

A document that Alice needs to read (socom-2012-

0000011)…

21

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

22

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

23

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

24

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

25

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

26

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

27

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

28

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

29

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

30

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

31

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

32

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

33

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

34

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

35

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

36

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

37

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

38

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

39

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

40

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

41

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

42

3/28/07 Today

5/2/11 5/3/11 5/12/11 5/14/11

Advanced enrichment: Topics?

• Some knowledge sources have rich connectivity between things, concepts, etc.

• Developers often ask for “Topic”

• Even advanced Topic approaches often yield “howlers”

• Better labels might be derived from node info or graph walking.

Advanced enrichment: Norms?

• By walking the graph in very specific ways, we can build one or more efficient representations of what is normal or expected context for an entity.

• This could be focused on particular entities, types of entities, relationships, etc.

• We could use these representations to affect result rankings, raise alerts.

• Note again, that this is not specific to text: elementary parts of other unstructured sources such as images and video might be connected/used in the same way.

Summary

• Extraction and resolution components like REX and RES, can reliably connect Strings to Things in a range of texts.

• This allows existing knowledge to be usefully applied:• We can add properties (like types), and other advanced enrichments• We can discover where existing knowledge is lacking

• Thing-based search can allow each query to be more precise and productive• Fewer queries, fewer adjustments, fewer results to read

• By using abundant human feedback, KB quality and resolution accuracy can be increased.• More subtle distinctions between entities can be learned, example by

example.

• But…

…these tools are like shoes…

Thank

You!gregor@basistech.c

om