SEIZE THE DATA. 2015h41382.€¦ · SEIZE THE DATA. 2015 IDOL Under the Hood: Indexing and Search...

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.1 SEIZE THE DATA. 2015

SEIZE THE DATA. 2015


IDOL Under the Hood:Indexing and Search of Human DataJohn Horman, Director of TechnologyBrian Cowe, Senior Product Manager

August 2015


IDOL: Fundamentals


Why is processing human information difficult?

Ideas don’t exactly match like transactional data does; they have “distance.”

Human information is not static – it’s dynamic and lives everywhere.

Human information represents a source of both opportunity and risk.

Legacy techniques fall short.

It is made up of ideas, is diverse, and has context

Social media Video Audio

Email Texts Mobile

Documents Search engine Images

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

If we toss a coin 100 times and get heads every time, what’s the probability of getting a head on the 101st?

Traditional probability says:

50%


If we toss a coin 100 times and get heads every time, what’s the probability of getting a head on the 101st?

Bayesian Inference says:

99+%Adaptive intelligence: prior information changes the model of understanding

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 SEIZE THE DATA. 2015© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.


Key words are small amounts of very strong information without contextLarger amounts of weaker information is what humans refer to as “context”

Strong information & weak information

“Mercury”

Is it a planet?Is it an element?Is it a car?

“A heavy element and the only metal that is liquid at standard conditions for temperature and pressure with the symbol Hg and atomic number 80,

commonly known as quicksilver”

With high certainty: it’s an element!


Uses pattern-matching and probabilistic modeling to form an understanding of content

HP IDOL: Natural Language Processing (NLP) engine

Fundamentally created to understand natural

human language using probabilistic modeling

and NLP algorithms• Allows incoming data to dictate the model,

not pre-defined rules, dictionaries, or semantic webs

Self-Learning/Machine Learning• Updates as more data is added or removed

• Adapts to changing definitions or meaning

Fundamentally language-independent• Treats words as symbols

Optimized with language packs• Eduction, sentiment analysis, speech analytics

Information Theoryand Bayesian Inference


• Sentiment analysis

• Social CRM/network analysis

• Churn mitigation

• Brand monitoring

• Cross and Up sell

• Loyalty & promotion analysis

• Web application optimization

• Marketing campaign optimization

• Brand management

• Social media analytics

• Pricing optimization

• Internal risk assessment

• Customer behavior analysis

• Revenue assurance

• Logistics optimization

• Clickstream analysis

• Influencer analysis

• IT infrastructure analysis

• Legal discovery

• Equipment monitoring

• Enterprise search

For every industry and use case

HP Big Data platform

Government Telecom Manufacturing Healthcare

• Drug development

• Scientific research

• Evidence-based medicine

• Healthcare outcomes analysis

• Supply chain optimization

• Defect tracking

• RFID Correlation

• Warranty management

• Broadcast monitoring

• Churn prevention

• Advertising optimization

• Law enforcement

• Counter terrorism

• Traffic flow optimization

Horizontal use cases

Finance

• Fraud detection

• Anti-money laundering

• Risk management

Energy

• Weather forecasting

• Natural resource exploration


IDOL: Key features


Language independence

• Free from linguistic restraints and rules

• Automatically adapts to changing definitions

• Over 170 live customer languages

• Single, multibyte and Unicode languages

• Optional language packs for localization

• Integrated with machine translation software

Department of Homeland Security - Requires extremely precise handling of foreign languages, including Chinese and Arabic

Open V – China’s largest online video website


HP IDOL Multi-language Search in action


HP IDOL retrieval methods

Conceptual

• Natural language

• Conceptual matching

• Unstructured refinement

Business rules

• Boolean

• Keyword

Parametric

• Structured refinement Improve“Enhance your data”

Inquire“Search your data”

Interact“Personalize your

data”

Investigate“Analyze your data”


Over 100 operators for Boolean search

Improve“Enhance your data”



data”


AND

OR

NOT

NEAR

NEARn

DNEAR

DNEARn

WNEAR

WNEARn

BEFORE

AFTER

EOR

WHEN

WHENn

vAND

vSUBSTRING

vMATCHES

NEAR

NEAR/n

SENTENCE

PARAGRAPH

BEFORE

AFTER

ORDER

SOUNDEX

MANY

[n] WORD

CASE

PHRASE

. >

. >=

. <

. <=

. !=

. =

LANG/x

TODAY

YESTERDAY

NOW

NOW+n

NOW-n

term

term*

term?

vOR

vNOT

vACCRUE

vANY

vALL

vIN

vWHEN

vCONTAINS

vENDS

vSTARTS

vSUBSTRING

vCONTAINS

vENDS

vSTARTS

FREETEXT

STEM

TYPO

TYPO/n

YES-NO

PRODUCT

SUM

COMPLEMENT

LOGSUM

LOGSUM/n

MULT

MULT/n

FREQ

term~

term[100]

term[*1.5]

"term"

"term phrase"

term:field

"term

phrase":field

~term

FUZZY()

FUZZYnn()

SOUNDEX()

APCMMOD[]

term[~]


Automatic hyperlinking

• Automatically retrieves conceptually related content

• Searches automatically done for the user

• Increase productivity and reduce duplicate work




data”



Add context to short queries by grouping results into concepts

Automatic query guidance




data”


Query ”Madonna”

Results: Documents containing ”Madonna”

Query search

Documents about:1. Singer2. Italian Renaissance3. Madonna Further

suggestions…

Most likely meaning…

Result documents

Conceptual clustering


Summarization




data”


Quick summary(N+ lines)

Context summary(What is this doc about with relation

to query terms?)

Concept summary(What do I look for with regards

to interest rates?)

Information Theory andBayesian Inference


Automatically partition the data so that similar information is clustered together

Clustering




data”


Product performance issues

Side letters

Off balancesheet transactions


Visualization of main topics




data”



Personalize your data




data”


Explicit profiling (Agent): user-defined • Define your interest using:

- Natural language descriptions

- Keyword/boolean rules

- Refine by example

• Automatically monitor information

• Customizable

• Share interests with knowledge community

Implicit profiling: capturebehavior data• Fully automatic• Ongoing monitoring of data consumption

and contribution• Multi-faceted profiles• Always up-to-date

Expertise

CommunitiesAgents

Profiles

Dynamic communities of interest• Expert identification

• Define business rules to guide relationships

• Automatically form and manage community

• Collaboration networks

• Document rating

• Consumer groups

Expertise Expertise


Understanding the customer at the level of a dialog





data”


Geo + demo + psychographicsegments Behavioral segments

Functions

Performance

Feature driven

Reviews

News

Adverts

Social media

Buzz driven

18-35 yrs

35-65

Seniors

Have Kids

Male

Female

Semantic segments

Large screen

Lots of storage

High res display

Would give it 5 stars

Great value for price




Search results personalized and targeted based on user and context

Profile developed through complete behavior analysis… implicit or explicit profiling

Gather data from content consumption,

content contribution, interaction with colleagues, etc. Improve

“Enhance your data”



data”



Hundreds of conceptual entities

Eduction

Quickly narrow search results with auto-identified facets and conceptual entities such as employee names from documents

Validate or customize entities

• Is this a valid credit card number?

• What are all docs that contain SSNs?

• If area code is 415, output as Home Office

Pinpoint accuracy for multibytelanguages such as CJK, Thai and some European languages Improve

“Enhance your data”



data”


NamesPlacesIP addressesCompaniesEventsRelationshipsMedicinesAirportsCarsSocial Security numbersPhone numbersCredit cardsDatesHolidaysJob titlesCurrencies… many more


Eduction




data”


<Organization> • National Security Agency

<Names>• President Obama• Vladimir Putin• Edward Snowden

<Places>• Moscow• St. Petersburg• Washington• Syria• Russia

<Author> • Carla Anne Robbins


Automatically redact sensitive or offensive entities

Profanity

Personally Identifiable Information (PII)

Payment Card Industry Data Security Standard (PCI-DSS)

…and hundreds of entities Improve“Enhance your data”



data”




Topical sentiment analysis

Decomposition and classification within a sentence to pull out specific topics

“I stayed at the Marriott last week, and though the mattresses were very nice, the service was awful.”

Is this positive? Negative? Neutral?

How positive? How negative? Improve“Enhance your data”



data”



IDOL: Techniques and algorithms


IDOL knowledge flow

Acquisition Preparation Indexing Querying & analytics


Language processing

God

temple

King

Egypt

fine

being

Ptolemy

father

excellence

manifest


“Fried green tomatoes at the Whistle Stop Café”

Language processing

StemDetect stopwordsTransliterateTokenize

Language specific optimizations


“Fried green tomatoes at the Whistle Stop Café”

“Fried” “green” “tomatoes” “at” “the” “Whistle” “Stop” “Café”

Language processingLanguage specific optimizations




“fried” “green” “tomatoes” “at” “the” “whistle” “stop” “cafe”





“fried” “green” “tomatoes” “whistle” “stop” “cafe”





“fri” “green” “tomato” “whistl” “stop” “caf”




“Fried green tomatoes at the Whistle” “Stop Café”




“fri” “green” “tomato” “whistl” “stop” “caf”




Language processing

N-gram

• 1-gram: “language” => “l” “a” “n” “g” “u” “a” “g” “e”

• 2-gram: “language” => “la” “an” “ng” “gu” “ua” “ag” “ge”

• 3-gram: “language” => “lan” “ang” “ngu” “gua” “uag” “age”

• etc

“映画版ではイジ” => “映画版” “画版で” “版では” “ではイ” “はイジ”

“C8H10N4O2” => “C8H” “8H1” “H10” “10N” “0N4” “N4O” “4O2”


IDOL knowledge flow



Index construction

DocID

1

2

3

4

5

6

7

8

Terms

It happened that a Dog had got a piece of meat and was carrying…

A gaunt Wolf was almost dead with hunger when he happened to…

A Man and a Lion were discussing the relative strength of men…

A Gentleman, having prepared a great feast, invited a Friend to…

On a summer day, when the great heat induced a general thirst…

Once when a Lion was asleep a little Mouse began running up and…

A Dog looking out for its afternoon nap jumped into the Manger…

A Lion once fell in love with a beautiful maiden and proposed…


Index construction

DocID

1

2

3

4

5

6

7

8

Terms










Index construction

DocID

1

2

3

4

5

6

7

8

Terms










Index construction

DocID

1

2

3

4

5

6

7

8

Terms










Index construction

DocID

1

2

3

4

5

6

7

8

Terms









Term

DOG

FEAST

LION

LOVE

MEAT

MOUSE

THIRST

WOLF

DocIDs

1(pos: 5, 38, 70), 2(pos: 42), 4(pos: 22, 29), 7(pos: 2, 28, 43, 49)…

1(pos: 27, 120), 4(pos: 7, 62, 112), 5(pos: 40), 8(pos: 27, 93)…

3(pos: 5, 21, 34, 64), 4(pos: 19, 41, 60), 6(pos: 4, 19, 28, 51)…

8(pos: 6, 37, 59), 16(20, 49, 104), 31(pos: 27, 30, 48, 61, 69)…

1(pos:11, 50), 4(pos: 23), 9(pos:23, 38), 10(pos: 7, 39, 104)…

6(pos: 9, 17, 27, 60, 68), 22(pos: 7, 39, 44, 68), 25(pos: 40, 49)…

3(pos: 45), 5(pos: 12, 27), 13(pos:17, 44, 91, 94)…

2(pos: 3, 23, 28, 54, 71), 28(pos:79, 92), 49(pos: 2, 18, 52, 93)…


Index construction

DocIDs

1(pos: 5, 38, 70), 2(pos: 42), 4(pos: 22, 29), 7(pos: 2, 28, 43, 49)…

1(pos: 27, 120), 4(pos: 7, 62, 112), 5(pos: 40), 8(pos: 27, 93)…

3(pos: 5, 21, 34, 64), 4(pos: 19, 41, 60), 6(pos: 4, 19, 28, 51)…

8(pos: 6, 37, 59), 16(20, 49, 104), 31(pos: 27, 30, 48, 61, 69)…

1(pos:11, 50), 4(pos: 23), 9(pos:23, 38), 10(pos: 7, 39, 104)…

6(pos: 9, 17, 27, 60, 68), 22(pos: 7, 39, 44, 68), 25(pos: 40, 49)…

3(pos: 45), 5(pos: 12, 27), 13(pos:17, 44, 91, 94)…

2(pos: 3, 23, 28, 54, 71), 28(pos:79, 92), 49(pos: 2, 18, 52, 93)…

Term

DOG

FEAST

LION

LOVE

MEAT

MOUSE

THIRST

WOLF


IDOL knowledge flow



Query processing

http://idol-server:9000/action=query&text=playing hide and seek

&fieldtext=MATCH{forest}:TERRAIN&maxresults=20&printfields=CONTENT,

DATE,COUNTRY&responseformat=JSON&highlight=terms


Query processing

action=query





Query processing

fieldtext=MATCH{forest}:TERRAIN

maxresults=20




text=playing hide and seek


Query processing

printfields=CONTENT,DATE,COUNTRY

responseformat=JSON

highlight=terms





Query processing

plai hide seek

text=playing hide and seek

OR

plai hide seek


Query processing

plai hide AND seek

text=playing hide AND seekOR

plai

hide seek

AND


Query processing

plai “ hide seek ”

text=playing “hide and seek”OR

plai

hide seek

PNEAR1


Query processing

OR

plai

hide seek

PNEAR1

plai “ hide seek ”

text=playing “hide and seek”

plai hide seek


plai hide seek

AN

D

NO

T

OR

Query processing


Query processing


Query processing

+ Term proximity

- Propername terms

+ / - Query time term weighting

+ / - Field weighting

+ Unstemmed match

+ Term Frequency

+ Inverse Document Frequency


Query processing

Ma

do

nn

a[9

9]

Ita

lia

n[5

4]

Re

na

issa

nce

[80

]

Sa

nta

[66

]

Ma

ria

[66

]

Pa

inte

r[7

0]

Leo

na

rdo

[10

0]

1582243 30 12 22 1 1 4 17 77.75% Themes in Italian Renaissance painting

1565858 1 55 27 1 1 70 3 73.73% 1480s in art

23768 0 1 3 1 2 0 0 51.04% Pope Paul II

2932666 0 0 2 1 1 0 0 40.74% List of streets in Rome

41748 17 0 0 0 0 0 0 32.67% Guy Ritchie


IDOL knowledge flow



Agents

SHAKESPEAR~[500] ROMEO~[400]

JULIET~[400] OTHELLO~[350]

HAMLET~[350] LEAR~[300]

STRATFORD~[250] RSC~[200]

CORIOLANU~[200] MACBETH~[150]

WILLIAMSHAKESPEAR~[100] ...


IDOL: Not just a search box!

TargetSource

Criteria Document Person Agent

Criteria Query expansion Query Expertise search Agent search

Document Categorization(agentboolean)

Suggest Profiling Classification

Person People categorization(agentboolean)

Profile search Community People classification

Agent Agent categorization (agentboolean)

Query-timeclassification

Expertise search Agent suggest


Q&A


Date post:	09-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times