Web ScienceIntroduction to Information Integration

Julien Gaugaz, October 26, 2010

2

Topics•1. Information Integration

•2. Web Information Retrieval

•3. Entity Search

•4. Web Usage

•5. Collaborative Web

•6. Web Archiving

•7. Medical Social Web

Scenarios

Why Integrating Information?

4

Company Mergers

5

Travelling Agent

AgentAgent

6

Booking Flights

AgentAgent

7

Leveraging Wikipedia Infoboxes

Query

Data Contribution

8

Evolution

Beginning ofDatabases

Wikipedia &Social Web

Rise of Internet & Wrapping

Websites

Num

ber

of

Sourc

es

Kinds of discrepancies

What is the Problem?

10

Wikipedia Infoboxeshttp://de.wikipedia.org/wiki/Berlin

http://en.wikipedia.org/wiki/Berlin| leader_title = [[List of mayors of Berlin|Governing Mayor]]||| leader = Klaus Wowereit| elevation = 34 - 115| pop_date = 2010-03-31| population = 3440441| pop_metro = 5000000

| [[(...)|Reg. Bürgermeister]]:|| [[Klaus Wowereit]]

| [[Höhe]] : || 34–115 m ü. NN

| [[Einwohner]] : || {{Metadaten Einwohnerzahl DE-BE|Berlin}}[...] (rendered as: 3.443.735 (31. Mai 2010))

11

Wikipedia Infoboxeshttp://de.wikipedia.org/wiki/Berlin

http://en.wikipedia.org/wiki/San_Francisco

|leader_title = [[Mayor of San Francisco|Mayor]]|leader_name = [[Gavin Newsom]] ([[Democratic [...]|D]])|elevation_ft =

52|elevation_max_ft = 925|elevation_min_ft = 0|population_as_of = 2008|population_total = 815358|population_metro = 4203898|population_urban = 3228605

| leader_title = [[List of mayors of Berlin|Governing Mayor]]||| leader = Klaus Wowereit| elevation = 34 - 115| pop_date = 2010-03-31| population = 3440441| pop_metro

= 5000000

12

Causes of Discrepancies

•Information sources are diverse

•Different cultural background

•Different domain of activity

•Different model of information

•Typos and other kinds of errors

•Evolution over time

•Use, usage and users of one source may change of over time

13

Places of DiscrepanciesInformation level where discrepancies appear:

•Semantic: meaning, sense

•Representational

• Lexical: word / term representing the meaning

• Structural: how are the terms arranged to represent the meaning

•Syntactic: how is the lexical and structural encoded into characters (and bits)

Discrepancies may concern:

•Schema elements (properties and structure) and values

14

Schema Discrepancies

Semantic

Representational

Syntactic

Einstein’s full name is “Albert Einstein”

EinsteEinsteinin

name first

last

“Albert”

“Einstein”“Albert Einstein”

full_nameEinsteEinste

inin

<Einstein> <full_name> “Albert Einstein”.

<Einstein> <full_name>Albert Einstein</full_name></Einstein>

15

SemanticRepresentational

Schema Ambiguity

Article title

“Prof. Dr. techn.”xyzxyztitle

“The Theory of Relativity”xyzxyztitle

Person title

16

Value Discrepancies

SemanticRepresentational

Einstein’s full name is “Albert Einstein”

“Albert Einstein”“Albert Einstin”“A. Einstein”“Einstein, Albert”

full_nameEinsteEinste

inin

Where discrepancies are addressed with standards

Syntactic Level

18

Encoding Bytes•Basic unit

•Universal standard: Bit (binary digit)

•Ternary digit (base 3, USSR 50’s, out of use)

•Bits into bytes

•Big or small endian

•System wise convention, easily convertible, defined in communication protocols

19

Encoding Characters

•De facto standards:

•UTF-8/16

•Many others exist: ASCII, ISO-8859’s, KOI-8, ...

•Trivial dictionary-based translation

•When the corresponding code exists in the target character map...

20

Encoding Lexico-Structural

•XML, XML Schema

•Structured document serialization format

•Base for:

•(X)HTML

•SVG: Scalable Vector Graphics

•DOCX: Microsoft Office Word 2007

Resource Description FrameworkEncoding information

RDF

22

•<subject> <property> <object>

•<subject>

•URI or blank node

•<property>

•URI

•<object>

•URI or blank node or (typed) literal22

source: http://www.xml.com/2003/02/05/graphics/graph1.gif

23

URI

•URI: Universal Resource Identifiers

•URL’s are URI’s

• scheme:scheme-specific-part

•RDF encourage using URL’s

•URL

• scheme://usr:passwd@domain:port/path?query_string#anchor

24

RDF•Resource Description Framework

•Data model specialized in conceptual information modeling

•Supported by various serialization formats:

•XML

•Notation3 (N3)

•Turtle

•...

25

RDF Schema (RDF/S)•Expressed in RDF

•Types subjects and objects with classes

•Class hierarchy (with multiple inheritance)

•Type of properties of a class

•Types properties

•Domain: type of property’s subject

•Range: type of property’s object

•OWL2 is more expressive: cardinality, etc...

26

When to use RDF?•RDF is good at

•Modeling information

•Especially when schema is unknown or changing

•When there is multiple schemas

•RDF is not for

•Representing documents (XHTML, CSS)

• Internal data management when schema is known and fixed (Relational Databases)

Discrepancies between the representational and semantic levels in

the schema

Schema Matching

28

• name• boxer id• weight• birthdate• total fights• residence

• first name• last name• age• address• street• city• tax id

•Input: Schemas to match

•Possibly data instantiating those schemas

•Output: Mappings between schema elements

•Possibly with confidence values and alternatives

•Possibly with value conversion rules (matchings)

Boxer Taxpayer

• ...

Company

• ...

Trainer

• ...

Tax Office

29

Mappings or Matching?

•Schema mapping identifies correspondences between schema elements

•Schema matching actually transforms an instance of one schema into an instance of another schema

General architectures

How to Use Mappings?

31

Mediated Schemas

Mediated

Schema

Query

Schema1

Schema2

Schema3

Query

Mediated

Schema

Schema1

Schema2

Schema3

Query

Schema x

32

Peer Data Management

Local MappingLocal Source

Peer Schema

Peer Mapping

Local Schema

33

Why not by hand?•Size and complexity of source schemas

•Number of schemas sources

•Leveraging data instance values

•Schemas not known in advance

source: http://www.geneontology.org/images/diag-godb-er.jpgsource: http://www.atutor.ca/development/documentation/database.gif

34

Schema Matching Features

•Schema-only vs schema & instances

•Representational

•Lexical vs structural

•Internal vs external

More in:•Rahm E, Bernstein PA. A survey of approaches to automatic schema matching. The VLDB Journal. 2001;10(4):334-350.•1. Shvaiko P, Euzenat J. A Survey of Schema-Based Matching Approaches. Journal on Data Semantics IV. 2005;3730:146-171.

35

•String-based

•Language-based

•Linguistic resources

•Constraint-based

•Alignment reuse

•Upper-level formal ontologies

Schema Matching Techniques

•Graph-based

•Taxonomy-based

•Repository of structures

•Model-based

Leveraging lexical features

A String-Based Technique

37

Edit Distance•String distance: measures distance

between two strings

•Edit distance: number of operations needed to transform one string into the other

•Common basic operations:

•Insert, delete or substitute one character

•Possibly with different weights depending on the operation and characters involved

•Java libraries:

•SecondString, SimMetrics

38

Levenshtein Distance•Edit operations: insert, delete,

substitute•Each has a weight of 1

S a t u r d a y0 1 2 3 4 5 6 7 81 0 1 2 3 4 5 6 72 1 1 2 2 3 4 5 63 2 2 2 3 3 4 5 64 3 3 3 3 4 3 4 55 4 4 4 4 4 4 3 46 5 5 5 5 5 5 4 37 6 6 6 6 6 6 5 4

insert to Sundays

dele

te f

rom

Sundays substitute in Sundays

SundaysSatundaysSatundaysSaturdaysSaturdaysSaturdaysSaturdaysSaturdays

Sundays

WordNet

A Linguistic Resource

40

WordNet•Fundamental components: Synonyn Sets

(Synsets)

•{car, auto, automobile, machine, motorcar}

•a motor vehicle with four wheels; usually propelled by an internal combustion engine

•{car, railcar, railway car, railroad car}

•a wheeled vehicle adapted to the rails of railroad

41

Hypernyms / Hyponyms

•Hypernyms: superordinates, isA relationships. A synset may have more than one hypernym.

•Hyponyms: subordinates

{car, auto, automobile, machine, motorcar}

{motor vehicle, automotive vehicle}

{cab, hack, taxi, taxicab} {ambulance}

hypernym

hyponyms

42

Holonym / Meronym•Meronym: name of a constituent part of, the

substance of, or a member of something. X is a meronym of Y if X is a part of Y.

•Holonym: name of the whole of which the meronym names a part. Y is a holonym of X if X is a part of Y.

{car, auto, automobile, machine, motorcar}

{ accelerator, accelerator pedal, gas pedal, gas, throttle, gun}

holonym meronym

43

Other relationships in WN•Antonym

•Entailment (for verbs)

•A verb X entails Y if X cannot be done unless Y is, or has been, done.

•Attribute (for adjectives)

•A noun for which adjectives express values. The noun weight is an attribute, for which the adjectives light and heavy express values.

Leveraging structure

A Graph-Matching Technique

45

Similarity Flooding•Uses structure of the data to help matching

schemas

• Similarity Flooding in Melnik et al. (2002)

• First maps schema elements with lexical similarity

• Then improves matching assuming that:

• If two elements are similar, then the elements adjacent to them are more probable to be similar

Selected paper 1:Melnik S, Garcia-Molina H, Rahm E. Similarity flooding: a versatile graph matching algorithm and its application to schema matching. IEEE Comput. Soc; 2002:117-128.

Detecting duplicate entries

Deduplication

47

Why is there Duplicates?

• first name: Mohamed• last name: Ali• age: 68• address: street: Nicestreet 17 city: Wondercity• tax id: #7234561

• name: Muhammad Ali• boxer id: 1234567• weight: 200 lb• total fights: 61• residence: 17, Nicestreet Louisville, KY

Sport Authorities Taxes Authorities

AdministratAdministration-wide ion-wide databasedatabase

48

•Input: 2 entities with matched attributes

•Output: M for matched or U for unmatched.

•Possibly R for reject between M and U for cases where supervised decision is necessary.

• name: Muhammad Ali• boxer id: 1234567• weight: 200 lb• total fights: 61• residence: 17, Nicestreet Louisville, KY

• first name: Mohamed• last name: Ali• age: 68• address: street: Nicestreet 17 city: Wondercity• tax id: #7234561• name: Muhammad Ali

• address:• city: Cairo• country: Egypt• tax id: #8244361

M

UR

Deduplication Features

50

•Value metrics

•Character-based

•Token-based

•Phonetic

•Numeric

Field Distance Metrics

String-based metrics seen for schema matchingSimilar to Information Retrieval techniques (Topic 2 next week)

Not much techniques other than considering them as strings or direct difference

51

Phonex1.First letter as prefix

2.Encode non-prefix consonants

3.Remove duplicate adjacent codes not separated by a vowel

4.Drop vowels and truncate to prefix and max 3 codes, resp. pad with zero if necessary

consonant code

b, f, p, v 1

c, g, j, k, q, s, x, z

2

d, t 3

l 4

m, n 5

r 6

h, wdroppe

d

Rupert•Rupert•Ro1e63•Ro1e63•R163

Robert•Robert•Ro1e63•Ro1e63•R163

Ashcraftson1.Ashcraftson2.A2 26a132o53.A26a132o54.A261

52

Other Phonetic Codes

•NYSIIS

•Developed and still in use at the New York State Division of Criminal Justice Services

•Encodes vowels (mostly to A)

•Codes are letters instead of digits

•Longer codes (6 instead of 4)

53

Other Phonetic Codes

•Metaphone

•Codes are letters instead of digits

•No maximum code length

•More elaborated coding rules

•Double Metaphone

•Returns a secondary code to help disambiguate

Detecting Duplicates

55

•M: match, U: unmatch

•Using Bayes rule

•Decision rule: likelihood ratio

•Using independence assumption

Bayes Decision Rule

56

Bayes Decision Rule

•Priors ( and ) can be learned on a training set

•Other methods based on Expectation-Maximisation (EM) algorithm can estimate priors without training set

57

Clustering-Based Decision•Using clustering techniques with appropriate

parameters

• X-Means

• Variant of K-Means without a fixed K

• Chauduri et al. observed that duplicates tend

1.to have small distances from each other (compact set property), and

2.2) to have only a small number of other neighbors within a small distance (sparse neighborhood property).

Selected paper 2:Chaudhuri S, Ganti V, Motwani R. Robust Identification of Fuzzy Duplicates. ICDE’05. 2005:865-876.

58

Dealing with O(n2)

Number of entities in repository

Num

ber

of

com

pari

sons

59

Canopies

●

●

●

●● ●● ●

●

●

●

●

●●

●

●

•Create canopies using a cheap similarity metric

•Overlapping clusters

•Compare entities pairwise using a more expensive similarity metric

Pay-as-you-go Information Integration

Dataspaces

61

Dataspaces•Note a data integration approach per

se

•Data co-existence appraoch

•Pay-as-you-go data integration

•Leveraging human contributions for data integration in a non-invasive manner

Selected paper 3:Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS ’06. New York, NY, USA; 2006:1-9.

62

•Are they duplicates?

•To compare field values we need schema matches

•To find schema matches we need duplicates

•etc...

Relationship between Schema Matching and

Deduplication

• name: Muhammad Ali• boxer id: 1234567• weight: 200 lb• total fights: 61• residence: 17, Nicestreet Louisville, KY

• first name: Mohamed• last name: Ali• age: 68• address: street: Nicestreet 17 city: Wondercity• tax id: #7234561

Selected paper 4:Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD 2007. Beijing, China; 2007:545-556.

63

Selected Topic Papers1.Schema Matching

• Melnik S, Garcia-Molina H, Rahm E. Similarity flooding: a versatile graph matching algorithm and its application to schema matching. IEEE Comput. Soc; 2002:117-128.

2.Deduplication• Chaudhuri S, Ganti V, Motwani R. Robust Identification of Fuzzy Duplicates.

ICDE’05. 2005:865-876.

3.Dataspaces1. Halevy AY, Franklin M, Maier D. Principles of dataspace systems. In: PODS ’06.

New York, NY, USA; 2006:1-9.

• Interdependence between schema matching and deduplication

1. Zhou X, Gaugaz J, Balke W-T, Nejdl W. Query relaxation using malleable schemas. SIGMOD 2007. Beijing, China; 2007:545-556.