Sar-graphs: A Language Resource Connecting Linguistic Knowledge with
Semantic Relations from Knowledge Graphs
Sebastian Krausea,⇤, Leonhard Henniga, Andrea Morob, Dirk Weissenborna, Feiyu Xua, Hans Uszkoreita,Roberto Naviglib
a
DFKI Language Technology Lab, Alt-Moabit 91c, 10559 Berlin, Germany
b
Dipartimento di Informatica, Sapienza Universita di Roma, Viale Regina Elena 295, 00161 Roma, Italy
Abstract
Recent years have seen a significant growth and increased usage of large-scale knowledge resources in bothacademic research and industry. We can distinguish two main types of knowledge resources: those thatstore factual information about entities in the form of semantic relations (e.g., Freebase), namely so-calledknowledge graphs, and those that represent general linguistic knowledge (e.g., WordNet or UWN). In thisarticle, we present a third type of knowledge resource which completes the picture by connecting the twofirst types. Instances of this resource are graphs of semantically-associated relations (sar-graphs), whosepurpose is to link semantic relations from factual knowledge graphs with their linguistic representations inhuman language.We present a general method for constructing sar-graphs using a language- and relation-independent,
distantly supervised approach which, apart from generic language processing tools, relies solely on theavailability of a lexical semantic resource, providing sense information for words, as well as a knowledge basecontaining seed relation instances. Using these seeds, our method extracts, validates and merges relation-specific linguistic patterns from text to create sar-graphs. To cope with the noisily labeled data arising in adistantly supervised setting, we propose several automatic pattern confidence estimation strategies, and alsoshow how manual supervision can be used to improve the quality of sar-graph instances. We demonstratethe applicability of our method by constructing sar-graphs for 25 semantic relations, of which we make asubset publicly available at http://sargraph.dfki.de.We believe sar-graphs will prove to be useful linguistic resources for a wide variety of natural language
processing tasks, and in particular for information extraction and knowledge base population. We illustratetheir usefulness with experiments in relation extraction and in computer assisted language learning.
Keywords: Knowledge graphs, language resources, linguistic patterns, relation extraction
1. Introduction
Knowledge graphs are vast networks which storeentities and their semantic types, properties andrelations. In recent years considerable e↵ort hasbeen invested into constructing these large knowl-edge bases in academic research, community-drivenprojects and industrial development. Prominentexamples include Freebase [1], Yago [2, 3], DBpe-dia [4], NELL [5, 6], WikiData [7], PROSPERA[8], Google’s Knowledge Graph [9] and also the
Google Knowledge Vault [10]. A parallel and inpart independent development is the emergenceof several large-scale knowledge resources with amore language-centered focus, such as UWN [11],BabelNet [12], ConceptNet [13], and UBY [14].These resources are important contributions to thelinked data movement, where repositories of world-knowledge and linguistic knowledge complementeach other. In this article, we present a methodthat aims to bridge these two types of resources byautomatically building an intermediate resource.
In comparison to (world-)knowledge graphs, theunderlying representation and semantic models oflinguistic knowledge resources exhibit a greater de-
Preprint submitted to Journal of Web Semantics: Science, Services and Agents on the World Wide Web January 27, 2016
gree of diversity. ConceptNet makes use of natural-language representations for modeling common-sense information. BabelNet integrates entity in-formation from Wikipedia with word senses fromWordNet, as well as with many other resources suchas Wikidata and Wiktionary [15]. UWN automat-ically builds a multilingual WordNet from variousresources, similar to UBY, which integrates multi-ple resources via linking on the word-sense level.Few to none of the existing linguistic resources,however, provide a feasible approach to explicitlylinking semantic relations from knowledge graphswith their linguistic representations. We aim to fillthis gap with the resource whose structure we de-fine in Section 2 and whose construction method wedetail in Section 3. Instances of this resource aregraphs of semantically-associated relations, whichwe refer to by the name sar-graphs. Our defini-tion is a formalization of the idea sketched in [16].We believe that sar-graphs are examples for a newtype of knowledge repository, language graphs, asthey represent the linguistic patterns for relationsin a knowledge graph. A language graph can bethought of as a bridge between the language andknowledge encoded in a knowledge graph, a bridgethat characterizes the ways in which a language canexpress instances of one or several relations, andthus a mapping between strings and things.The construction strategies of the described
(world-)knowledge resources range from 1) integrat-ing existing structured or semi-structured knowl-edge (e.g., Wikipedia infoboxes) via 2) crowd-sourcing to 3) automatic extraction from semi- andunstructured resources, where often 4) combina-tions of these are implemented. At the same timethe existence of knowledge graphs enabled the de-velopment of new technologies for knowledge engi-neering, e.g., distantly supervised machine-learningmethods [8, 17, 18, 19, 20]. Relation extractionis one of the central technologies contributing tothe automatic creation of fact databases [10], onthe other hand it benefits from the growing num-ber of available factual resources by using them forautomatic training and improvement of extractionsystems. In Section 3, we describe how our ownexisting methods [18], which exploit factual knowl-edge bases for the automatic gathering of linguisticconstructions, can be employed for the purpose ofsar-graphs. Then in turn, one of many potential ap-plications of sar-graphs is relation extraction, whichwe illustrate in Section 7.An important aspect of the construction of sar-
World Wide Web
Sar-graphs
Lexical semantic resources
Relation-specif ic semantic graphs
Factual knowledge bases
…
is the source of represents linguistic constructions for particular relations
merged into
validate extend
search for mentions of instance tuples
are the basis for
Relation-specif ic dependency structures
subj obj
lex-mod mod
Figure 1: Relation of sar-graphs to other knowledge re-sources.
graphs is the disambiguation of their content wordswith respect to lexical semantics knowledge repos-itories, thereby generalizing content words withword senses. In addition to making sar-graphs moreadjustable to the varying granularity needs of possi-ble applications, this positions sar-graphs as a linkhub between a number of formerly independent re-sources (see Figure 1). Sar-graphs represent linguis-tic constructions for semantic relations from factualknowledge bases and incorporate linguistic struc-tures extracted from mentions of knowledge-graphfacts in free texts, while at the same time anchoringthis information in lexical semantic resources. Wego into further detail on this matter in Section 6.
The distantly supervised nature of the proposedconstruction methodology requires means for au-tomatic and manual confidence estimation for theextracted linguistic structures, presented in Sec-tion 4. This is of particular importance when un-structured web texts are exploited for finding lin-guistic patterns which express semantic relations.Our contribution is the combination of battle-tested confidence-estimation strategies [18, 21] witha large manual verification e↵ort for linguistic struc-tures. In our experiments (Section 5), we continuefrom our earlier work [18, 22], i.e., we employ Free-base as our source of semantic relations and thelexical knowledge base BabelNet for linking wordsenses. We create sar-graphs for 25 relations, whichexemplifies the feasibility of the proposed method,also we make the resource publicly available for thiscore set of relations.
We demonstrate the usefulness of sar-graphs byapplying them to the task of relation extraction,
2
preimage(f) Af Example for ⌃f
V from lexical tokens word form, word lemma, word class, word sense married, to marry, verb, bn:00085614vV from entity mentions entity type, semantic role person, Spouse2E from syntactic parsing dependency labels nsubjpassE from resource linking lexical semantic relation synonym
V [ E frequency in training set 2V [ E identifiers for sentences & dependency structures [sent:16, sent:21], [pat:16#1, pat:21#2]
Table 1: Names and example values for attributes of sar-graph elements.
where we identify and compose mentions of argu-ment entities and projections of n-ary semantic re-lations. We believe that sar-graphs will prove tobe a valuable resource for numerous other applica-tions, such as adaptation of parsers to special recog-nition tasks, text summarization, language genera-tion, query analysis and even interpretation of tele-graphic style in highly elliptical texts as found inSMS, Twitter, headlines or brief spoken queries.We therefore make this resource freely available tothe community, and hope that other parties willfind it of interest (Section 8).
2. Sar-graphs: A linguistic knowledge re-
source
Sar-graphs [16] extend the current range ofknowledge graphs, which represent factual, rela-tional and common-sense information for one ormore languages, with linguistic knowledge, namely,linguistic variants of how semantic relations be-tween abstract concepts and real-world entities areexpressed in natural language text.
2.1. Definition
Sar-graphs are directed multigraphs containinglinguistic knowledge at the syntactic and lexical se-mantic level. A sar-graph is a tuple
G
r,l
= (V,E, s, t, f, A
f
,⌃
f
),
where
• V is the set of vertices,• E is the set of edges,• s : E 7! V maps edges to their start vertex,• t : E 7! V maps edges to their target vertex.
As both vertices and edges are labeled, we alsoneed an appropriate labeling function, denoted byf . f does more than just attaching atomic labels toedges and vertices but rather associates both with
sets of features (i.e., attribute-value pairs) to ac-count for the needed complexity of linguistic de-scription:
f : V [ E 7! P(Af
⇥⌃
f
)
where
• P(·) constructs a powerset,• A
f
is the set of attributes (i.e., attributenames) which vertices and edges may have, and
• ⌃
f
is the value alphabet of the features, i.e.,the set of possible attribute values for all at-tributes.
The information in one instance of such a graphis specific to a given language l and target relationr. In general, r links n � 2 entities wrt. their se-mantic relationship in the real world. An examplerelation is marriage, connecting two spouses to oneanother, and optionally to the location and date oftheir wedding, as well as to their date of divorce:1
r
mar.
(Spouse1,Spouse2,Ceremony,From,To).
The function of sar-graphs is to represent the lin-guistic constructions a language l provides for re-porting instances of r or for just referring to suchinstances. A vertex v 2 V corresponds to a wordin such a construction. The features assigned to avertex via the labeling function f provide informa-tion about lexico-syntactic aspects (word form andlemma, word class), lexical semantics (word sense)and semantic points (global entity identifier, entitytype, semantic role in the target relation). Addi-tionally, they provide statistical and meta informa-tion (e.g., frequency). Table 1 presents an overviewof the possible attributes.
1In the remainder of this article, we refer to the arguments
of semantic relations at times via labels for the arguments(in SmallCaps, e.g., Spouse1) and at other times via theentity types of possible argument fillers (with sans-serif font,e.g., person), depending on the context.
3
The linguistic constructions are modeled as sub-trees of dependency-graph representations of sen-tences. In this article, we refer to these trees asdependency structures or dependency constructions.Each such structure typically describes one particu-lar way to express a semantic relation in a given lan-guage. Edges e 2 E are consequently labeled withdependency tags via f , in addition to frequency in-formation.In the literature, linguistic constructions of this
kind are often referred to as extraction patterns, mo-tivated by the application of such structures for theextraction of relations from sentences. A di↵erenceto sar-graphs is that individual dependency struc-tures may or may not be present in a sar-graph asdisjunct trees, i.e., we merge constructions or partsthereof. The joint representation of common pathsof linguistic expressions allows for a quick identifi-cation of dominant phrases and the calculation offrequency distributions for sub-trees and their com-binations. This merging step is not destructive, theinformation about the linguistic structures found inoriginal sentences is still available. We believe thatfor language expressions, an exhaustive, permanentmerging does not make sense, as it would mean los-ing the language variety which we aim at capturing.The merging process is implemented with a con-
servative default strategy, which cautiously con-nects dependency constructions at their argumentpositions, followed by a customizable second step,which further superimposes nodes and paths in anon-destructive manner. We describe this two stepprocess in Section 3.4. In the remainder of this sec-tion, we want to convey a general intuition of whatsar-graphs are, hence a more abstract and uniformview on the merging process is assumed.We expect that novel constructions emerge in
sar-graphs, coming from the combination of two ormore known phrases. See for example these twophrases, each connecting two arguments of relationmarriage:
• Ann wed in York.
• Ann wed on July 27, 2007.
A joint representation of them in a sar-graph givesus a three-argument dependency structure, corre-sponding to the following sentence, in which boththe location and date argument are attached to theverb, and not just one of them: Ann wed in York
on July 27, 2007.
If a given language l only provides a single con-struction to express an instance of r, then the de-pendency structure of this construction forms the
SPOUSE1 SPOUSE2 husband (noun)
poss dep
conj_and
marry (verb)
nsubjpass nsubjpass
be auxpass
FROM prep_since
I met Eve’s husband Jack.
SPOUSE1 SPOUSE2
poss dep
Lucy and Peter are married since 2011.
SPOUSE1 SPOUSE2 FROM
auxpass nsubjpass
nsubjpass
conj_and prep_since
Figure 2: Sar-graph example generated from two Englishsentences. The sar-graph connects the dependency struc-tures via their shared Spouse arguments and additionallyincludes edges and vertices linking the From argument fromthe second sentence.
entire sar-graph. But if the language o↵ers alter-natives to this construction, i.e., paraphrases, theirdependency structures are also added to the sar-graph. The individual constructions superimposeone another based on shared properties and labelsof vertices and edges. Specifically, we merge
• vertices without a semantic role based on theirword lemma or entity type
• vertices with argument roles wrt. their seman-tic role in the target relation
• edges on the basis of dependency labels.Our data-driven approach to the creation of sar-graphs integrates not just constructions that in-clude all relation arguments but also those men-tioning only a subset thereof. As long as these con-structions indicate an instance of the target rela-tion, they are relevant for many applications, suchas high-recall relation extraction, even though theyare not true paraphrases of constructions fully ex-pressing the n-ary relation.
A sar-graph for the two English constructions inExample 1, both with mentions of projections ofthe marriage relation may look as presented in Fig-ure 2.
4
SPOUSE1
vow (noun)
SPOUSE2
husband (noun)
exchange (verb)
poss
dep
dobj
nsubj
nsubj
conj_and
marry (verb)
nsubjpass nsubjpass FROM
prep_since
ceremony (noun)
wedding (noun)
nn
prep_in
prep_from
prep_of
TO
divorce (verb)
prep_in
nsubjpass
prep_of CEREMONY
prep_in
syn
syn
syn
wedding event
wedding event
wedding party
wedding party
wedding wedding nuptials nuptials
split up split up
hubby hubby
hubbie hubbie
I attended the wedding ceremony of Lucy and Peter in 2011.
Lucy was divorced from Peter in 2012.
I met Eve’s husband Jack.
SPOUSE1 SPOUSE2 poss dep
Lucy and Peter are married since 2011.
SPOUSE1 SPOUSE2 FROM auxpass
nsubjpass nsubjpass
conj_and prep_since
SPOUSE1 SPOUSE2 FROM nn
prep_of prep_of
prep_in
Peter and Lucy exchanged the vows in Paris.
conj_and nsubj dobj nsubj
det
prep_in
SPOUSE2 SPOUSE1 CEREMONY prep_in
prep_from auxpass nsubjpass
TO SPOUSE1 SPOUSE2
syn syn
syn
syn
Figure 3: More complex example for a sar-graph. This graph also includes lexical semantic information (dashed vertices andedges) obtained by linking content words to a lexical semantic resource.
Example 1
• I met Eve’s husband Jack.
• Lucy and Peter are married since 2011.
From the dependency parse trees of these sen-tences, we can extract two graphs that connectthe relation’s arguments. The first sentence liststhe spouses with a possessive construction, the sec-ond sentence using a conjunction. In addition,the second sentence provides the marriage date.The graph we extract from the latter sentencehence includes the dependency arcs nsubjpass andprep since, as well as the node for the content wordmarry. We connect the two extracted structures bytheir shared semantic arguments, namely, Spouse1and Spouse2. As a result, the graph in Figure 2contains a path from Spouse1 to Spouse2 via thenode husband for sentence (1), and an edge conj andfrom Spouse1 to Spouse2 for sentence (2). Thedependency relations connecting the From argu-ment yield the remainder of the sar-graph. Notethat the graph contains two types of vertices: ar-gument nodes labeled with their semantic role, andlexical semantic nodes labeled with their lemma andPOS tag.
Figure 3 illustrates the structure and content ofa more complex sar-graph example, again for themarriage relation. We extend the previous exam-ple with three more sentences, which provide alter-native linguistic constructions, as well as the addi-tional arguments ceremony and To. The graphnow includes the paraphrases exchange vows, wed-ding ceremony of, and was divorced from. Note thatboth sentence (2) and (4) utilize a conj and to con-nect the spouses. The sar-graph includes this in-formation as a single edge, but we can encode thefrequency information as an edge attribute. Thegraph also contains additional lexical semantic in-formation, represented by the dashed vertices andedges (see Section 2.3).
2.2. Less explicit relation mentions
A key property of sar-graphs is that they storelinguistic structures with varying degrees of explic-itness wrt. to the underlying semantic relations.Constructions that refer to some part or aspect ofthe relation would normally be seen as su�cient ev-idence of an instance even if there could be contextsin which this implication is canceled; consider the
5
Relation marriage SpouseA Brad Pitt SpouseB Jennifer Aniston Ceremony Malibu From 2000/07/29 To 2005/10/02
Sar-graphs Dependency constructions Seeds
• Fact search in Web • Linguistic preprocessing • Sense disambiguation • Mention detection
Relation mentions
Relation SpouseA SpouseB Ceremony From To
SPOUSE1 SPOUSE2
obj marry (verb) subj
SPOUSE1 SPOUSE2
obj marry (verb) subj
✔
Verif ied constructions
• Structure merge
• Validation of candidates:
! Automatically ! Expert-driven
• Extraction of relation-relevant dependency paths
Figure 4: Outline of sar-graph construction. Arrows correspond to processing steps, while boxes show intermediate results.
sentences in Example 2:
Example 2
• Joan and Edward exchanged rings in 2011.
• Joan and Edward exchanged rings during the
rehearsal of the ceremony.
Other constructions refer to relations that entailthe target relations without being part of it:
Example 3
• Joan and Edward celebrated their 12th
wedding anniversary.
• Joan and Edward got divorced in 2011.
And finally there are constructions referring tosemantically connected relations that by themselvesmight not be used for safely detecting instances ofr, but that could be employed for recall-optimizedapplications or for a probabilistic detection processthat combines several pieces of evidence:
Example 4
• I met her last October at Joan'sbachelorette (engagement) party.
Some entirely probabilistic entailments arecaused by social conventions or behavioral prefer-ences:
Example 5
• Two years before Joan and Paul had their
first child, they bought a larger home.
2.3. Graphs augmented with lexical semantics
The lexico-syntactic and semantic informationspecified in sar-graphs is augmented with lexicalsemantic knowledge by disambiguating all contentwords in the dependency structures. This results in
a labeling of content word vertices with sense iden-tifiers and additional (synonymous) surface formsfor the sar-graph vertices and also implicit lexicalsemantic links among words already contained inthe sar-graph. These implicit links bear tags suchas hypernym, synonym, troponym, or antonym.
In the sar-graph of Figure 3, additional sur-face forms are illustrated by dashed vertices andedges. For example, for the vertex representingthe lemma husband, the colloquial synonyms hubbyand hubbie are listed.
Among the benefits of this injection of lexical-semantic information into sar-graphs is a largeramount of resulting paraphrases. In sar-graph ap-plications like relation extraction, additional para-phrases lead to a higher number of detected relationmentions. Furthermore, the disambiguation infor-mation allows us to employ a sophisticated confi-dence estimation method for the underlying depen-dency constructions, which we describe in Section 4.With these confidence assessments, we can reliablyidentify the constructions in a sar-graph which mayonly entail the target relation of interest, in contrastto those explicitly expressing it.
3. Sar-graph construction
In this section, we describe a general method forconstructing sar-graphs. Our method is language-and relation-independent, and relies solely on theavailability of a set of seed relation instances froman existing knowledge base. Figure 4 outlines thisprocess. Given a target relation r, a set of seedinstances I
r
of this relation, and a language l, wecan create a sar-graph G
r,l
with the following pro-cedure.
6
1. Acquire a set of textual mentions Mr,l
of in-stances i for all i 2 I
r
from a text corpus.
2. Extract candidate dependency constructionsD
r,l
from the dependency trees of elements ofM
r,l
.
3. Validate the candidate structures d 2 Dr,l
, ei-ther automatically or via human expert-drivenquality control, yielding a derived set D0
r,l
ofacceptable dependency constructions.
4. Merge elements of d 2 D0r,l
to create the sar-graph G
r,l
.
We discuss each of these steps in more detail inthe following sections.
3.1. Textual mention acquisition and preprocessing
The first step in the processing pipeline is to col-lect a large number of textual mentions of a giventarget relation, ideally covering many di↵erent lin-guistic constructions used to express the relation.Following [18, 23, 24], we collect textual mentionsusing as input a set of seed instances I
r
of the tar-get relation r. Every sentence which contains theentity tuples of the seed instances is regarded as atextual mention of the relation. As in standard dis-tantly supervised approaches, this seed instance setcan be easily obtained from an existing knowledgebase (KB).The seeds are used as queries for a web search
engine to find documents that potentially containmentions of the seeds. We construct a separatequery for each seed by concatenating the full namesof all seed argument entities. Documents returnedby the search engine are downloaded and convertedinto plain text, using standard methods for HTML-to-text conversion and boilerplate detection.We then perform standard NLP preprocess-
ing of the text documents, including sentencedetection, tokenization, named-entity recognition(NER), lemmatization, part-of-speech tagging, us-ing o↵-the-shelf tools. To enable a better under-standing and exploitation of the extracted depen-dency structures, we link their relevant elements(i.e., the content words) to a lexical-semantic re-source. We also link entity mentions to seed enti-ties with a simple dictionary-based linking strategythat matches name variations of the seed’s entitiesas provided by the KB.We discard all sentences not mentioning a seed
instance, as well as sentences not expressing all es-sential arguments of the relation. Which arguments
of a relation are essential or optional is defined a-priori by the user. The remaining sentences are pro-cessed by a dependency parser outputting Stanforddependency relations2 [25]. We use the output ofthe NER tagger to generalize the dependency parseby replacing all entity mentions with their respec-tive NE tags.
3.2. Dependency structure extraction
The next step of the sar-graph construction pro-cess is to extract candidate dependency structuresdenoting the target relation from the full depen-dency parse trees of the source sentences. Typi-cally, shortest path or minimum spanning tree al-gorithms are used to select the subgraph of the de-pendency tree connecting all the arguments of therelation instance mentioned in a given sentence [23].In [22], we present an alternative, knowledge-drivenalgorithm which employs a large lexical semanticrepository to guide the extraction of dependencystructures. The algorithm expands the structure toinclude semantically relevant material outside theminimal subtree containing the shortest paths, andalso allows us to discard structures without any ex-plicit semantic content (e.g., highly ambiguous ap-pos constructions).
Figure 5 shows an example source sentence,along with a shortest-path dependency structureextracted from its parse tree. The example sen-tence (5a) mentions an instance of the marriagerelation with the arguments ⟨Brad Pitt, JenniferAniston, Malibu, 2001/07/29⟩. In the figure, ar-gument fillers are underlined. Figure 5b depictsthe output of the dependency-structure extractionstep. This structure is then generalized by replacingwords with their lemmas, deriving coarse-grainedpart-of-speech tags, and replacing entity mentionswith their respective NE tags (5c). We discard allstructures which do not contain at least one con-tent word, such as a verb, noun or adjective. Westore word sense information for all content wordsas a property of the extracted dependency structure(not shown in the figure).
The use of the dependency-relation formalismfor constructing sar-graphs is an important designchoice. We assume that any given mention of atarget-relation instance can be identified by a some-how characteristic pattern in the sentence’s under-
Brad Pitt married Jennifer Aniston in a private wedding ceremony in Malibu on July 29, 2000.
(a) Sentence with a mention of the marriage relation.
marriednsubj
tt dobj ✏✏prep in
$$
prep on
''Brad Pitt Jennifer Aniston ceremony
prep in
✏✏
July 29, 2000
Malibu
(b) Dependency parse of (a part of) (a).
2
66666666666666666666666664
head
"lemma marryPOS V
#
dobj
"type person
role Spouse2
#
prep in
2
66664
head
"lemma ceremonyPOS N
#
prep in
"type location
role Ceremony
#
3
77775
prep on
"type date
role From
#
nsubj
"type person
role Spouse1
#
3
77777777777777777777777775
(c) Generalized dependency construction derived from (b);
WSD information omitted for clarity.
Figure 5: Data flow for gathering candidate dependency constructions from distantly labeled text.
lying dependency graph. This approach has lim-itations, e.g., it does not cover mentions requir-ing some kind of semantic understanding, or men-tions with arguments spread across several sen-tences [26, 27], but it has been shown to work wellin general [24, 28].
3.3. Dependency structure validation
Our approach to extracting relation-specific de-pendency structures is based on a distantly super-vised learning scheme. Distant supervision makesseveral strong assumptions that may significantlya↵ect the quality of the set of learned dependencystructures. First, it assumes that for every rela-tion tuple r
. Thisassumption typically does not hold for most sen-tences, i.e., the entity mentions may co-occur in asentence without it actually expressing the relation.Extracted dependency structures may therefore beirrelevant or even wrong for a given relation, andshould not be included in its sar-graph. Further-more, distant supervision implicitly assumes thatthe knowledge base is complete: entity mentionswithout known relations are ignored during extrac-tion. This may result in a loss of recall (of lessfrequent dependency structures), and in a bias ofextracted dependency structures towards popular
relations and entities.The goal of the next step of sar-graph construc-
tion is therefore the validation of the quality ofthe candidate dependency structures. Validationcan be performed automatically, e.g., by comput-ing confidence values or similar metrics for each de-pendency structure, or by manually verifying struc-tures. Candidate structures that have a low confi-dence score, or that are rejected during manual ver-ification, are discarded. The remaining set of val-idated dependency structures is the output of thisprocessing step.
We present and discuss several approaches to au-tomatically and manually estimating the quality ofcandidate dependency structures in Section 4.
3.4. Dependency structure merging
Given the set of validated dependency construc-tions, we superimpose these structures onto one an-other to create a sar-graph. We follow a technicallystraight-forward approach to sar-graph creation bymerging dependency constructions step-wise intolarger graphs, based on the equality of propertiesof the graph elements. Initially, this process cre-ates a graph by only merging argument nodes, whileotherwise retaining the independence of structures.Figure 6 presents two functions in pseudocode thatoutline this first step. The input to the functioncreateSarGraph is the set of dependency struc-
8
Function name: createSarGraph
Input: DependencyConstruction[] dcsOutput: a SarGraph
// Each dependency construction is a weakly connected, directed, simple graph.3 for each edge e in dc from n
1
to n
2
:
4 e
0 new edge5 sg.E sg.E [ { e0 }6 update function sg.s: Set sg.s(e0) to result of addNode(sg,n
1
)7 update function sg.t : Set sg.t(e0) to result of addNode(sg,n
2
)6 update function sg.f : Set sg.f (e0) to attributes of e7 return sg
Function name: addNode
Input: SarGraph sg,Node n
Output: a Node
1 if n 2 sg.V then :
2 return n
3 elseif 9n 0 2 sg.V | n,n 0 are derived from entity mentions^n,n 0 share entity type and argument role information then :
4 update function sg.f : Extend sg.f (n 0) with attributes of n5 return n
0
6 else :
7 sg.V sg.V [ {n }8 update function sg.f : Set sg.f (n) to attributes of n9 return n
10 endif
Figure 6: Pseudocode outlining the creation of a sar-graph from a set of dependency constructions. f ,Af
,⌃f
are defined inSection 2. Nodes and edges of dependency constructions have the same attributes as sar-graph elements; see Table 1 for a list.
tures accepted by the previous validation step. Thesar-graph is built by subsequently adding structuresto the graph, one edge at a time. Whenever anode is to be added to the graph, it is first verifiedthat the node is not already contained in the graphand checked whether there is a matching argumentnode present, in which case the history informationof the currently handled node (identifiers of sourcesentences and dependency structure, statistical in-formation) is merged with the information of theexisting node. If neither is the case, the node isadded to the sar-graph.
In order to deal with task-specific needs for thegranularity of information in a sar-graph, applica-tions can view sar-graphs at varying detail levels.For the task of relation extraction (see Section 7),the coverage of the original patterns is already veryhigh [18], and merging paths would trade o↵ higherrecall with lower precision. Thus, the employedview does not impose any additional merging re-quirements and is identical to the originally con-structed sar-graph. Figure 8b illustrates this strat-egy with a sar-graph constructed from the three ex-
ample sentences shown in Figure 8a. The resultingsar-graph resembles the union of the original set ofdependency structures, i.e., each path through thegraph has a frequency of one.
For analysis purposes, e.g., for carrying out anexploratory analysis of the linguistic expressionsused to express particular target relations, a morecondensed representation is advantageous. Thepseudocode in Figure 7 shows the general work-flow of the generation of sar-graph views in func-tion createCondensedSarGraphView. Func-tions exampleNodeCompressor and example-
EdgeCompressor provide a custom implementa-tion for the merging of nodes and edges. Two nodesare combined if they contain the same lexical in-formation, likewise, edges between equal nodes arecombined if the dependency labels attached to theseedges are the same. In an application where a greatnumber of linguistic expressions will be inspected,a user is likely just interested in a coarse-graineddistinction of word classes, which is why example-
NodeCompressor generalizes the part-of-speechtags of all lexical nodes.
// Generalize part-of-speech tag of n.4 update function sg.f : Replace (“word class”, p) 2 sg.f (n) with (“word class”, upcast(p))5 if 9n 0 2 sgView .V | n 0 is derived from a lexical token
^n,n 0 share word form, word lemma, and word class then :
6 update function sgView .f : Merge sg.f (n) into sgView .f (n 0)7 return n
0
8 endif
9 endif
// Neither is n contained in sgView, nor is there an equivalent node.10 sgView .V sgView .V [ {n }11 update function sgView .f : Set sgView .f (n) to sg.f (n)12 return n
Function name: exampleEdgeCompressor
Input: SarGraph sg, SarGraph sgView ,Edge e,Node n
1
,Node n
2
Output: none1 if 9e0 2 sgView .E | e0 originates from syntactic parsing
^ sgView .s(e0) = n
1
^ sgView .t(e0) = n
2
^ e, e0 have the same dependency label2 update function sgView .f : Merge sg.f (e) into sgView .f (e0)3 else :
4 e
0 new edge5 sgView .E sgView .E [ { e0 }6 update function sgView .s: Set sgView .s(e0) to n
1
7 update function sgView .t : Set sgView .t(e0) to n
2
8 update function sgView .f : Set sgView .f (e0) to sg.f (e)9 endif
Figure 7: Pseudocode for producing a condensed view of a sar-graph, tailored for applications. f ,Af
,⌃f
are defined in Section 2.In this example, the call createCondensedSarGraphView(sg, exampleNodeCompressor, exampleEdgeCompressor)generates a sar-graph suited for manual explorative analysis of linguistic phrases. The produced graph uses a coarse-grainedinventory of part-of-speech tags. The function upcast() generalizes a given tag, e.g., it maps verb classes (verb in past tense,verb in 3rd person singular present, . . . ) to a single base verb class.
10
Br ad Pi t t mar r i ed Jenni f er Ani st on i n a pr i vat e weddi ng cer emony i n Mal i bu on Jul y 29, 2000.
I n 1983, Depp mar r i ed Lor i Anne Al l i son. Paul Newman was mar r i ed t o Jacki e Wi t t e i n 1949.
dobjprep_in
prep_onnsubj
nsubjprep_in
dobjauxpass
nsubjpass
prep_to
prep_in
( 3)( 2)
( 1)
(a) Example sentences and dependency structures.
PER PER
marry (1)
DATE LOC
be
marry (2)
marry (3)
nsubj
dobjprep_on
prep_in
dobj
nsubj
prep_in
prep_tonsubjpass
auxpass
prep_in
(b) A sar-graph retaining the independence of origi-
nal structures.
PER PER
marry
DATE
LOC
be
nsubjdobj prep_on
prep_in
nsubjpass
auxpass
prep_inprep_to
(c) A more condensed representation of linguistic
phrases.
Figure 8: Two di↵erent sar-graph views created from the same three sentences..
This strategy merges all nodes and vertices thatare equal according to the above definition. Struc-tures that fully or partially overlap (even with justa single edge or node) are merged. This couldmean that in the resulting sar-graph, some of thepaths connecting argument nodes are linguisticallyinvalid. The frequency of a dependency edge inthe sar-graph is equal to the number of dependencystructures containing that edge. Since the same de-pendency structure can appear multiple times inthe source data, with di↵erent arguments and / orcontext, we represent word sense information as afrequency distribution (over senses of content wordsfor a given dependency structure). This approachenables a more flexible and noise-resistant annota-tion of word senses for the context words used toexpress the target relation. Figure 8c shows an ex-ample sar-graph created with this strategy.
In order to cope with applications which require
a di↵erent balance of detail vs. generalization ofthe various sar-graph elements, all one has to do isto provide matching implementations of functionsexampleNodeCompressor and exampleEdge-
Compressor. For example, dependency structurescould be generalized by merging all vertices be-longing to the same synset in a lexical-semantic re-source, ignoring di↵erences on the lexical level.
4. Quality control
As discussed in the previous section, our ap-proach to sar-graph construction uses distant super-vision for collecting textual mentions of a given tar-get relation. In this section, we present several ap-proaches to automatically compute confidence met-rics for candidate dependency structures, and tolearn validation thresholds. We also describe anannotation process for manual, expert-driven qual-ity control of extracted dependency structures, and
11
briefly describe the linguistic annotation tool andguidelines that we developed for this purpose.
4.1. Automatic ways of quality improvement
4.1.1. Data-oriented path-quality estimation
Semantic relations coming from the same domainmight have a similar entity-type signature, in par-ticular, they might share the types of their essentialarguments. For example, numerous semantic rela-tions can be defined for the great variety of wayspersons interact and relate to one another. When-ever two relation definitions are similar in this par-ticular way, we say they are of the same essentialtype.Relations of the same essential type may have
some instances in common, for example, the sametwo persons might be involved in various relationssuch as marriage and romantic relationships. Thiscan be the case, for example, if the relations over-lap, or if the relevant linguistic expressions are am-biguous. Most dependency constructions we findfor two or more relations, however, are not appro-priate for one or both relations. Such constructionsmight be learned for wrong relations because of er-roneous entity recognition and dependency parsing,false seed facts, or false statements of a relation intexts. Especially when we extract the same depen-dency construction for two disjoint relations, some-thing must be wrong. Either the construction ex-hibits a much higher frequency for one of the two re-lations, then it can be safely deleted from the other,or we consider it wrong for both relations.In [18] we proposed a general and parameteriz-
able confidence estimation strategy for dependencystructures using information about their frequencydistribution wrt. other relations of the same essen-tial type. If a construction occurs significantly moreoften in a relation r than in another relation r
0, thisconstruction probably expresses r in contrast to r
0.Let D
r,l
be the set of extracted dependency struc-tures for r and language l, and let f
r,l
(d) denotethe frequency of dependency structure d in r, l (i.e.,the number of sentences for relation r and languagel from which d has been extracted). We define therelative frequency of d for r, l as:
rfr,l
(d) = f
r,l
(d)
,X
d
02Dr,l
f
r,l
(d0) (1)
Let R be a set of relations of the same essen-tial type. The dependency structure d most likely
expresses the relation r 2 R in l if the relative fre-quency of d in r, l is higher than its relative frequen-cies for all other relations in R (i.e., if 8r0 2 R\{r} :rf
r,l
(d) > rfr
0,l
(d)). Because this judgment aboutthe semantic meaning of a dependency structuredepends much on the specific set of selected targetrelations, we do not use it for the construction ofsar-graphs in general, however, it proves useful forparticular applications (Section 7).
Instead of excluding dependency structures withthis relation-overlap heuristic, we augment individ-ual dependency paths in the sar-graphs with infor-mation about their frequency wrt. a single relation.This allows applications to pick certain sub-parts ofthe sar-graphs for which there is much support inthe training data, i.e., evidence that a structure be-longs to the target relation from an absolute pointof view. Depending on application needs, this canbe combined with information from the relation-overlap criterion.
4.1.2. Utilizing external resources
Another automatic quality estimator for relation-specific dependency structures can be definedthrough the construction of the so-called relation-specific semantic graphs [21]. The considered de-pendency structures, and consequently the sar-graphs, already contain semantic information thatcan be exploited for di↵erent tasks (see Section 2.3).In this section, we show how we use this informationto improve the quality of the generated sar-graphs.In comparison with statistical methods, the use ofsemantic analysis can better handle cases of high-frequency structures which do not express the con-
sidered relation (e.g., personsubj ��� met
obj��! person
for the relation marriage) and also in cases of lowfrequency structures which are indeed semanticallyrelevant for the considered semantic relation (e.g.,
person
poss ��� widower
appos����! person for the same re-lation).
Given the frequency distributions of content wordmeanings (i.e., senses) encoded within the depen-dency structures, we can produce an overall fre-quency distribution of all the considered meaningsfor a relation r. Then, thanks to the links to alexical-semantic resource, we can induce a seman-tic graph from it which contains the most relevantmeanings for the considered relation.
More precisely, we first get the top-k most fre-quent meanings (i.e., the core senses of the rela-tion) from the overall distribution of meanings. For
12
marry
1v
wife
1n
husband
1n
marriage
1n
divorce
1n
divorce
2v
Figure 9: An excerpt of the semantic graph associated withthe relation marriage with k = 2.
example, with k = 2 and the relation marriage,the core meanings are {marry1
v
, wife
1n
}.3 Then, weadd all the remaining meanings in the overall dis-tribution if and only if they are connected to atleast one of the core meanings. For example, withk = 2 and the relation marriage, we add husband
1n
,marriage
1n
and divorce
2v
to the graph, among oth-ers, but we do not add meet
1v
. In this manner weare able to extract a set of highly-relevant meaningsfor the considered relation (see Figure 9 [21] for anexcerpt of the semantic graph for the marriage re-lation). These highly-relevant meanings likely con-stitute most of the senses of the lexical-semanticresource which are useful for expressing the targetrelation in natural language.Finally, to filter out dependency structures which
do not contain any relation-relevant lexical seman-tic elements, we check if any of the dependencystructure’s content words matches a lexicalizationof the meanings contained in the semantic graph.If that is the case we mark it as a good structure,otherwise we filter it out. For instance, our filter
recognizes person
subj ��� married
obj��! person as a
good rule, while it filters out personsubj ��� met
obj��!person because it does not match any lexicalizationsof the meanings contained in the semantic graph.By generating relation-specific semantic graphs
for various values of k and repeatedly applying thecorresponding filter, we can estimate the degree ofrelevancy for all dependency structures. If a struc-ture is accepted by the filter with a low k it is morelikely to express the relation than a structure onlyaccepted at a greater value of k. When construct-ing the sar-graph from the individual dependencystructures, we choose not to filter out any struc-tures, but rather associate the information aboutthe filter output with them.
3For ease of readability, in what follows we use senses todenote the corresponding synsets. We follow [29] and denotewith wi
p the i-th sense of w with part of speech p.
4.2. Expert-driven quality control
The automatic estimation of dependency-structure quality described in the previous sectionis limited to statistical / distributional metricsand to a metric based on the lexical semanticsof words appearing in the structure. These met-rics, however, tell us only very little about the(grammatical) correctness and semantic appropri-ateness of the dependency structures themselves.Therefore, we developed a process for a manual,intrinsic evaluation of the learned dependencystructures. This expert-driven quality controlhas two major goals: to validate the structuresselected by automatic means for the subsequentconstruction of sar-graphs, and to identify commonclasses of extraction errors. In this section, wedescribe the tools and guidelines that we developedfor the manual evaluation process.
4.2.1. Selection of dependency structures
Since the number of possible dependency struc-tures expressing a given relation is potentially un-bounded, a complete manual evaluation would betoo resource-intensive. We therefore limit theexpert-driven quality control to a subset of struc-tures, as chosen by the following process: Foreach relation and dependency structure, we firstcompute an automatic quality metric (e.g., thesemantic-graph score presented in the previous sec-tion), and also determine the structure’s relation-specific occurrence frequency in a large web corpus.Per relation, we experimentally determine thresh-old values for these two measures to exclude low-confidence and low-frequency structures (see Sec-tion 7). We then sample a small set of sentencesfor each structure, and conduct an initial pass overthe data with human annotators that judge whetherthese sentences express the target relation or not.We discard all dependency structures whose sen-tences do not express the target relation. The man-ual evaluation dataset is then created from the re-maining dependency structures. For each structureand relation, the final dataset comprises all sourcesentences and not just the ones sampled for the ini-tial judgments.
4.2.2. Quality control guidelines
Based on an initial, exploratory analysis of thedataset, we define three qualitative categories,“correct”, “correct, but too specific” and“incorrect”, as well as a set of annotation guide-lines for the evaluation of dependency structures.
13
We label a relation-specific structure as correct(i.e., as useful for integration into a sar-graph) if it isgrammatically and semantically correct. A depen-dency structure is grammatically correct if there areno parsing or other preprocessing errors, and it issemantically correct if its source sentences expressthe target relation. Correspondingly, we label a de-pendency structure as incorrect if it is grammat-ically incorrect, or if it does not express the targetrelation. Typically, the annotators aim to identifyone or more of the error classes described in Sec-tion 5.4 to decide whether a pattern is incorrect.
For deciding whether a sentence expresses a givenrelation, we use the ACE annotation guidelines’conceptual definition of relations and their men-tions [30], and define the semantics of relationsbased on Freebase descriptions (see Section 5). Incontrast to the ACE tasks, we also consider n-ary relations in addition to binary relations. Inthe course of this evaluation, sentences must ex-press the target relation explicitly, e.g., “X won theY award” explicitly expresses the relation awardhonor. We treat implicit mentions as semanticallyincorrect, e.g., “X won the Y award” does not implythe relation award nomination as this implicationrequires knowledge about relation entailments. Adependency structure that captures only a subsetof all arguments mentioned in a sentence (e.g., itcovers only one of several children of the same par-ent listed in the same sentence) is still consideredcorrect.
A third category, correct, but too spe-cific, was added based on our initial analysis ofthe dataset, and applies to dependency structuresmostly found in the long tail of the frequency distri-bution. Too specific structures, while both gram-matically and semantically correct, are structuresthat are overly complex and/or include irrelevantparts of the sentence specific to a particular rela-tion instance. Figure 10 shows an example struc-ture, which includes the head word voice. Suchdependency structures do not generalize well, andare hence unlikely to be very “productive” for manyapplication tasks (e.g., they are unlikely to yieldnovel relation instances when applied to additionaltext). The distinction between correct and cor-rect, but too specific is often not clear-cut;to improve the consistency of annotation decisions,we collected illustrative examples in the annotationguidelines.
2
6666666666666664
head
"lemma voicePOS NN
#
conj
2
66664
head
"lemma brotherPOS NN
#
prep of
"type person
role Sibling2
#
3
77775
nsubj
"type person
role Sibling1
#
3
7777777777777775
Figure 10: “Correct, but too specific” dependencystructure extracted from the sentence “Jansen Panettiereis an American voice and film actor, and the youngerbrother of actress Hayden Panettiere.” for the relationsibling relationship.
4.2.3. Evaluation tool - PatternJudge
To facilitate the manual evaluation of depen-dency structures, we have developed a simple anno-tation tool, dubbed PatternJudge. With Pattern-Judge, annotators can inspect dependency struc-tures (patterns) and their associated source sen-tences (learning tracks), and evaluate the struc-tures’ grammatical and semantic correctness.
Figure 11 shows a screen shot of the user inter-face. The interface is split into three main com-ponents. The left part displays the list of re-lations and patterns available for judgment, andallows searching for specific pattern or sentences.The center part visualizes the currently selecteddependency structure in attribute-value-matrix no-tation, and lists the source sentences this struc-ture was observed in. The annotation tab on theright-hand side collects the human expert’s feed-back on the quality of this pattern. Current op-tions include labeling the pattern as “correct”,“correct, but too specific”, “incorrect” or“uncertain/don’t know”. In addition, annota-tors can provide a comment. Comments are mainlyused for discussion and clarification, but also foradding error class information in cases where theannotator decided to label a pattern as incorrect.All pattern judgments are persisted in a database.The tool includes a simple user management, whichenables keeping track of di↵erent annotators, andundoing or updating previous judgments (which isparticularly useful in the early stages of pattern ex-ploration and analysis).
5. Implementation
So far, we have described our methodology forcreating the proposed resource of combined lexical,
14
Figure 11: User interface of the PatternJudge tool. The tool allows annotators to judge the quality of automatically learneddependency structures.
syntactic, and lexical semantic information. In thissection, we outline the concrete experiments carriedout to compile sar-graphs for 25 semantic relations.
5.1. Leveraging factual knowledge
We kick-o↵ the construction process by leverag-ing factual knowledge from Freebase, a large knowl-edge base containing millions of assertions aboutnumerous entities such as people, locations, orga-nizations, films, and books. For our experiments(see also [18]) we focus on a set of 25 relations fromthe domains Award, Business, People for which weexpect to find mentions in human texts and whichwe deem fruitful wrt. application scenarios.Table 2 lists the target relations along with their
entity-type signature, grouped by solid horizontallines wrt. their domain. Essential arguments aremarked by ~. All relations from a domain have theentity type of the first essential argument in com-mon. If two such relations share the entity type ofanother essential argument, we say that they are ofthe same essential type. For example, the followingBusiness domain relations all belong to the sameessential type since their first two arguments allowentities of type organization:
All relation definitions used in this paper are basedon the data available in Freebase. By utilizing Free-base’ query API, we retrieved several thousand in-stances per target relation, yielding a total of 233Kseeds for the 25 target relations. Table 3 lists thedistribution of seeds per relation.
5.2. Creating sar-graphs from web text
The next step was concerned with the acquisi-tion of a corpus of relation-mention examples. Wedecided against using an o✏ine dataset (e.g., aWikipedia crawl) because the processing of suchwould restrict collected phrases to a particular styleof writing. Furthermore, the infamous long-tailproblem of linguistic expressions would make find-ing sentences for less prominent facts improbable.Instead, we directly accessed the Web via a searchengine.
Search-engine querying. The relation instancesfrom Freebase were transformed to search-enginequeries and submitted to Bing. We stopped thequerying of Bing early in case one million search re-sults per relation had already been retrieved. This
15
Relation Slot 1 Slot 2 Slot 3 Slot 4 Slot 5
award nomination ~ prize ~ org/per date — —award honor ~ prize ~ org/per date — —
country of nationality ~ per ~ loc — — —education ~ per ~ org degree area date
marriage ~ per ~ per loc (Ceremony) date (From) date (To)person alternate name ~ per ~ per — — —person birth ~ per (~) loc (~) date — —person death ~ per (~) loc (~) date (~) cause —person parent ~ per (Person) ~ per (ParentA) per (ParentB) — —person religion ~ per ~ religion — — —place lived ~ per ~ loc date (From) date (To) —sibling relationship ~ per ~ per — — —
acquisition ~ org (Buyer) ~ org (Acquired) org (Seller) date —business operation ~ org ~ business space — — —company end ~ org (~) date (~) termination type — —company product relationship ~ org ~ product date (From) date (To) —employment tenure ~ org ~ per position date (From) date (To)foundation ~ org (Org) ~ org/per (Founder) loc date —headquarters ~ org ~ loc — — —organization alternate name ~ org ~ org — — —organization leadership ~ org ~ per position date (From) date (To)organizationmembership ~ org (Org) ~ loc/org/per (Member) date (From) date (To) —organization relationship ~ org (Parent) ~ org (Child) date (From) date (To) —organization type ~ org ~ org type — — —sponsorship ~ org (Sponsor) ~ org/per (Recipient) date (From) date (To) —
Table 2: Definition of the 25 target relations of the domains Award, Business and People. ~ denotes the essential argumentsof the relation, i.e., the core part of a relation instance defining its identity. (~) marks alternatives for essential arguments.loc/org/per are short for location/organization/person. Labels for arguments (in SmallCaps) omitted in unambiguous cases.
Table 3: Statistics for various steps of the sar-graph creation process, as well as for the produced sar-graphs. #doc. refers tothe number of web documents in which a relation mention was found; no duplicate detection was performed. #sent. statesthe count of duplicate-free sentences with a relation mention, #struct. the number of unique dependency structures learnedfrom these sentences. #nodes/#,edges corresponds to the number of respective elements in the sar-graphs created from thestructures in column five.
16
limit was arbitrarily chosen, as at this point of thesystem run, no data was available on how muchsearch results would be needed to create a satisfac-torily large text corpus for a relation. One millionseemed both large enough for generating enoughsamples and small enough to keep the needed pro-cessing time at a reasonable level. To reach thelimit, for some relations not all the facts stored inFreebase had to be utilized. This is particularlytrue for the relations of the People domain, in con-trast to the domains Award and Business. A pos-sible explanation is that there are less web pagesdealing with Award and Business related topicsthan there are for People relations. It might also bethe case that the instances of the former domainsare simply less prominent in current web pages andmore of historical character. This concurs with anon average greater absolute number of instances forPeople relations, which is not surprising given thatFreebase (in part) utilized Wikipedia for gatheringknowledge.4
Text retrieval and entity recognition. The searchresults were subsequently processed by download-ing the respective web page and extracting plaintext from the HTML source code. This process suf-fered from various problems, leading to a fractionof “lost” documents up to 40% for some relations(e.g., person death). Among the reasons for thelosses are problems when accessing web pages (con-nection timeouts, . . .), issues when extracting textfrom them (malformed HTML code, insu�cienciesof text-extraction algorithm), and web pages whichdid not contain any article text at all.After the creation of the text corpus, we ran the
previously outlined entity-recognition componentson it to find occurrences of named entities, in par-ticular those of the respective documents sourceseed. For the recognition of coarse-grained types(i.e., persons, organizations and locations, we reliedon the Stanford Named Entity Recognizer as partof the Stanford CoreNLP package5 [31] and sup-plemented this with our own date recognizer. Toidentify the seed entities, we implemented a simplegazetter-based recognizer with the name variationsof the seeds’ entities as provided by Freebase.
4Freebase contains 3M topics and 20M facts for the do-main People, more than for the domain Business (1M topicsand 4M facts). Retrieved from http://www.freebase.com/on 2015/03/25.
5http://nlp.stanford.edu/software/corenlp.shtml
Relation-mention detection. Only for a relativelysmall fraction of the search-result web-page ad-dresses we got from Bing, we eventually end upwith a plain-text document in which we detectedan intra-sentential relation mention. The actualnumber of such documents per relation is given incolumn three of Table 3. Documents which wereclassified as not being written in English accountfor a large fraction of the successfully downloaded,but still unproductive documents. By far the mostdocuments fail because for at least one essentialargument of the source seed no entity occurrencewas found anywhere in the text. This means thatthe search engine returned results which indeed didnot contain all essential seed arguments, despite thequeries always containing them. Another explana-tion for documents without a mention is of coursethat the NER component was unable to locate theseeds argument in the text.
Column four of Table 3 lists the number of uniquesentences with relation mentions we extracted perrelation. The large di↵erence to the number ofdocuments with an intra-sentential relation men-tion can be attributed to (a) duplicate and near-duplicate documents retrieved from the Web6 and(b) errors in the sentence-processing pipeline.7 Alsonote that these sentences are duplicate-free.
The average number of mention-containing sen-tences per document is approximately 1.5. Thisis reasonable given the underlying assumption [17]that any such sentence indeed expresses the targetrelation. A higher number of sentences might comewith a low quality of the training examples, as forsome relations it is not likely that an entity tupleis referred to multiple times within the same docu-ment as an instance of this very relation.
Dependency structure extraction and merge. Af-ter the identification of sentences containing men-tions, the sentences were processed by a de-pendency parser (MaltParser8, MDParser9) out-putting Stanford dependency relations, followed bythe extraction of the minimum spanning tree con-taining all the seed’s arguments present in the sen-tence. Trees are also extracted for all argument
6Multiple URLs for the same web page, web pages justdi↵ering in boilerplate/navigational elements, extensive re-use of paragraphs from old articles or from press agencies.
7Garbled sentences, overly long sentences, parser errorsin next step, conflicts between NER and parser output.
8Release v1.7.2, engmalt-linear model v1.7, http://www.maltparser.org/
9http://mdparser.sb.dfki.de/
17
subsets where at least two essential arguments arepresent in the sentence, i.e., projections of the de-pendency tree which corresponds to the full set ofarguments are extracted as well. The dependencystructures were then assembled into the sar-graph,as described in Section 3.4. The final sar-graphs forthe 25 relations range in size from 1k to 25k verticesand 3K to 170k edges each.10
5.3. Augmenting the sar-graphs with lexical-semantic information
We grounded the sar-graphs’ dependency struc-tures by linking them against the lexical-semanticresource BabelNet.
BabelNet. BabelNet [12] is a large-scale multilin-gual semantic network which, di↵erently from themanually created WordNet [32], was automaticallybuilt through the algorithmic integration of re-sources like Wikipedia andWordNet, among others.Its core components are so-called Babel synsets,which are sets of multilingual synonyms. EachBabel synset is related to other Babel synsetsvia semantic relations obtained from WordNet andWikipedia, such as hypernymy, meronymy and se-mantic relatedness. Moreover, since BabelNet isthe result of the integration of lexical resources andencyclopedic resources, it goes exactly in the samedirection of the multilingual linguistic Linked OpenData project [33] which consists of a vision of theSemantic Web in which a wealth of linguistic re-sources are interlinked to each other to obtain a big-ger and optimal representation of knowledge [34].BabelNet contains roughly 13M synsets, 117M
lexicalizations and 354M relation instances. Giventhe multilingual nature of BabelNet (it consid-ers 271 di↵erent languages), this resource canexploit multilinguality to perform state-of-the-artknowledge-based Word Sense Disambiguation [35](in contrast to WordNet that encodes only Englishlexicalizations), thereby enabling new methods forthe automatic understanding of the multilingual(Semantic) Web.
Semantic graphs. The sar-graph construction wasfinalized by utilizing the lexical semantic links to
10Note that the sar-graphs also contain many dependencystructures that do not always signal instances of the targetrelation. Instead of filtering these out, we associate themwith confidence values determined by a semantic filter andby their positive and negative yield, see Section 4.
BabelNet for the creation of the relation-specific se-mantic graphs, as outlined in Section 4.1.2. The ob-tained information about the relation-relevancy ofdependency structures was subsequently integratedinto the sar-graphs.
5.4. Pattern observations from sar-graph validation
We conducted an error analysis of dependencystructures for all relation types using the Pattern-Judge tool. The relation sibling relationship servedas an initial test bed for establishing a set of sharedevaluation principles among the annotators, whothen screened the patterns in the remaining rela-tions for recurring errors. There were three anno-tators in total; they identified six main error classes,which are listed in Table 4.
Three of the classes describe errors based on de-fective output of the preprocessing pipeline, namelysentence boundary detection, named entity recog-nition and parse tree generation. We label these er-ror types PIPE-S, PIPE-NER and PIPE-PT. Theother three types refer to errors that cannot be at-tributed to accuracy deficits of linguistic prepro-cessing. Semantic understanding is required in or-der to detect them. The first class corresponds topatterns that do not express the relation of inter-est (NEX-P), whereas the other two describe de-pendency structures generated from sentences thateither do not express the target relation (NEX-S )or do so, but in a way that is too implicit (IMP-S ).Because relevance and explicitness are, to some de-gree, vague criteria, this second set of error classes ismore susceptible to debate than the first. We usedthe guidelines developed during the iterative dis-cussion of the sibling relationship relation to defineboundaries and to help evaluate borderline cases.
The category PIPE-S pertains to ungrammaticalsentence and dependency structures resulting fromsentence boundary detection errors. In example (1)in Table 4, the tokens “Personal life” (which aremost likely a headline / news category identifier)are not only interpreted as part of the sentence, butalso as relevant elements of the extracted pattern.
PIPE-NER is the error class that is used for pat-terns which contain arguments that are semanti-cally or grammatically incongruent with the onestagged in the sentence. In example (2), for the re-lation award honor, the title of the book has notbeen recognized as an entity. The inclusion of thelemmas “leave” and “us” as lexical nodes resultsin a pattern that is unlikely to be applicable toother text. The error class also includes instances
18
# Error class Description Example
1 PIPE-S Sentence seg-mentation error
Personal:::life
:::On July 5, 2003, Banks
::::::married sportswriter and producer
Max Handelman, who had been her boyfriend since she met him on herfirst day at college, September 6, 1992. (marriage)
2 PIPE-NER NER tagging er-ror
Rahna Reiko Rizzuto is the:::::author
:::of the
::::novel, Why She
:::Left
:::Us, which
:::won an American Book Award
::in 2000. (award honor)
3 PIPE-PT Dependencyparsing error
⇤Say:::won a Caldecott Medal for his illustrations in Grandfather’s
Journey. (award honor)
4 NEX-P Relation is notexpressed inpattern
Julian:::::joined Old Mutual in August 2000 as Group Finance Director,
:::::moving on to become
:::CEO
::of Skandia following its purchase by Old Mutual
in February 2006. (acquisition)
5 NEX-S Relation is notexpressed intext
The 69th Annual Peabody Awards:::::::ceremony will
::be
::::held on May 17 at the
Waldorf-Astoria in New York City and will be:::::hosted
:::by Diane Sawyer,
the award-winning anchor of ABCs World News. (award honor)
6 IMP-S Relation is tooimplicit
The looming expiration of Lipitors patent in 2012 is a big reasonPfizer
:::felt
:::::::::compelled to
:::buy a
::::::company
:::::like Wyeth. (acquisition)
Table 4: Common error classes of dependency structures for the relations marriage, acquisition and award honor. Underlinedtoken sequences denote relation arguments,
::::::concepts
::::with
:a:::::wavy
:::::::underline are additional pattern elements. Error classes are
described in more detail in Section 5.4.
⇤Say
dobj
!!win
dobj
&&Caldecott Medal
Figure 12: Erroneous dependency structure extractedfrom the sentence “Say won a Caldecott Medal for hisillustrations in Grandfather’s Journey.”
in which named entities are assigned to the wrongsemantic categories (e.g., a person mention appearsin the parse tree as an organization).
The category PIPE-PT is applied to dependencystructures extracted from defective parse trees. Theexample sentence (3) is erroneously parsed as shownin Figure 12, with the proper name Say interpretedas a finite verb. Other typical parsing errors weobserved included wrong PP attachment, conjunc-tions attached as dependents of one of the con-juncts, or the inclusion of superfluous punctuationtokens in the parse tree.
The category NEX-P is used for dependencystructures that do not include any relation-relevantcontent words occurring in the sentence. In exam-ple (4), the most explicit element expressing an ac-quisition is the lemma “purchase”. The patterndoes not include this word, but focuses on less rel-evant parts of the associated source sentence.
NEX-S applies to dependency structures that arebased on sentences which do not express the rela-tion of interest, i.e., sentences which violate the dis-
tant supervision assumption.11 In example (5), thetarget relation award honor is not expressed. In-stead, the host of the ceremony is erroneously iden-tified as the winner of the prize. This example alsohighlights an additional di�culty of the distant su-pervision assumption: The source sentence reportsan event that lies in the future. Similar errors mayoccur for sentences containing modal verbs or re-porting fictional events.
The category IMP-S marks dependency struc-tures based on sentences in which a relation is ex-pressed merely implicitly. Judging from the sourcesentence in example (6), we cannot be entirely surewhether or not an acquisition took place because“felt compelled to” might only express a mo-mentary mindset of the company’s leaders that wasnot followed by action. If it was, it is not clear if“Wyeth” or “a company like Wyeth” (i.e., a simi-lar company) was acquired.
We limit the expert-driven quality control to asubset of structures as described in Section 4.2.1.Table 5 presents the results of the error analysis.The second column lists for each relation the num-ber of dependency structures contained in this man-ual evaluation dataset. The left part of Table 5summarizes the distribution of correct and incor-rect dependency structures for all relations. Wefind that between 25.7% and 80.4% of the learned
11Although we applied this criterion when creating thedataset, at the time it was based on a very small sample.
Table 5: Distribution of evaluation categories and error classes for dependency patterns manually reviewed using the Pattern-Judge tool. The table lists the total number of evaluated patterns per relation, and the distribution across categories. It alsoshows the distribution of error classes for the Incorrect category.
dependency structures are erroneous, between 7.3%and 64.5% are labeled as correct. For example,more than 70% of the patterns of the relation orga-nization alternate name are labeled as incorrect.Correct, but too specific patterns make upbetween 1.6% and 36.5% of the total number ofpatterns.
The right-hand part of the table gives details onthe distribution of the error classes. The two pre-dominant error classes parsing errors (PIPE-PT)and pattern extraction errors (NEX-P). We observethat the distribution of the error classes varies be-tween the di↵erent relations: PIPE-NER is the er-ror type most frequently occurring for award honorand award nomination patterns. Sentences in thiscategory often mention the titles of works the prizewas awarded for. If those titles are not recognizedas entities by the NER tagger, the dependency pars-ing fails and parts of the title can erroneously endup in the extracted dependency structure. For theacquisition relation, the vast majority of errors canbe assigned to the category NEX-S. In these cases,a relation between two or more organizations is of-ten expressed in the source sentences, e.g., thatcompany x is a subsidiary of company y, but no
statement is made about the act of purchase. Forthe marriage relation, the most frequent error type(with the exception of theOther error class) is IMP-S, mainly due to sentences stating a divorce, which,according to our annotation guidelines, is not anexplicit mention of the marriage relation. A finalobservation that can be made from Table 5 is thatapproximately 44.0% of the incorrect patterns re-sult from preprocessing pipeline errors.
6. Related Work
In the previous sections, we have motivated theconstruction of sar-graphs and outlined a methodof building them from an alignment of web textwith known facts. Taking into account the im-plemented construction methodology, it may seemthat sar-graphs can be regarded as a side-productof pattern discovery for relation extraction. How-ever, sar-graphs are a further development of this,i.e., a novel linguistic knowledge resource on top ofthe results of pattern discovery.
In comparison to well-known knowledge basessuch as YAGO [2, 3], DBpedia [4], Freebase [1],Google’s Knowledge Graph [9] or the recent Google
20
Knowledge Vault [10], sar-graphs are not a databaseof facts or events, but rather a repository of lin-guistic representations expressing facts or events.As explained above, the acquisition of sar-graphelements is more related to pattern discovery ap-proaches developed in traditional schema-based IEsystems (such as NELL [5, 6], PROSPERA [8] orWeb-DARE [18]) than it is to open information ex-traction (Open IE; e.g., ReVerb [36] and the workby [37, 38, 39, 40, 41, 42]). The principal idea ofopen IE approaches is that automatically learnedpatterns are not fixed to a certain schema or ontol-ogy. Even though post-processing steps are avail-able for these systems which align the patterns intaxonomies or prepare them otherwise for variousdownstream tasks [43, 44, 45, 46, 47, 48, 49], sar-graphs are still somewhat closer to schema-drivenknowledge graphs, meaning that sar-graphs can bedirectly applied to free texts for enlarging a struc-tured repository of knowledge. In Sections 7.1 &7.2, we compare sar-graphs to other systems basedon lexico-syntactic patterns.For the current sar-graphs, we employ a web-
driven approach to collect and manage a wide va-riety of linguistic representations for each seman-tic relation. The formalism used for intermedi-ate storing of phrases is based on own prior work[23]; we leave a more sophisticated methodologywhich could return expressions at various gran-ularities [50] for the future. Our work is novelin comparison to traditional pattern-discovery ap-proaches, since we reorganize the collected struc-tures into a coherent, relation-specific linguistic re-source, instead of viewing them as sets of indepen-dent, statistically-enriched patterns. Sar-graphsmerge high-confidence linguistic structures, andcombine syntactic information with lexical semanticand probabilistic information. The merged struc-tures can be taken as input for further inductionand generalisation.In comparison to the various construction meth-
ods for knowledge bases discussed in [10], the gen-eration of sar-graphs is more in the trend of theKnowledge-Vault approach, since 1) the depen-dency patterns in sar-graphs are automatically ac-quired from free texts on the Web, therefore scal-able with the growth of the Web; 2) it is drivenby a fixed ontology/schema and 3) the patterns areassigned with probabilistic confidence values, thus,adaptable to performance requirements of variousapplications. Since sar-graphs have been acquiredautomatically, the construction method is poten-
tially scalable to any new schema.In the context of knowledge graphs, sar-graphs
are one of the first resources to link repositoriesof facts with linguistic knowledge (i.e., word mean-ings) in an automatic manner. Each sar-graph cor-responds to a linguistic representation of seman-tic relations from knowledge bases, at the sametime the arguments of facts and events are explic-itly modeled in sar-graphs. Since a sar-graph is ageneric resource, linguistic patterns automaticallylearned by other systems, e.g., NELL patterns, canalso be employed as input. Therefore, sar-graphscan be further developed as a platform for merg-ing and fusing all available extraction patterns fromvarious sources.
Many linguistic repositories, such as WordNet[32], FrameNet [51], and VerbNet [52] already ex-isted before the recent development of large knowl-edge bases. These linguistic resources have beenconstructed with the motivation of modeling thesemantics of the language at the word or syntac-tic level, without an explicit link to the real worldor applications. Most of them are relatively smallscale, due to their manual construction. WordNetcaptures lexical semantic relations between indi-vidual words, such as synonymy, homonymy, andantonymy. FrameNet focuses on fine-grained se-mantic relations of predicates and their arguments.VerbNet is a lexicon that maps verbs to prede-fined classes which define the syntactic and seman-tic preferences of the verb. In contrast to these re-sources, sar-graphs are data-driven, constructed au-tomatically, and incorporate statistical informationabout relations and their arguments. Therefore,sar-graphs complement these manually constructedlinguistic resources. Furthermore, since word-senseinformation is integrated into sar-graphs, a link-ing to other linguistic resources via word senses isstraightforward. Thus, sar-garphs can contributeto the linked open data movement (LOD). On theother hand, FrameNet and VerbNet can be em-ployed as useful resources for validating the au-tomatically learned dependency patterns in sar-graphs.
Parallel to the development of large factdatabases, there is also increasing research and de-velopment in creating similarly sized linguistic re-sources, e.g., BabelNet, ConceptNet[13] and UBY[14] automatically. Many of them are build on topof existing resources like WordNet, Wiktionary andWikipedia. As mentioned before, BabelNet is a newdevelopment of acquiring large-scale lexical seman-
21
tics network automatically. It merges Wikipediaconcepts including entities with word senses fromWordNet. BabelNet can thus be regarded as aknowledge base combining word sense and entity in-formation. In sar-graphs, we employ BabelNet forour word sense disambiguation task since BabelNetis large-scale and its multilinguality is importantfor extending sar-graphs to other languages. Par-allel to BabelNet, [11] create a multilingual wordnet, called Universal WordNet (UWN), by integrat-ing di↵erent word nets, bilingual dictionaries, andinformation from parallel corpora. This resource islargely an extension of already available word netsand does not provide any explicit linking to a factknowledge base.ConceptNet is a semantic network encoding
common-sense knowledge and merging informationfrom various sources such as WordNet, Wiktionary,Wikipedia and ReVerb. The nodes are words andphrases expressing concepts and relations amongthem. Relations are represented by phrases suchas “UsedFor” or “HasContext”. In comparison tosar-graphs, there is no explicit linguistic knowledgelike syntactic or word-sense information assignedto the content elements. The semantic relationsamong concepts are not fixed to an ontology orschema. However, ConceptNet is a very useful re-source which can potentially be utilized to enrichand validate the sar-graphs.UBY is an approach to combine and align vari-
ous linguistic resources by employing the so-calledISO-standard Lexical Markup Framework (LMF).They provide a uniform and standardized represen-tation of the individual resources to enable theirinteroperability. It includes WordNet, Wiktionary,Wikipedia, FrameNet, VerbNet, also the multilin-gual OmegaWiki. Since UBY is a platform for in-tegration and is therefore open to various applica-tions, sar-graphs can be integrated into UBY as afurther resource and can be linked to other linguis-tic resources via UBY.Finally, ontology formalizations such as Lemon
[53] and manual and semi-automatic methods to en-rich knowledge bases such as DBpedia and NELL interms of relation lexicalizations have been presented[54, 55, 56, 57]. The main goal of these methods isto extend the capabilities of the knowledge basesfor extracting novel relation instances by using alarge set of patterns [58, 59, 60] and to generateNL descriptions of their content [61]. In this paper,we go one step further by not only extending thereference set of extraction patterns associated with
the considered relations in terms of lexicalizationsand ontological classes but also linking the auto-matically discovered patterns to word meanings byexploiting word sense disambiguation.
7. Applications and experiments
We believe that sar-graphs, in addition to theirrole as an anchor in the linked data world and as arepository of relation phrases, are also a very usefulresource for a variety of natural-language process-ing tasks and real-world applications. In particular,sar-graphs are well-suited for (a) the generation ofphrases and sentences from database facts and (b)the detection of fact mentions in text.
The first aspect makes sar-graphs a good can-didate for the employment in, e.g., business intel-ligence tools aiming to generate natural-languagereports for recurring review periods. Because ofthe range of paraphrases available in sar-graphs,generation could produce stylistic variation as ex-tensively used in reports written by human au-thors. An application that combines both aspectsis summarization, where sar-graphs permit to iden-tify fact-heavy parts of a text (i.e., constructionsthat express all or most arguments of a relation inone sentence) and also allow these parts of a textto be rephrased in a shorter manner.
The most obvious application of sar-graphs, how-ever, is information extraction. As the sar-graphswe have already built contain all the dependencystructures we had learned and tested for several re-lations, we know that the information in a sar-graphcan be successfully applied to regular relation ex-traction, i.e., the detection of fact mentions in sen-tences.
7.1. Potentials for relation extraction
In order to show that sar-graphs can be success-fully applied for information extraction, we con-ducted a series of experiments [21] for six out of the25 target relations for which sar-graphs are avail-able. We used the Los Angeles Times/WashingtonPost (henceforth LTW) part of the English Gi-gaword v5 corpus [62]. LTW is comprised of400k newswire documents from the time 1994–2009.With the help of Stanford NER, we found 4.8M sen-tences in LTW that mention at least two entitieswhich correspond to argument types defined in thesix relations.
We match the individual dependency construc-tions in the sar-graphs with the dependency parses
Table 6: Statistics about RE performance of sar-graphs forsix relations. Freq.-overlap/Sem.-graph are described in Sec-tions 4.1.1/4.1.2, respectively. Each cell is displaying theprecision/recall/f1-measure results in % from the respectivef1-best parameter setting.
of the sentences in LTW and evaluate the correct-ness of the relation mentions detected this way. Thedi↵erent means of sar-graph quality control (seeSection 4) are compared wrt. their impact on REperformance. To estimate the precision of RE, wemanually check a random sample of 1K extractedmentions per relation, and we proceed similarly toget an estimate of the RE coverage.12
Table 6 presents the results of the experiment.While both types of dependency-structure filteringallow to obtain reasonably good results for the REtask, it is interesting to see that their combina-tion gets an enormous precision boost at almostno cost in terms of recall. Note that the generallylow recall values (around 30%) are quite commonamong state-of-the-art RE systems. See, for exam-ple, [18] for an analysis of the relation mentions notextracted.In a previous study [21], we compared the ex-
traction performance of a sar-graph predecessor toNELL’s [6] lexico-syntactic patterns. NELL is asystem designed to learn factual knowledge froman immense, web-like corpus over a long period.We found that when applied to the English Giga-word corpus mentioned above, the amount of re-lation mentions covered by the patterns of NELLwas substantially lower than what the sar-graph-like patterns covered. For a similar selection of re-lations as in Table 6, the NELL patterns matchedapproximately 10% of the number of facts extractedby our patterns. Interestingly, the overlap of factswas rather low, i.e., both systems extracted factsthe other system was not capable of finding. By
12We use our dataset from [21], where we manually verifieda large number of Freebase-fact mentions found on a sub-part of LTW, i.e., only sentences actually referring to thecorresponding target relation are part of this dataset.
stating these results here, we do not aim to providea comprehensive comparison of the capabilities ofthe two systems, but rather only want to provideevidence that sar-graphs are indeed a useful mem-ber of the diverse landscape of RE systems. Wecontinue our explorative comparison of sar-graphsto other systems in the following section.
Sar-graphs for CALL. Another application forwhich we applied our sar-graphs is related tothe area of computer-assisted language learning(CALL). We implemented a prototype [63] forthe task of semi-automatic generation of reading-comprehension exercises for second-language learn-ers. A language teacher, who has to prepare suchexercises for an upcoming class, is presented withnews texts retrieved from the Web, along with can-didate multiple-choice questions and answers, re-lating to certain facts mentioned in the text. Theteacher then has to pick the useful question-answerpairs.
Sar-graphs are utilized here both for the fact-finding phase (i.e., for the detection of true-answercandidates), and for the generation of paraphrasesfor true facts as well as the question asking aboutthe facts. During the evaluation of this setting,we found that for average-length news texts, sev-eral correct and potentially useful question-answerpairs are generated for each input text, which led usto conclude that sar-graphs would be of real-worlduse in such an application setting.
7.2. Similarities with pattern stores
In this section, we present an explorative compar-ison of the linguistic expressions in the sar-graphswith a typical representative of a RE-pattern storefrom the literature. We selected the PATTY system[45]13 because it shows very good performance onthe information extraction task and it implementsa generic approach for the creation of a taxonomyof textual patterns, a goal similar to what we aimat with our sar-graphs.
PATTY implements an open-IE approach to thecollection of phrases, which is followed by an align-ment phase where a subsumption hierarchy of pat-terns is created. The PATTY authors provide a dis-ambiguation of their patterns to a knowledge base,from which we select four relations with a consider-able semantic overlap with the sar-graph relations
Table 7: Comparison of patterns from PATTY and sar-graphs, for a set of relations present in both resources.
and for which a reasonable number of patterns isavailable for both systems.In order to get an estimate of the quality of
PATTY’s patterns, we took a sample of 200 pat-terns from each relation and, based on the enti-ties with which a pattern co-occurred in Wikipedia,generated instantiations for all associated entity-type signatures. Three annotators were then askedto judge whether the majority of instantiations perpattern does express the respective target relation.For example, a person joining another person doesnot indicate a mention of the employment tenurerelation, but a sports person joining a sports team
does. For the sar-graph patterns, we followed asimilar strategy, which resulted in 200 instantiatedand string-serialized patterns with entities shownto three human raters. If the annotators could notmake sense of a pattern (e.g., because it was overlyspecific or contained superfluous elements not re-quired for matching a mention of the respective tar-get relation), their guideline was to rate this patternas wrong.Table 7 compares PATTY with sar-graph pat-
terns retained after applying the semantic-graphfilter14 (Section 4.1.2) and additionally presentsstatistics for the sar-graph patterns in the manual-evaluation dataset from the error analysis in Sec-tion 5.4. Along with the pattern numbers, we puta lexical-diversity score which states the averageamount of distinct, non-function words a patternhas wrt. other patterns in the same set. This scoreallows to better interpret the absolute pattern num-bers, which are subject to the di↵erent pattern-representation formalisms.For the relations of this analysis, sar-graphs pro-
vide more linguistic expressions at a higher rate oflexical diversity, i.e., the sar-graph patterns are asleast as well suited for RE as the PATTY system
14We set k = 3 as this typically results in a good preci-sion/recall trade-o↵.
wrt. coverage of lexically variations of relation men-tions. Furthermore, more than twenty percent ofthe sar-graph patterns in this analysis link three ormore arguments to one another, in contrast to thePATTY patterns which are all binary. Note thatwhile PATTY and sar-graphs were created fromdi↵erent corpora, the size of these corpora is simi-lar, i.e., 2.2 million Web documents in the case ofsar-graphs, and 1.8 million New York Times arti-cles/3.8 million Wikipedia articles for PATTY.
Both systems produce patterns at similar levelsof precision, where for some relations one systemtrumps the other. Looking for an explanation of thelowmarriage precision of PATTY, we found that onaverage the patterns contained less tokens than theones in the sar-graphs, which makes them more de-pendent on the disambiguation power of the entity-type signature. marriage entity types are generic(two persons), in particular given the source corpusWikipedia; consequently the precision of PATTYfor this relation is lower than for the others. Forperson parent, the situation is similar.
7.3. Knowledge-base population
One of the main goals of relation extraction is theautomatic identification of novel relation instancesfrom raw text to populate a knowledge base withnew facts. It is well known that existing knowledgebases, such as Freebase [1], while quite sizable, arestill far from complete. For example, Dong et. alobserve that 71% of the people in Freebase haveno known place of birth, and 75% have no knownnationality [10]. Large-scale text corpora, such asClueWeb [64], are assumed to contain thousands ofpotentially interesting relations and facts, but it isunclear whether these are novel or already coveredby the knowledge base. To answer this question, weconducted a small experiment to get a rough esti-mate of how many new facts we can identify witha sar-graph based relation extraction approach.
We create a test dataset by randomly sampling12K documents from the FACC1 corpus [65], a pre-processed version of the ClueWeb dataset where en-tity mentions have been linked to their correspond-ing Freebase identifiers. From these documents,we select sentences containing at least two Free-base entity mentions. We parse sentences with theMaltParser [66], and extract relation mentions us-ing sar-graph dependency patterns. For each ex-tracted relation mention, we determine its true la-bel(s) by looking up valid relations of the mentions’entities in Freebase. We estimate the percentage of
24
Known All Est. Correct
Predicted Predicted Novel Facts
1171 5552 ⇡ 1314
Table 8: Estimated number of novel “facts” found by sar-graph-based relation extraction on a ClueWeb dataset sam-ple. The estimate is derived from the number of known andpredicted facts, and combined with the approximate preci-sion of 30% of the pattern-based approach.
novel facts by computing the di↵erence of knownand predicted relation mentions, multiplied by anestimated precision of 30% of the pattern-based ap-proach. The precision value was determined on afully-annotated reference dataset in prior work [22].Table 8 lists the number of predicted, known andestimated correct novel facts. The rough estimatesuggests that approximately 50% of the identifiedrelation mentions are not yet included in the knowl-edge base.
8. Public availability, release
We release our sar-graph data to the public, hop-ing to support research in the areas of relationextraction, question answering, paraphrase genera-tion, and others. The data is available for downloadat http://sargraph.dfki.de.
Properties of the release. The released dataset con-tains English sar-graphs for 25 target relations (Ta-ble 2) about corporations, award topics, as wellas biographic information, many of them linkingmore than just two arguments. For now, we limitthe released data to manually verified dependencystructures (Section 5.4), i.e., only structures judgedas correct are published. Sar-graphs were createdfrom the dependency structures using the strat-egy depicted in the pseudocode in Figures 6 & 7.Properties of the currently released sar-graphs areshown in Table 9.15 The sar-graph creation (seeTable 3) involved the utilization of 230k instancesof the semantic relations, which resulted in over-all two million crawled web documents with sup-posed mentions of the seeds. From the resultingone million unique sentences we extracted 600k de-pendency structures, leading to sar-graphs for the
15The sar-graph data available on our website constitutesa superset of the structures judged as correct in Section 5.4.The additional structures were manually verified in a furtherannotation e↵ort with the PatternJudge tool (Section 4.2.3).
#nodes # edges
award honor 242 707award nomination 134 363
country of nationality 182 546education 154 407marriage 199 538person alternate name 199 512person birth 74 175person death 114 292person parent 110 396person religion 110 230place lived 114 256sibling relationship 67 166
target relations that add up to 300k vertices and1.5 million edges. The curated set of sar-graphs wepublish has 8k vertices connected by 20k edges.
Along with the sar-graphs, we release the indi-vidual dependency constructions used to constructthe published set of sar-graphs, as well as the cor-responding word-sense information and the assess-ments from the automatic quality control.
As described in Section 6, we see our sar-graphsas a natural extension to the resources already es-tablished as part of the Semantic Web, comple-menting the existing information and linking to theavailable resources. We link the data to lexical-semantic resources via the synset information at-tached to the vertices and also to factual knowl-edge bases via the covered semantic relations. Byadditionally providing the sar-graphs in the Lemonformat16, we hope to facilitate further research onthe interesting area of intersecting factual knowl-edge with linguistic information.
Java-based API. Accompanying the data, we pro-vide a Java-based API which simplifies the loading,processing, and storing of sar-graphs, and also al-lows to visualize the individual dependency struc-tures we exploited for the generation of sar-graphs.
16http://www.lemon-model.net/
25
One particularly helpful feature of the API is theconcept of materialized views, already broached inSection 3. The basic idea is that with di↵erenttasks and goals, varying aspects of a sar-graph be-come relevant. For example, an application havinga lexical-semantic point-of-view on data might beinterested in possible dependency connections be-tween sets of synonymous words or word classes(hence a very broad view on the sar-graphs isneeded), while another application might want topartition linguistic information based on the spe-cific facts the knowledge was derived from, circum-venting even the abstraction of entities to entitytypes. Hence, the sar-graph data should be pre-sented in the respective most informative way toan application. The API from our recent releasesprovides this possibility.17
Future plans for the resource. For the coming re-leases, we plan (1) to publish the non-curated partof the sar-graph data, which, for example, provedto be useful for tasks like relation extraction (Sec-tion 7), (2) to provide more detailed informationabout the source of linguistic expressions (i.e., ex-pand the public data with source sentences and seedfacts, (3) to extend the sar-graph approach to moresemantic relations and domains.
9. Conclusion
In this article, we present a new linguistic re-source called sar-graph, which aggregates knowl-edge about the means a language provides for ex-pressing a given semantic target relation. We de-scribe a general approach for automatically accu-mulating such linguistic knowledge, and for mergingit into a single connected graph. Furthermore, wediscuss di↵erent ways of assessing the relevancy ofexpressions and phrases with respect to the targetrelation, and outline several graph merging strate-gies. We show the validity of our approach by im-plementing it on top of a large English web corpus.In our experiments, we created and evaluated sar-graphs for 25 relations from the domains Award,Business and People. A curated subset of thesegraphs is publicly available at sargraph.dfki.de.
17Although we are not providing all aspects of the poten-tially interesting information at the moment. For example,we are not yet publishing source sentences along with thederived dependency structures, due to licensing issues.
We believe linguistic resources like sar-graphsshould be created in a bottom-up fashion, therebybeing empirically grounded on the actual ways peo-ple communicate about semantic relations in dif-ferent languages. Even though we admit that afully automatic approach is hardly feasible due toshortcomings of the unsupervised quality assess-ments, we think that a fully curated approach,i.e., language-independent engineering of ontolo-gies, would constitute a throwback to a researchparadigm in which knowledge engineering precedesany attempt of language understanding.
From experience we have learned that there couldbe numerous di↵erent ontologies just for the the-matic area marriage. Lawyers, event managers,relationship counselors, vital statisticians may allcome up with completely di↵erent ways to selectand structure the respectively relevant knowledgepieces. How could we decide on the best ontologyfor, e.g., the task of relation extraction? Would anyof such intellectually created ontologies contain arelation for exchanging the vows and one for tyingthe knot? How would the vows and the knot berepresented? The great advantage of an empiricalbottom-up approach is that it is guided by the ac-tual ways people use to refer to a relation (or event,process, etc.), and that one is not pressured to makesuch a-priori ontology-level decisions.
Another important choice we make is the asso-ciation of graphs to specific languages. A Greekreport on a wedding may refer to wedding crownsfor bride and groom, while in an English sar-graphfor the marriage relation, such crowns would notshow up. In a Greek wedding the betrothal can bea part of the entire ceremony, in other cultures itmust have taken place a certain period before thewedding. In some cultures, exchanging the ringsmeans getting married in others there is no suchconcept.
We are convinced that we need the interactionof two strategies to build up a growing stock ofstructured knowledge in the spirit of a semanticweb. One strategy starts from structuring grow-ing portions of textual knowledge sources (such asWikipedia) and extends this by already structureddata (such as linked open data). Another strat-egy uses and extends the resulting repositories ofstructured knowledge by extracting from all sorts oftexts much more facts, especially contingent ones.The novel type of resource we propose will on theone hand facilitate the latter process and on theother hand maintain the link of the accumulated
26
domain-sorted linguistic knowledge with structuredresources from the Semantic Web.
Acknowledgements
This research was supported by the Ger-man Federal Ministry of Education and Re-search (BMBF) through the projects Deepen-dance (contract 01IW11003), ALL SIDES (contract01IW14002) and BBDC (contract 01IS14013E), aswell as by the ERC Starting Grant MultiJEDI No.259234, and a Google Focused Research Awardgranted in July 2013.
References
[1] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, J. Tay-lor, Freebase: A collaboratively created graph databasefor structuring human knowledge, in: Proc. of SIG-MOD, ACM, 2008. doi:10.1145/1376616.1376746.
[2] F. M. Suchanek, G. Kasneci, G. Weikum, Yago: A coreof semantic knowledge, in: Proc. of WWW, ACM, 2007.doi:10.1145/1242572.1242667.
[3] F. M. Suchanek, G. Kasneci, G. Weikum, YAGO: Alarge ontology from Wikipedia and WordNet, Web Se-mantics: Science, Services and Agents on the WorldWide Web 6 (3) (2008) 203–217. doi:10.1016/j.websem.2008.06.001.
[4] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kon-tokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. vanKleef, S. Auer, C. Bizer, DBpedia - a large-scale, mul-tilingual knowledge base extracted from Wikipedia, Se-mantic Web Journal 6 (2) (2015) 167–195. doi:10.3233/SW-140134.
[5] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R.Hruschka, T. M. Mitchell, Toward an architecture fornever-ending language learning, in: Proc. of AAAI,2010.
[6] T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar,J. Betteridge, A. Carlson, B. Dalvi, M. Gardner,B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis,T. Mohamed, N. Nakashole, E. Platanios, A. Ritter,M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta,X. Chen, A. Saparov, M. Greaves, J. Welling, Never-ending learning, in: Proc. of AAAI, 2015.
[7] D. Vrandecic, M. Krotzsch, Wikidata: A free collabo-rative knowledgebase, Communications of the ACM 57(2014) 78–85.
[8] N. Nakashole, M. Theobald, G. Weikum, Scalableknowledge harvesting with high precision and high re-call, in: Proc. of WSDM, ACM, 2011. doi:10.1145/1935826.1935869.
[9] A. Singhal, Introducing the Knowledge Graph: things,not strings, Google O�cial Blog, http://goo.gl/KCwaV6 (May 2012).
[10] X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao,K. Murphy, T. Strohmann, S. Sun, W. Zhang, Knowl-edge vault: A web-scale approach to probabilisticknowledge fusion, in: Proc. of SIGKDD, ACM, 2014.doi:10.1145/2623330.2623623.
[11] G. de Melo, G. Weikum, Towards a universal wordnetby learning from combined evidence, in: Proc. of CIKM,ACM, 2009. doi:10.1145/1645953.1646020.
[12] R. Navigli, S. P. Ponzetto, BabelNet: The automaticconstruction, evaluation and application of a wide-coverage multilingual semantic network, Artificial In-telligence 193 (2012) 217–250. doi:10.1016/j.artint.2012.07.001.
[13] R. Speer, C. Havasi, ConceptNet 5: A large semanticnetwork for relational knowledge, in: The People’s WebMeets NLP, Springer, 2013, pp. 161–176.
[14] I. Gurevych, J. Eckle-Kohler, S. Hartmann, M. Ma-tuschek, C. M. Meyer, C. Wirth, Uby: A large-scale uni-fied lexical-semantic resource based on LMF, in: Proc.of EACL, ACL, 2012.
[16] H. Uszkoreit, F. Xu, From Strings to Things SAR-Graphs: A New Type of Resource for ConnectingKnowledge and Language, in: Proc. of NLP & DBpediaWorkshop co-located with ISWC, CEUR-WS.org, 2013.
[17] M. Mintz, S. Bills, R. Snow, D. Jurafsky, Distant su-pervision for relation extraction without labeled data,in: Proc. of ACL-IJCNLP, ACL, 2009.
[18] S. Krause, H. Li, H. Uszkoreit, F. Xu, Large-scale learn-ing of relation-extraction rules with distant supervisionfrom the web, in: Proc. of ISWC, Springer, 2012.
[19] L. Jean-Louis, R. Besanon, O. Ferret, A. Durand, Usingdistant supervision for extracting relations on a largescale, in: Knowledge Discovery, Knowledge Engineer-ing and Knowledge Management, no. 348 in Communi-cations in Computer and Information Science, Springer,2013, pp. 141–155. doi:10.1007/978-3-642-37186-8_9.
[20] B. Min, R. Grishman, L. Wan, C. Wang, D. Gondek,Distant supervision for relation extraction with an in-complete knowledge base, in: Proc. of NAACL-HLT,ACL, 2013.
[21] A. Moro, H. Li, S. Krause, F. Xu, R. Navigli, H. Uszko-reit, Semantic rule filtering for web-scale relation ex-traction, in: Proc. of ISWC, Springer, 2013.
[22] H. Li, S. Krause, F. Xu, A. Moro, H. Uszkoreit, R. Nav-igli, Improvement of n-ary relation extraction by addinglexical semantics to distant-supervision rule learning,in: Proc. of ICAART, SciTePress, 2015.
[23] F. Xu, H. Uszkoreit, H. Li, A seed-driven bottom-upmachine learning framework for extracting relations ofvarious complexity, in: Proc. of ACL, ACL, 2007.
[24] F. Xu, Bootstrapping relation extraction from semanticseeds, Ph.D. thesis, Saarland University (2007).
[25] M.-C. de Marne↵e, C. D. Manning, Stanford Depen-dencies Manual (2008).URL http://nlp.stanford.edu/software/dependencies_manual.pdf
[26] M. Stevenson, Fact distribution in information extrac-tion, Language Resources and Evaluation 40 (2) (2006)183–201. doi:10.1007/s10579-006-9014-4.
[27] K. Swampillai, M. Stevenson, Inter-sentential relationsin information extraction corpora, in: Proc. of LREC,ELRA, 2010.
[28] R. Grishman, Information Extraction: Capabilities andChallenges, Tech. rep., NYU Dept. CS (2012).
[29] R. Navigli, Word sense disambiguation: A survey, ACMComput. Surv. 41 (2) (2009) 10:1–10:69. doi:10.1145/1459352.1459355.
27
[30] G. R. Doddington, A. Mitchell, M. A. Przybocki, L. A.Ramshaw, S. Strassel, R. M. Weischedel, The Auto-matic Content Extraction (ACE) program - tasks, data,and evaluation, in: Proc. of LREC, 2004.
[31] J. R. Finkel, T. Grenager, C. Manning, Incorporatingnon-local information into information extraction sys-tems by gibbs sampling, in: Proc. of ACL, ACL, 2005.doi:10.3115/1219840.1219885.
[32] C. Fellbaum (Ed.), WordNet: an electronic lexicaldatabase, Language, Speech, and Communication, MITPress, 1998.
[33] C. Chiarcos, S. Nordho↵, S. Hellmann (Eds.), LinkedData in Linguistics. Representing and Connecting Lan-guage Data and Language Metadata, Springer, 2012.doi:10.1007/978-3-642-28249-2.
[34] R. Navigli, BabelNet goes to the (multilingual) seman-tic web, in: Proc. of Workshop on the Multilingual Se-mantic Web, 2012.
[35] A. Moro, A. Raganato, R. Navigli, Entity linking meetsword sense disambiguation: A unified approach, Trans-actions of the Association for Computational Linguis-tics 2 (2014) 231–244.
[36] A. Fader, S. Soderland, O. Etzioni, Identifying relationsfor open information extraction, in: Proc. of EMNLP,ACL, 2011.
[37] O. Etzioni, A. Fader, J. Christensen, S. Soderland,Mausam, Open information extraction: The secondgeneration, in: Proc. of IJCAI, IJCAI/AAAI, 2011.
[38] Mausam, M. Schmitz, S. Soderland, R. Bart, O. Et-zioni, Open language learning for information extrac-tion, in: Proc. of EMNLP-CoNLL, ACL, 2012.
[39] E. Alfonseca, D. Pighin, G. Garrido, HEADY: Newsheadline abstraction through event pattern clustering,in: Proc. of ACL, ACL, 2013.
[40] Y. Xu, M.-Y. Kim, K. Quinn, R. Goebel, D. Barbosa,Open information extraction with tree kernels, in: Proc.of NAACL-HLT, ACL, 2013.
[41] L. Del Corro, R. Gemulla, ClausIE: Clause-based openinformation extraction, in: Proc. of WWW, Interna-tional World Wide Web Conferences Steering Commit-tee, 2013.
[42] D. Pighin, M. Cornolti, E. Alfonseca, K. Filippova,Modelling events through memory-based, open-ie pat-terns for abstractive summarization, in: Proc. of ACL,ACL, 2014.
[43] A. Yates, O. Etzioni, Unsupervised resolution of objectsand relations on the web, in: Proc. of HLT-NAACL,ACL, 2007.
[44] A. Moro, R. Navigli, WiSeNet: Building a Wikipedia-based semantic network with ontologized relations, in:Proc. of CIKM, ACM, 2012. doi:10.1145/2396761.2398495.
[45] N. Nakashole, G. Weikum, F. Suchanek, PATTY: Ataxonomy of relational patterns with semantic types,in: Proc. of EMNLP-CoNLL, ACL, 2012.
[46] C. Zhang, D. S. Weld, Harvesting parallel news streamsto generate paraphrases of event relations, in: Proc. ofEMNLP, ACL, 2013.
[47] O. Levy, I. Dagan, J. Goldberger, Focused entailmentgraphs for open IE propositions, in: Proc. of CoNLL,ACL, 2014.
[48] A. Grycner, G. Weikum, HARPY: Hypernyms andalignment of relational paraphrases, in: Proc. of COL-ING, DCU & ACL, 2014.
[49] S. Krause, E. Alfonseca, K. Filippova, D. Pighin, Idest:
Learning a distributed representation for event pat-terns, in: Proc. of NAACL-HLT, 2015.
[50] K. Beedkar, R. Gemulla, LASH: large-scale sequencemining with hierarchies, in: Proc. of SIGMOD, ACM,2015. doi:10.1145/2723372.2723724.
[51] C. F. Baker, C. J. Fillmore, J. B. Lowe, The BerkeleyFrameNet project, in: Proc. of ACL-COLING, ACL,1998.
[52] K. K. Schuler, Verbnet: A broad-coverage, comprehen-sive verb lexicon, Ph.D. thesis, University of Pennsyl-vania, Philadelphia, PA, USA (2005).
[53] J. McCrae, D. Spohr, P. Cimiano, Linking lexical re-sources and ontologies on the semantic web with Lemon,in: Proc. of ESWC, Springer, 2011. doi:10.1007/978-3-642-21034-1_17.
[54] S. Walter, C. Unger, P. Cimiano, ATOLL - a frameworkfor the automatic induction of ontology lexica, Data &Knowledge Engineering 94, Part B (2014) 148 – 162,S.I. following NLDB 2013. doi:http://dx.doi.org/10.1016/j.datak.2014.09.003.
[55] S. Walter, C. Unger, P. Cimiano, M-ATOLL: A frame-work for the lexicalization of ontologies in multiple lan-guages, in: Proc. of ISWC, Vol. 8796, Springer, 2014.doi:10.1007/978-3-319-11964-9_30.
[56] C. Unger, J. McCrae, S. Walter, S. Winter, P. Cimi-ano, A lemon lexicon for DBpedia, in: Proc. of NLP &DBpedia Workshop co-located with ISWC, Vol. 1064,CEUR-WS.org, 2013.
[57] N. Lao, A. Subramanya, F. Pereira, W. W. Cohen,Reading the web with learned syntactic-semantic infer-ence rules, in: Proc. of EMNLP-CoNLL, ACL, 2012.
[58] P. Cimiano, C. Unger, J. McCrae, Ontology-Based In-terpretation of Natural Language, Synthesis Lectureson Human Language Technologies, Morgan & ClaypoolPublishers, 2014. doi:10.2200/S00561ED1V01Y201401HLT024.
[59] M. Gardner, P. P. Talukdar, J. Krishnamurthy, T. M.Mitchell, Incorporating vector space similarity in ran-dom walk inference over knowledge bases, in: Proc. ofEMNLP, ACL, 2014.
[60] M. Gardner, P. P. Talukdar, B. Kisiel, T. M. Mitchell,Improving learning and inference in a large knowledge-base using latent syntactic cues, in: Proc. of EMNLP,ACL, 2013.
[61] P. Cimiano, J. Luker, D. Nagel, C. Unger, Exploitingontology lexica for generating natural language textsfrom rdf data, in: Proc. of European Workshop on Nat-ural Language Generation, ACL, 2013.
[62] R. Parker, English Gigaword fifth edition, linguisticData Consortium, Philadelphia (2011).
[63] R. Ai, S. Krause, W. Kasper, F. Xu, H. Uszkoreit,Semi-automatic generation of multiple-choice tests frommentions of semantic relations, in: Proc. of Workshopon Natural Language Processing Techniques for Educa-tional Applications, ACL, 2015.
[64] J. Callan, M. Hoy, C. Yoo, L. Zhao, The ClueWeb09dataset.URL http://lemurproject.org/clueweb09
[65] E. Gabrilovich, M. Ringgaard, A. Subramanya, FACC1:Freebase annotation of ClueWeb corpora, version 1 (re-lease date 2013-06-26, format version 1, correction level0).
[66] J. Nivre, J. Hall, J. Nilsson, Memory-Based Depen-dency Parsing, in: Proc. of CoNLL, ACL, 2004.
