Rela%onal Representa%ons
Daniel Lowd University of Oregon
April 20, 2015
Caveats
• The purpose of this talk is to inspire meaningful discussion.
• I may be completely wrong.
My background: Markov logic networks, probabilis%c graphical models
Q: Why rela%onal representa%ons?
A: To model rela%onal data.
Rela%onal Data • A rela%on is a set of n-‐tuples:
Friends: {(Anna,Bob), (Bob,Anna), (Bob,Chris)} Smokes: {(Bob),(Chris)} Grade: {(Anna, CS612, Fall2012, “A+”),…}
• Rela%ons can be visualized as tables:
• Typically make closed world assump%on: all tuples not listed are false.
Anna Bob
Bob Anna
Bob Chris Friend
s Bob
Chris Smokes
Rela%onal Knowledge • First-‐order logic • Descrip%on logic • Logic programs General form: A set of rules of the form “For every tuple of objects (x1,x2,…,xk), certain rela%onships hold.”
e.g., For every pair of objects (x,y), if Friends(x,y) is true then Friends(y,x) is true.
Sta%s%cal Rela%onal Knowledge • First-‐order logic • Descrip%on logic • Logic programs General form: A set of rules of the form “For every tuple of objects (x1,x2,…,xk), certain rela%onships probably hold.” (Parametrized factors or “parfactors”.)
• Bayesian networks • Markov networks • Dependency networks
e.g., For every pair of objects (x,y), if Friends(x,y) is true then Friends(y,x) is more likely.
Applica%ons and Datasets
What are the “killer apps” of rela%onal learning?
They must be rela%onal.
Graph or Network Data • Many kinds of networks: – Social networks – Interac%on networks – Cita%on networks – Road networks – Cellular pathways – Computer networks – Webgraph
Graph Mining
Graph Mining • Well-‐established field within data mining • Representa%on: nodes are objects, edges are rela%ons • Many problems and methods – Frequent subgraph mining – Genera%ve models to explain degree distribu%on and graph evolu%on over %me
– Community discovery – Collec%ve classifica%on – Link predic%on – Clustering
• What’s the difference between graph mining and rela%onal learning?
Social Network Analysis
Specialized vs. General Representa%ons
In many domains, the best results come from more restricted, “specialized” representa%ons and algorithms. • Specialized representa%ons and algorithms – May represent key domain proper%es beeer – Typically much more efficient – E.g., stochas%c block model, label propaga%on, HITS
• General representa%ons – Can be applied to new and unusual domains – Easier to define complex models – Easier to modify and extend – E.g., MLNs, PRMs, HL-‐MRFs, ProbLog, RBNs, PRISM, etc.
Specializing and Unifying Representa%ons
There have been many representa%ons proposed over the years, each with their own advantages and disadvantages. • How many do we need? • Which comes first, representa%onal power or algorithmic convenience?
• What are the right unifying frameworks? • When should we resort to domain-‐specific representa%ons?
• Which domain-‐specific ideas actually generalize to other domains?
Applica%ons and Datasets
What are the “killer apps” of general rela%onal learning?
They must be rela%onal.
They should probably be complex.
BioNLP Shared Task Workshop
In 2009, Riedel et al. win with a Markov logic network!
• They claim Markov logic contributed to their success: “Furthermore, the declara%ve nature of Markov Logic helped us to achieve these results with a moderate amount of engineering. In par%cular, we were able to tackle task 2 by copying the local formulae for event predic%on and adding three global formulae.”
• However, conver%ng this problem to an MLN was non-‐trivial: "In future work we will therefore inves%gate means to extend Markov Logic (interpreter) in order to directly model event structure.”
Task: Extract biomedical informa%on from text.
event (i) ) 9t.eventType (i, t)
eventType (i, t) ) event (i)
eventType (i, t) ^ t 6= o ) ¬eventType (i, o)
¬site (i) _ ¬event (i)
role (i, j, r) ) event (i) j r i i
role (i, j, r1) ^ r1 6= r2 ) ¬role (i, j, r2)
eventType (e, t) ^ role (e, a, r) ^ event (a) ) regType (t)
role (i, j, r) ^ taskOne (r) ) event (j) _ protein (j)
role (i, j, r) ^ taskTwo (r) ) site (j)
site (j) ) 9i, r.role (i, j, r) ^ taskTwo (r)
event (i) ) 9j.role (i, j, )
eventType (i, t) ^ ¬allowed (t, r) ) ¬role (i, j, r)
role (i, j, r1) ^ k 6= i ) ¬role (k, j, r2)
j < k ^ i < j ^ role (i, j, r1) ) ¬role (i, k, r2)
47
BioNLP Shared Task Workshop
For 2011, Riedel and McCallum produce a more accurate model as a factor graph: Is this a victory or a loss for rela%onal learning?
... phosphorylation of TRAF2 inhibits binding to the CD40 ...
Phosphorylation
Regulation
BindingTheme
Cause Theme
ThemeTheme
Regulation BindingPhosphorylationTheme
Cause
ThemeTheme Theme
Same Binding
2 3 4 5 6 7 8 9
b4,9
e2,Phos.a6,9,Theme
(a)
(b)
Figure 1: (a) sentence with target event structure; (b) pro-jection to labelled graph.
sentence, as seen figure 1b).
We will first present some basic notation to sim-plify our exposition. For each sentence x we havea set candidate trigger words Trig (x), and a set ofcandidate proteins Prot (x). We will generally usethe indices i and l to denote members of Trig (x), theindices p, q for members of Prot (x) and the index jfor members of Cand (x) def= Trig (x) � Prot (x).
We label each candidate trigger i with an eventType t � T (with None � T ), and use the binaryvariable ei,t to indicate this labeling. We use binaryvariables ai,l,r to indicate that between i and l thereis an edge labelled r � R (with None � R).
The representation so far has been used in previ-ous work (Riedel et al., 2009; Björne et al., 2009).Its shortcoming is that it does not capture whethertwo proteins are arguments of the same bindingevent, or arguments of two binding events with thesame trigger. To overcome this problem, we intro-duce binary “same Binding” variables bp,q that areactive whenever there is a binding event that hasboth p and q as arguments. Our inference algorithmwill also need, for each trigger i and protein pair p, q,a binary variable ti,p,q that indicates that at i there isa binding event with arguments p and q. All ti,p,q aresummarized in t.
Constructing events from solutions (e,a,b) canbe done almost exactly as described by Björne et al.(2009). However, while Björne et al. (2009) grouparguments according to ad-hoc rules based on de-pendency paths from trigger to argument, we simplyquery the variables bp,q.
3 Model
We use the following objective to score the struc-tures we like to extract:
s (e,a,b) def=�
ei,t=1
sT (i, t) +�
ai,j,r=1
sR (i, j, r) +
�
bp,q=1
sB (p, q)
with local scoring functions sT (i, t) def=�wT, fT (i, t)�, sR (i, j, r) def= �wR, fR (i, j, r)�and sB (p, q) def= �wB, fB (p, q)�.
Our model scores all parts of the structure in isola-tion. It is a joint model due to the three types of con-straints we enforce. The first type acts on trigger la-bels and their outgoing edges. It includes constraintssuch as “an active label at trigger i requires at leastone active outgoing Theme argument”. The secondtype enforces consistency between trigger labels andtheir incoming edges. That is, if an incoming edgehas a label that is not None, the trigger must not belabelled None either. The third type of constraintsensures that when two proteins p and q are part ofthe same binding (as indicated by bp,q = 1), thereneeds to be a binding event at some trigger i thathas p and q as arguments. We will denote the set ofstructures (e,a,b) that satisfy all above constraintsas Y .
To learn w we choose the passive-aggressiveonline learning algorithm (Crammer and Singer,2003). As loss function we apply a weighted sum offalse positives and false negative labels and edges.The weighting scheme penalizes false negatives 3.8times more than false positives.
3.1 FeaturesFor feature vector fT (i, t) we use a collection ofrepresentations for the token i: word-form, lemma,POS tag, syntactic heads, syntactic children; mem-bership in two dictionaries used by Riedel et al.(2009).For fR (a; i, j, r) we use representations ofthe token pair (i, j) inspired by Miwa et al. (2010) .They contain: labelled and unlabeled n-gram depen-dency paths; edge and vertex walk features (Miwa etal., 2010), argument and trigger modifiers and heads,words in between (for close distance i and j). ForfB (b; p, q) we use a small subset of the token pairrepresentations in fR.
47
Task: Extract biomedical informa%on from text.
Other NLP Tasks? Hoifung Poon and Pedro Domingos obtained great NLP results with MLNs: • “Joint Unsupervised Coreference Resolu%on with Markov Logic,” ACL 2008. • “Unsupervised Seman%c Parsing,” EMNLP 2009.
Best Paper Award. • “Unsupervised Ontology Induc%on from Text,” ACL 2010.
…but Hoifung hasn’t used Markov logic in any of his follow-‐up work: • “Probabilis%c Frame Induc%on,” NAACL 2013.
(with Jackie Cheung and Lucy Vanderwende) • “Grounded Unsupervised Seman%c Parsing,” ACL 2013. • “Grounded Seman%c Parsing for Complex Knowledge Extrac%on,” NAACL
2015. (with Ankur P. Parikh and Kris%na Toutanova)
MLNs were successfully used to obtain state-‐of-‐the-‐art results on several NLP tasks. Why were they abandoned? Because it was easier to hand-‐code a custom solu%on as a log-‐linear model.
Soqware • There are many good machine learning toolkits – Classifica%on: scikit-‐learn, Weka – SVMs: SVM-‐Light, LibSVM, LIBLINEAR – Graphical models: BNT, FACTORIE – Deep learning: Torch, Pylearn2, Theano
• What’s the state of soqware for rela%onal learning and inference? – Frustra(ng. – Are the implementa%ons too primi%ve? – Are the algorithms immature? – Are the problems just inherently harder?
Hopeful Analogy: Neural Networks
• In computer vision, specialized feature models (e.g., SIFT) outperformed general feature models (neural networks) for a long %me.
• Recently, convolu%onal nets are best and are used everywhere for image recogni%on.
• What changed? More processing power and more data.
Specialized rela%onal models are widely used. Is there a revolu%on in general rela%onal learning wai%ng to happen?
Conclusion
• Many kinds of rela%onal data and models – Specialized rela%onal models are clearly effec%ve. – General rela%onal models have poten%al, but they haven’t taken off.
• Ques%ons: – When can effec%ve specialized representa%ons become more general?
– What advances do we need for general-‐purpose methods to succeed?
– What “killer apps” should we be working on?