Deep learning pour la RI - Laboratoire d'informatique de Paris...

Deep learning pour la RI

© Some slides are directly extracted fromhttp://nn4ir.com/

o Traditional approach

o Modern approach (guided by machine learning)

n Conducting query and document understandingn Representing query and document as multiple feature vectorsn Calculating multiple matching scores between query and document n Training ranker with matching scores as features using learning to rank

From traditional to modern approaches in IR

Recherche d'information textuelle 266

RSV(d, q)

f(d, q)

Lexical vs. Semantic matching


© http://nn4ir.com/

o Pioneer work for text semanticsn Word embeddings

o Skip-gram & CBOWo Following methods: Gloves,

FastText, …

o Principle: learn wordrepresentations using contextualinformations

o Unsupervised methods (not task-driven)

Deep learning for semantic matching


From Word to Document/Queryembeddings



From Word to Document/Queryembeddings



Dual Embedding Space Model (DESM) [Nalisnick et al., 2016]



Dual Embedding Space Model (DESM) [Nalisnick et al., 2016]



Neural Translation Language Model [Zuccon et al., 2015]


© Zuccon et al. 2015


Neural Translation Language Model [Zuccon et al., 2015]



Pre-trained word embedding for queryterm weighting [Zheng and Callan, 2015]



Pre-trained word embedding for queryexpansion



Pre-trained word embedding for queryexpansion



o Document/query representations are generallylearnt independently from the retrieval taskn Well designed for semantic matchingn Not specifically designed for relevance matching

o One solution is to guide representation according to the retrieval taskn Goal of retrieval task: rankingn Need to build neural network able to rank documents

(learning-to-rank)

From document/query representations to end-to-end neural IR models


n Representation-basedneural models

n Interaction-based neural models

End-to-end neural IR approaches


© Nguyen 2018

© Nuyen

o Triplet loss

Learning-to-rank with neural models


©https://filebox.ece.vt.edu/~jbhuang/teaching/ece6554/sp17/lectures/Lecture_08_Siamese_Triplet_Networks.pdf

! =max(cos *, , + −cos *, , − +/01234, 0)

Representation-based neural modelsDeep Semantic Matching Model (Huang et al, 2014)


o Word representations: letter trigram

o Document scoring

o Loss



n Dataset: commercial search engine (over 16.000 queries)



Representation-based neural modelsAn extension with convolution (sentence level)



Interaction-based neural models



Interaction-based neural modelsDeep Relevance Matching model [Guo et al., 2016]


o Histogramsn Count-based Histogram (CH): the count of local

interactions in each bin as the histogram value.n Normalized Histogram (NH): normalization of the count

value in each bin by the total count. n LogCount-based Histogram (LCH): logarithm applied over

the count value in each bin.

o Loss





o Challengesn Traditional baselines as BM25 are difficult to beatn Relatively good results on short texts (titles) but not

on long text

o Solutions

Representation-based neural modelsLimitations and challenges



Neural Ranking Models with WeakSupervision [Dehghani et al., 2017]



Neural Ranking Models with WeakSupervision [Dehghani et al., 2017]



o Knowledge resources (thesaurus, nomenclature, …)

Another type of semantics


© Nguyen

o Graph-oriented knowledge resources

Another type of semantics


Comment on fait en RI classique ?


RI sém

antique R

I Conceptuelle

Désambiguïsation des mots (Voorhees 93, Sanderson97, Gonzalo 98, Baziz et al. 05,Boughanem et al.10)

Reformulation de la requête (Voorhees 99, Baziz et al.03, Croft et al. 2011)

RI basée sur les concepts (Bedersky et al, 2012)

�

�

�

Approches basées sur l’analyse de corpus

•Proximité des mots : Bi-grammes, Bi-termes, , Apprentissage profond (deep learning)

Approches basées sur des ressources sémantiques

•Ontologies : (WordNet), thésaurus (UMLS) , Encyclopédie (wikipedia)

© Lynda Said L’hadj

Approches basées sur l’analyse de corpus


Travaux Descripteurslongs Relations Pertinence

revisitéeAuteurs Description

Berger & LaffertySIGIR 99

Statistical translation Modélisation de la synonymie par un modèle de cooccurrence

- + +

Song & CroftSIGIR

A general language model for information retrieval

Modèle de langue de bigrammes + - -

Srikanth & srihari(SIGIR’02-03)

Bi-term language Model Modèle de langue de bi-termes et concepts (bigrammes revisités)

+ - -

Gao et al. (SIGIR’04)

Dependence language model for information retrieval

Liens pondérés entre les mots de la requête modélisée par un graphe de liens L

- + -

Metzler & croft (SIGIR’05)

A markov random field model for term dependencies

Liens pondérés entre la requête et le document (FI,SD,FD)

- + +

Shi & Nie (SIGIR’09)

Phrase inseparability in phrase-based model

Mots composés identifié selon un facteur discriminant + + +

Hammache et al (KIS’ 13)

Compound and singleterms under LM

Mot composés identifiés selon facteur de dominance + + +

Bendersky et al(SIGIR 12)

Modeling higher-order term dependencies in IR using query hypergraphs

Concepts identifiés selon un processus d’apprentissage + - +


Approche basée sur des ressources sémantiques


Travaux Descripteur

Long Désambiguïsa

tionRelation

s Pertinen

ce revisitée Auteurs Description

Voorhees et al.SIGIR’94

Query expansion using lexical-semantic relations

Représentation de la requête étendue par un vecteur Résultat négatif (requête longues)

- - + -

Baziz et al. RSTI’ 03

expansion de requêtes

Expansion prudente (long concepts). + + + +

Baziz et al. CoLIS'05

Document and queryexpansion (Doc Core)

Noyau sémantique des documents. Désambiguïsation basée sur les relations sémantiques

+ + + +CF.IDF

Baziz et al. RIAO’07

IR driven by ontology from query to document expansion. (Doc Tree)

Représentation par les sous arbres appariement flow . Meilleure que DoCore

+ + + +

Cao et al. SIGIR’05

WordNet relationshipsin LM

Combinaison linéaire de (UL,LM,CM), (UL,LM)-- - - + -

Bao et al. 05 Language SenseModel

Lissage sémantique des mots avec leurs sens (Requêtes courtes ++)

- + +

Zhou et al. 08 Topic signature language models

Lissage du ML par des concepts + - - -

Tu et al. 11 Wikipedia-basedsemantic smoothing

Lissage du ML par des concepts + + - -

Meij et al. 12 Conceptual language models for domain-specific retrieval

Réécriture de la requête avec les mots simples engendré par les concepts des document feedback

- + -


o Word embeddings are not sufficient to conveymeanings expressed in knowledge resources

n Hybrid approaches for representing word/documents

n End-to-end approach leveraging knowledge resources

And with deep learning methods?


o Triplet embedding [Bordes et al., 2013] – TransE, TransR, …n Concepts h and tn Related by l

Knowledge-guided word representations


o RC-NET [Xu et al., 2014]



o Retrofitting word representations [Faruqui et al., 2015]n 2 steps:

o Estimate word embeddingso Retrofit word embeddings with KB

n New embeddings close to first onesn Close to concept embeddings in the KB



Knowledge-guided word and concept representations


o [Mancini et al., 2017]

Knowledge-guided word and concept representations


??????

Future direction in IR


o GAN

Future directions of deep learning in IR


Source : http://wnzhang.net/tutorials/sigir2018/docs/sigir18-irgan-full-tutorial.pdf

IRGAN (Wang et al., 2017)

Recherche d'information textuelle 305Source : http://wnzhang.net/tutorials/sigir2018/docs/sigir18-irgan-full-tutorial.pdf

IRGAN (Wang et al., 2017)

Recherche d'information textuelle 306Source : http://wnzhang.net/tutorials/sigir2018/docs/sigir18-irgan-full-tutorial.pdf

o Reinforcement learning ?

Future directions of deep learning in IR


Source : http://wnzhang.net/tutorials/sigir2018/docs/sigir18-irgan-full-tutorial.pdf

o Reinforcement learningn User modeling: sequence of actions, predicting next

steps, …n Chatbot !



o Reinforcement learning for natural languageunderstanding (Nogueira et al., 2018)




o Reinforcement learning for natural languageunderstanding (Aissa et al., 2018)


Bibliographieo Ouvrages généraux

n Baeza-Yates R, Ribeiro-Neto B., 1999, Modern Information Retrieval, Addison-Wesleyn Manning D., Schütze H., 1999, Foundations of statistical natural language processing, MIT pressn Christopher D. Manning, Prabhakar Raghavan and Heinrich Schütze, Introduction to Information

Retrieval, Cambridge University Press. 2008.n Amini M., Gaussier E., Recherche d’information, Eyrolles 2012

311Recherche d'information textuelle

Date post:	18-Jun-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Deep learning pour la RI - Laboratoire d'informatique de Paris...

Documents