Science in Data Science Science in Text Mining
by Tanay Chowdhury
Data Scientist,Zurich North America
Word2vec model • Creates the vector representation of every word using one-hot
encoding.
• The displacement vector (the vector between two vectors) describes the relation between two words.
• Comparing displacement vectors helps find similarity. Eg: vqueen−vking=vwoman−vman.
Skip gram model It’s a simple neural network with a single hidden layer.
• For training words w1,w2, . . . ,wt proper position of output word wo for input word wI is found by minimizing the loss function
• Weight update equation using stochastic gradient descent.
• Then back propagation for learning rate is
Skip gram model
Skip gram model If we now re-compute the probability of each word given the input word queen, we see that the probability of king given queen goes up
Softmax regression
Weight update Backpropagation
Example available at : https://github.com/fbkarsdorp/doc2vec/blob/master/doc2vec.ipynb
cbow model
cbow model • If the context C comprises multiple words, instead of copying the
input vector we take the mean of their input vectors as our hidden layer:
• The update functions remain the same except that for the update of the input vectors, we need to apply the update to each word in the context C:
•
Softmax • Softmax regression (or multinomial logistic regression) is a generalization
of logistic regression to the case where we want to handle multiple classes.
• In text classification every word in vocabulary is a label, making it a multiclass problem.
• Hence using softmax regression, we can compute the posterior probability P(wO|wI)
Hierarchical Softmax • It consists of a binary Huffman tree based on word frequencies,
with each word wo as leaf.
• Probability of the path from root to the word causes normalization.
• One N-way normalization is replaced by a sequence of O(logN) local (binary) normalizations, as it is all about traversing subtree with higher probability.
•
“.”makes the result -1
or +1
Negative Sampling
• The objective is to distinguish the target word from draws from the noise distribution using logistic regression, where there are k negative samples for each data sample.
• References • Gutmann and Hyvarinen 2012 . Noise-contrastive estimation of unnormalized statistical models, with
applications to natural image statistics. • Mnih and The 2012. A fast and simple algorithm for training neural probabilistic language models.
Vectoring paragraph
• Paragraph Vector model attempts to learn fixed-length continuous representations from variable-length pieces of text.
• Some of the attempts made in that direction are paragraph2vec,vector averaging and clustering.
• Reference : Le & Mikolov (2014)
Pragraph2vec
Pragraph2vec
• At each step h is computed by concatenating or averaging a paragraph vector d with a context of word vectors C
• The weight update functions are the same as in Word2Vec except that paragraph vectors also gets updated.
Pragraph2vec • Vector representation of a paragraph with different length
sentence.
Paragraph w/17
different words
Pragraph2vec
Paragraph vector of
dimension 17x3
Document vector of
dimension 5x3
Softmax regression
Weight update
Vector Averaging
• Vector averaging is the simplest technique to get the vector representation for a sentence.
• It calculates the sum of all word vectors and divides that with number of words.
• It’s considered fastest but not so accurate in result.
Clustering method
• It creates predetermined number of cluster of words.
• Synonymous words (mainly adjectives) ends up in same cluster.
• Rather than using bag of words, it uses bag of centroids (counting the occurrence of a word in a certain cluster)then.
• Reference : https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-3-more-fun-with-word-vectors
Performance comparison
• Paragraph2vec is generally the most accurate one when used in classification algorithm.
• Vector averaging is the fastest but clustering is considered to be decent in terms of accuracy and speed.
Questions
