Topic ModellingAssigning topic to any text

Team SMMMohit Sharma 201505508Hari Naga Raghavendra Manohar 201505551K S Chandra Reddy 201505544

Aim

To identify small number of topics/categories that best characterize a given document.

The Categories we considered are:

wiki, art, reference, people, culture, books, design, politics, technology, psychology, interesting, wikipedia, research, religion, music, math, development, theory, philosophy, article, language, science, programming, history and software.

Dataset

1.2.

Approach Used:LDA

We can use LDA to classify documents in different tags. We know that LDA divides the given corpus in fixed no. of topics and can also provide which topics are contained in a document and with what probability. For the experiments performed using LDA, we don’t need to worry about internal implementation of LDA. We used gensim’s implementation of LDA. To use the library, we just need to know few points about input and output format. Read the documentation on following link.https://radimrehurek.com/gensim/wiki.html

During Learning phaseINPUT:We provide all the wiki documents in single XML file zipped in bz2 format.

LEARNT MODEL:Word distribution for each topic eg: “topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam”

During Testing phaseINPUT:We provide the document to be classified in bag of words form to the learnt model

OUTPUT:Topic distribution for a the text eg: “[(34, 0.023705742561150572), (60, 0.017830310671555303), (62, 0.023999239610385081), (83,0.029439444128473557), (87, 0.028172479800878891), (90, 0.1207424163376625), (116,0.022904510579689157)]” represents the probabilities of the doc to fall under topics like 34,60,62….

Major challenge in classification:

It seems to be fairly simple to classify a document in different topics as we can see in output of testing phase. But our aim is to classify the document under different tags like “politics, science” etc. and not under topic numbers.

Possible SolutionsClearly we need some way to map all the topics learnt by LDA to the most suitable tags. If we are able to do this then we simply test the unknown text against the model learnt by LDA and then report the tag corresponding to the topic given by LDA in output. We tried two different solutions to map topics to the tags:

1. As each topic of LDA is represented by distribution of words. We can create a query by combining those words and find best matched document on tf-idf basis for that query. That particular document must be the best match for that topic. So we can map the topic to tag of best matched document.

2. We can find probability distribution of topics for all the documents. Represent each document as a topic vector. Now find the closest document or the most similar document for each topic. Map the topic to the tag of that particular document.

Approach 1We can specify the major steps to implement this approach as follows:

1. Divide the documents in training and test data with 4000 docs in test data.2. On training data run gensim's LDA and save the learnt model. Set the number of topics as 300.3. Save all the topics in a file and convert them to queries. Example topic: 2016-04-06 00:05:52,466 : INFO : topic #299 (0.003): 0.014*insurance + 0.009*scott + 0.007*samurai + 0.007*hipster + 0.006*forecasting + 0.006*fbi + 0.006*imf + 0.005*skeptical + 0.005*bass + 0.005*hidden Query corresponding to above topic#299: 299:insurance scott samurai hipster forecasting fbi imf skeptical bass hidden

4. For each query, retrieve the most relevant document in training set on tf-idf basis and create topic to doc Id mapping. Example: 299:cae3757420fbc4008bbfe492ab0d4cb5

5. Create a topic to tag mapping using the docId to tag mapping (already available in tagData.xml) and doc ID to topic mapping created in above step.

Example docId to tag from tagData.xml: cae3757420fbc4008bbfe492ab0d4cb5 : ['wiki', 'en', 'wikipedia,', 'activism', '-‘, 'political', 'poetry', 'free', 'person', 'music', 'encyclopedia', 'the', 'biography', 'history'] Example topic to docId: 299:cae3757420fbc4008bbfe492ab0d4cb5 Example topic to tag: 299:['wiki', 'en', 'wikipedia,', 'activism', '-', 'political', 'poetry', 'free', 'person', 'music‘, 'encyclopedia', 'the', 'biography', 'history'] Now each topic is mapped to multiple tags.

6. For each of the test documents (from 4000 docs in test data), find out the relevant topics using learnt LDA model. Combine the tags corresponding to them and match them against already available target tags

(from tagData.xml) for that particular document. If even one tag is matched, we say that document is correctly classified.

Example:Topic distribution returned by LDA for a particular doc:[(34, 0.023705742561150572), (60, 0.017830310671555303), (62, 0.023999239610385081), (83,0.029439444128473557), (87, 0.028172479800878891), (90, 0.1207424163376625), (116,0.022904510579689157), (149, 0.010136256627631658), (155, 0.045428499528247894), (162,0.014294122339773195), (192, 0.01315170635603234), (193, 0.055764500858303222), (206,0.015174121956574787), (240, 0.052498569359746373), (243, 0.016285345117555323), (247,0.019478047862044864), (255, 0.018193391082926114), (263, 0.030209722561452931), (287,0.042405659613804568), (289, 0.055528896333028231), (291, 0.030064093091433357)]

Tags combined for above topics (from topic to tag mapping created in above step):['money', 'brain', 'web', 'thinking', 'interesting', 'environment', 'teaching', 'web2.0', 'bio','finance', 'government', 'food', 'howto', 'geek', 'cool', 'articles', 'school', 'cognitive', 'cognition','energy', 'computerscience', '2read', 'culture', 'computer', 'video', 'home', 'todo', 'investment','depression', 'psychology', 'wikipedia', 'research', 'health', 'internet', 'medicine', 'electronics','tech', 'math', 'business', 'marketing', 'free', 'standard', 'interface', 'article', 'definition','anarchism', 'of', 'study', 'economics', 'programming', 'american', 'games', 'advertising', 'social','software', 'apple', 'coding', 'maths', 'learning', 'management', 'system', 'quiz', 'pc', 'music','memory', 'war', 'nutrition', 'comparison', 'india', 'info', 'science', 'dev', '@wikipedia', 'future','behavior', 'design', 'history', '@read', 'mind', 'hardware', 'webdev', 'politics', 'technology‘]

Target tags for this particular doc from tagData.xml:['reference', 'economics', 'wikipedia', 'politics', 'reading', 'resources']

Accuracy from this approach: 97%

Problem with this approach:1. If there is any match between our found tags and true tags, then we call it as correctly classified. Probability

of such scenario is very high as we have multiple found tags and multiple true tags. So even if we are doing something wrong, chances of getting good accuracy is very high.

2. As we are doing tf-idf based matching then there is high chance that the document we get on top is not best match for that particular topic. It can also happen because we are not considering all the representative words of a particular topic to frame the query, we just considered top 10.

Approach 2After analyzing the data we found that only 25 of the tags represent around 19K documents out of 20K. Which simply means that we can eliminate the less frequent tags and docs corresponding to them. Which means we have to divide the corpus among 25 topics at most. Which makes it easier to implement approach 2, as each document can be easily represented in 25 dimensional topic space. We can specify the major steps to implement this approach as follows:

1. Eliminate the less frequent tags and documents related to them. keep only top 25. Docs left will be around 19K.

2. On complete data run gensim's LDA and saved the learnt model. Set number of topics set as 25.3. Save all the topics in a file and convert them to queries as in previous approach.4. Test each of 19K documents against the learnt model and find the topic distribution eg:

“42d1d305d10b4b025e01e8237c44c87e:0 0 0 0 0.0242823647949 0 0.037682372871 0 0 0 0.0988683434224 0.0113662521741 0.0157100377468 0 0 0.182273317591 0.205447648234 0 0.0524222798936 0.167240557357 0 0.178899361052 0 0 0” represents the probabilities of the doc with given id in 25 different topics.

5. Using above distribution find out the most relevant document for a particular topic and map it to the tag of that document. It gives the similar topic to tag mapping as in previous approach.

6. Now many topics must have matched to more than one tag. Manually check which tag is best suited for that particular topic depending on words contained in the topic. As a result we have each topic mapped to at most one tag.

7. Now perform the testing as done in step 6 of previous approach but on all 19K docs.

Accuracy from this approach: 88%

Problem with this approach:1. Mapping topics to tags manually is an issue. We can’t always find out the best suited tag just by seeing the

topic words. Sometimes tags don’t reflect anything eg: ‘wikipedia’, ‘wiki’, ‘reference’ create problem.

Modification:Performed the above experiment again but just with meaningful tags i.e. no tag like ‘wikipedia’, ‘wiki’,‘reference’ etc. After eliminating these documents left were 17K. But the approach posed another issue:

1. There are similar tags which can represent a topic at the same time eg: [research, science], [web, internet], [programming, math], [literature, language].

If we keep all such similar tags then accuracy is : 80% but if we strictly keep just one tag then accuracy drops to 65%.Reason for the drop is possibly manual work. We can’t surely say which tag should be kept when both tags are same.

Conclusion: 2ND approach is better as there is very less chance of false good accuracy and accuracy is also not bad considering just ~19K documents for learning. Time reuired for leaning is aound 10 minutes and assigning topic to any new text is intantaneous.

Thank YouIIIT Hyderabad