IR: open source state

Dmitry Kan, AlphaSense, Insider Solutions

University of Helsinki, Information Retrieval and Search Engines course, Feb 21, 2017

About me● PhD in CS (Saint Petersburg State University), 2011

● Running a Search Engine team at AlphaSense since 2014

● Founded Insider Solutions in 2009: text analytics solutions +

consulting

● Co-committer on luke project: toolbox for Lucene index since 2013

What is AlphaSense● Google for financial analysts● Semantic research engine● Edit, tag, annotate, share you data in a team● Oracle, JP Morgan, Credit Suisse● Engineering is 98% in Helsinki + 1% NYC + 1% India● #1 fastest growing IT startup in Finland by Deloitte

(2015)

www.alpha-sense.com

● Founded 2009

● BigText Analytics APIs and on-premise solutions

○ Sentiment analysis: Russian, Chinese, English

○ Searchable trend extraction

● Consulting: startups and corporates

https://semanticanalyzer.info

Insider Solutions

Outline● Search engine architecture

● Open source search ecosystem

● Research directions for applied IR

Search engine: building blocks● Web crawler: Apache Nutch (based on Hadoop)

● Data ingestion pipeline: receiving, cleaning, data

extraction

● SolrCloud OR Elasticsearch (both based on Lucene)

● Shards: storing index on disk and / or memory

Lucene / Solr history timeline

Inject URLs

Create segments

New URLs

Search Engine Software Components● Schema

● Query parser

● Scoring algorithm

● Snippet highlighter

● Index (on-disk or in-memory)

Query analysis and suggestions

British vs US English handling

One shard of the index

Content extractionApache Tika for parsing formats:

● Html, XML

● PDF

● Microsoft Office & iWorks document formats

● Audio, image, video

● Mail

● Source code

Inspecting Lucene index with LukeImplemented by Andrzej Bialecki. Since 2013 → by Dmitry Kan (Finland) and Tomoko Uchida (Japan)

● Perform index maintenance

● Prototype similarity functions

● Search for documents, reconstruct field values from the index

● Read index from HDFS (Hadoop’s distributed file system)

● Supports Apache Solr and Elasticsearch

Learning to rank: SolrContributed by Bloomberg

Machine learnt model for reranking documents based on user feedback

Trained on features: views, popularity, was hit in the title, length, can view on mobile device?

LamdaMART, RankSVM

Lucene scoring formula

Feature: is person and executive?

Feature: recency of the document

Features as signal of result importance

Learnt model

Word vectors with LuceneWord2vec was released by Google to open source

Possible to train word2vec on Lucene index: https://github.com/kojisekig/word2vec-lucene

● NO need to provide a text file besides Lucene index

● NO need to normalize text. Normalization already done in the index or

Analyzer does it for you when processing

● Use part of the index by specifying a filter query

Questions?Reach me at:

dk@semanticanalyzer.info

Twitter: @dmitrykanQuora: https://www.quora.com/profile/Dmitry-Kan

References1. Luke: https://github.com/DmitryKey/luke2. My blog: http://dmitrykan.blogspot.fi/3. Solr vs Elasticsearch (overview): https://sematext.com/blog/2015/01/30/solr-elasticsearch-comparison/4. Solr vs Elasticsearch (in-depth): https://sematext.com/blog/2012/08/23/solr-vs-elasticsearch-part-1-overview/5. Introduction to Apache Solr http://www.slideshare.net/ChristosManios/introduction-to-apache-solr-540761896. Word2vec-lucene: https://github.com/kojisekig/word2vec-lucene7. Apache Tika: https://tika.apache.org/8. Apache Solr: http://lucene.apache.org/solr/9. Elasticsearch: https://github.com/elastic/elasticsearch

10. Learning to rank in Solr (video): https://www.youtube.com/watch?v=M7BKwJoh96s11. Learning to rank in Solr (slides): https://lucidworks.com/2016/08/17/learning-to-rank-solr/12. Word2vec: https://en.wikipedia.org/wiki/Word2vec#Analysis13. Lucene scoring formula:

https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html