Out of the Lab and Into the Field Real Search and Big Data Implementations

The Evolution of Search and the Union with Big Data

Kamran KhanPresident & CEO16th November 2017

#My name is Kamran Khan and Im the CEO and founder of Search Technologies.

Ive presented a few times at this event and some of my earlier presentations introduced the idea that search and big data are complimentary technologies and when used together can be very powerful. 4 years on from my first presentation about that topic, we now have a number of use cases we built for customers that demonstrate exactly that and Id like to share a few of those use cases and give you a practical view into the problems they are solving. 1

AgendaAbout Search TechnologiesChanges in the Search Space Over the Past Decade Search & Big Data Case Studies Making Implementations EasierClosing Remarks

#Search Technologies: Who We Are

The leading independent IT services firm specializing in the design, implementation, and management of enterprise search and big data search solutions.

#3

200+ People800+ Customers

#We started just over 10 years ago and in that time weve grown to over 200 people in 8 offices around the world. Weve delivered services to over 800 customers in many industry verticals and weve done that while growing the company quite consistently and profitably. In year 1, our projects were mainly centered around search technologies like FAST, GSA and Solr but today, our projects invariably include a Big Data platform of some flavor as well as analytics tools.4

Profitable7$32M FY16

#

5

Consolidation

#Larger players bought smaller players (FAST bought by Microsoft, Endeca by Oracle, Autonomy by HP, Vivisimo by IBM). This changed the direction of nearly all of these products

The rise of open source (Solr Elastic). More than 50% of our revenue based on open source. Some very large implementations now driven by Solr/Elastic search

6

#GSA has exited the market 7

The Rise of Open Source

#

8

Search Can Power Big Data AnalyticsOur Vision

You know, people today think that search and big data are separate but in two or three years, everyone will wonder why we ever thought that.

- Doug Cutting

Search is very good at finding interesting things buried under mounds of information. It can provide access across a wide variety of content types and a flexible, and intuitive interface for exploring data. - Darin Stewart

#9The merging of search and big data

Forrester Validates Search Market EvolutionModern search is about knowledge, not keywords

New data types and sources

New query methods

New methods to display results

Search applications enable new use cases

VS. simple standalone search

*Forrester Wave: Big Data Search and Knowledge Discovery Solutions Q3 2015

#The merging of search and big data

Search is getting smarter10

#The merging of search and big data

11

Insider Threat Detection

Insider Threat Detection

#Currently working with several government organizations worldwide to lead big data intelligence analysis projects.

Insider threats caused by malicious or accidental offenders can put an organization and data at risk. The threats can come from not only internal staff but also outsiders that have access to certain organizational data (partners/suppliers). The more access privileges, the higher risk.

While the public might be most familiar with intelligence situations like the Edward Snowden, the reality is that insider threat is a critical concern for all modern businesses. 12

#The merging of search and big data

13

#Big Data in 2025 Per Year Growth Estimates

1-17 PB/Year1 EB/Year1-2 EB/Year2-40 EB/YearRef: Big Data Astronomical or Genomical 2015 Stephens et al

#Sequencing your genome will be a regular part of everyones patient care in the future.A genomic sequence can tell your doctor about your current health and potentially about your future risks.

This data adds up over time.

The scale of data for Genomics is NOT of the retail / e-commerce data size.Genomics data will be bigger than Twitter, bigger than YouTube. Its on the scale of Internet of Things data.And its this data that will be analyzed to help design customized treatments for diseases.15

#

#Components in the ST Enterprise Search StackOpen Source Search EngineElasticsearch or SolrNo-SQL DatabaseContent ProcessingStaging RepositoryNode.JS Application ServerAngular & Bootstrap End-User InterfaceAdmin User InterfaceThe ST Enterprise Search StackSearch UISearch EngineNode.JSContent ProcessingStaging Repository No-SQLDBAdmin UI

#Big Data The Open Source Search Stack

API Server (Node.JS)Web Browser

Search Engine (Elasticsearch or Solr) No-SQLDatabase

MobileWeb

Apps

Staging Repository (Node.JS)

Aspire

SharePoint

File Shares

RDBMSApps

DataWarehouse

Other SourcesBig Data FrameworkEntire Stack runs insideClouderaReplace search engine with Cloudera Search

HBaseCloudera Search

#Thank you! Come visit us at Booth 200

@SearchTechCorpwww.searchtechnologies.comVisit our partners: Sinequa: Booth #106Hitachi: Booth #208

#We have a lot more interesting use cases which are documented on our website in the form of video blogs. Feel free to stop by our booth if you want more information here at the show.20