Post on 13-Jul-2015
transcript
WHERE DO WE COME FROM?
• ActivePivot by QuartetFS: – InMemory ultra fast business
intelligence tool – Mainly for traders and market risk
analysts
• One of the biggest french success (& secret) story
• We are big data crunchers for a long time now
HOW DID EVERYTHING START?
• Created in may 2012
• We wanted to build the perfect tool to understand the social WEB
• We started with a very famous
NoSQL engine ! But we quickly had problems
(performance, clustering, query/txns overlaps, etc…)
ELASTICSEARCH 2 YEARS AGO
BigData'ready''Easy'to'use/manage''Performance/Scalability''Analy:cal'capabili:es''Primary'document'storage?'
IT DIDN’T MATCH OUR ANALYTICS REQUIREMENTS
• FieldData cache – High memory consumption – Memory is expensive on the cloud
• No multi-field & multi-metric aggregations – We could not build the product of our dreams
• But ElasticSearch is modular – We decided to implement our own analytics plugin
A SEARCH ENGINE WITH OLAP SKILLS
• Support lazy loading of fields
• Multi-Fields & Multi-Metric aggregations
• Ultra-Fast & Efficient – Usage of columnar storage with primitive types – Sub-seconds queries over tens of millions of elements
ELASTICSEARCH GETS EVEN BETTER
• Release 1.0.0 – February 2014 – First version of the Aggregation Engine – Introduction of doc_values
• Release 1.2.0 – May 2014 – global_ordinals / Faster Aggregations
• Release 1.4.0 – November 2014 – Improv. Circuit Breakers / Safer Aggregations – Improv. doc_values
• Every releases since 1.0.0 – More stability – More aggregation capabilies
• We had more time to develop other things!
INTRODUCING LOGMATIC LOG GOODNESS POWERED BY ELASTICSEARCH
• Introduced Logmatic.io in private beta this year – Beginning of 2014: A lot of logs projects around us – Our logs experiment: It was an eye opener! (30 VMs / ~6 apps) – 2 friendly startups tried: they went live – Market Study: 12 projects launched – We faced new challenges and had to build a new product!!
OUR APPROACH
Customer’s applications Our cloud based infrastructure
CTO, devops, developers Tomorrow, we’ll even have business people. We’ll tell you more…
alerts, reports'
queries'
• Centralises & enriches all data
• Fully hosted (SaaS) • Advanced analytics • Real-Time
• Beautiful dataviz • Rapidfire answers
secured connection'
And'more…'
UDP'TCP'(SSL)'
HTTP(S)'
INCOMING FEATURES
• Integrated Grok parsing: – Logs shipping should be
the only concern – Logs structuration is done
totally in the cloud – We extended Grok to
simplify issues like date parsing
INCOMING FEATURES
• Integrated Grok parsing: – Logs shipping should be
the only concern – Logs structuration is done
totally in the cloud – We extended Grok to
simplify issues like date parsing
• And much more… – Security and limited views – Dimension contexts – Complex metrics /
formulas
WE CONTINUE WITH ELASTICSEARCH
• Scalability
• Heterogeneity
• Query performance
• Great analytics
• Reactivity of the team