Elasticsearch, Logstash, KibanaTechnical Walk-Through
Mark Walkom, Hat Wearer @warkolm
www.elastic.co2
Elasticsearch
www.elastic.co3
Elasticsearch Terminology
•A node is a single Elasticsearch instance, a single JVM
•Multiple nodes can form a cluster
•A cluster can manage multiple indices
•A cluster is agile & self managing
•Clusters often 3-10 nodes but can scale to 100s of nodes
•Clusters can have Petabytes of data
•Clusters can be federated for larger scale
www.elastic.co4
an open source, distributed, scalable,
highly available, document-oriented, RESTful
full text search engine
with real-time search and analytics capabilities
built on lucene and java
Elasticsearch is...
www.elastic.co5
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
Apache 2.0 License
https://www.apache.org/licenses/LICENSE-2.0
www.elastic.co6
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
www.elastic.co7
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
www.elastic.co8
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
www.elastic.co9
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
Source: http://json.org/
www.elastic.co10
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
Source: https://httpwg.github.io/asset/http.svg
www.elastic.co11
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
www.elastic.co12
an open source, distributed, scalable, highly available, document-oriented,
RESTful, full text search engine with real-time search and analytics capabilities
Elasticsearch is...
www.elastic.co13
Search
Search with Elasticsearch
www.elastic.co14
CRUD
www.elastic.co15
CRUD
www.elastic.co16
CRUD
www.elastic.co17
CRUD
www.elastic.co18
Searching
www.elastic.co19
Searching
www.elastic.co20
Aggregation
Analytics with Elasticsearch
www.elastic.co21
Aggregations
GET /person/person/_search?search_type=count{
"aggs": {
"by_country": {
"terms": {
"field": "address.country"
}
}
}
}{ ..., "aggregations" : { "by_country" : { "buckets" : [ { "key" : "England", "doc_count" : 30051 }, { "key" : "Germany", "doc_count" : 30004 }, { "key" : "France", "doc_count" : 15034 }, { "key" : "Spain", "doc_count" : 14912 } ]}}}
17%
17%
33%
33%
EnglandGermanyFranceSpain
www.elastic.co22
Histograms
GET /person/person/_search?search_type=count{
"aggs": {
"by_date": {
"date_histogram": {
"field": "dateOfBirth",
"interval": "year",
"format": "yyyy"
}
}
}
}
{ ..., "aggregations": {
"by_date": {
"buckets": [
{
"key_as_string": "1960",
"key": -946080000000,
"doc_count": 39
},
{
"key_as_string": "1961",
"key": -630720000000,
"doc_count": 12677
},
{
"key_as_string": "1962",
"key": -315360000000,
"doc_count": 12936
}, ...
]
}
}}0
7500
15000
22500
30000
1940 1950 1960 1970 1980 1990 2000 2010
www.elastic.co23
A Lot More
www.elastic.co24
More than search
Elasticsearch
www.elastic.co25
Text Analysis - Analyzers
• Tokenizer
Breaks the text into tokens and produces a token stream Example: keyword, whitespace, regex, etc...
• Token Filter
Acts on the token stream - can drop and modify existing tokens, or add new ones. Example: lowercase, stopword, ngram, etc..
www.elastic.co26
Free steak knives!
• Relational documents Parent/child Nesting
• Suggestion API Predictive typing/search
• Highlighting Emphasise results, e.g. <em>w00t</em>
• Percolators - search for searches Does this document match this search?
www.elastic.co27
Geo Search
• Geo points and shapes Polygon Polygon with holes Multi polygon
• Bounding boxes, distance from point, distance in a range
• Supports multiple coordinate formats; “location”: { "lat" : 41.12, "lon" : -71.34 } "location" : “41.12,-71.34” "location" : [-71.34, 41.12]
www.elastic.co28
Elasticsearch & Hadoop
www.elastic.co29
Elasticsearch for Apache Hadoop™
www.elastic.co30
Logstash
www.elastic.co31
Logstash
Logstash
Input OutputFilter
? ?
collect and split alter and enrich store and visualise
www.elastic.co32
Logstash
www.elastic.co33
Logstash
71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
www.elastic.co34
Logstash110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 123.125.71.35 - - [16/Feb/2014:09:49:02 -0500] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 50.150.204.184 - - [16/Feb/2014:09:49:37 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=https//:google.com&source=lnms&tbm=isch&sa=X&ei=4-r8UvDrKZOgkQe7x4CICw&ved=0CAkQ_AUoAA&biw=320&bih=441" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-us; LG-MS770 Build/IMM76I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" 207.241.237.225 - - [16/Feb/2014:09:50:06 -0500] "GET /blog/tags/examples HTTP/1.0" 200 9208 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 200.49.190.101 - - [16/Feb/2014:09:50:10 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "-" 200.49.190.100 - - [16/Feb/2014:09:50:08 -0500] "GET /blog/tags/web HTTP/1.1" 200 44019 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 200.49.190.101 - - [16/Feb/2014:09:50:12 -0500] "GET /style2.css HTTP/1.1" 200 4877 "-" "-" 200.49.190.101 - - [16/Feb/2014:09:50:19 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 66.249.73.185 - - [16/Feb/2014:09:51:19 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:26 -0500] "GET /blog/tags/munin HTTP/1.1" 200 9746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:47 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:52:34 -0500] "GET /blog/geekery/eventdb-ideas.html HTTP/1.1" 200 11418 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 67.214.178.190 - - [16/Feb/2014:09:53:19 -0500] "GET / HTTP/1.0" 200 37932 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 67.214.178.190 - - [16/Feb/2014:09:53:30 -0500] "GET /blog/geekery/installing-windows-8-consumer-preview.html HTTP/1.0" 200 8948 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 207.241.237.220 - - [16/Feb/2014:09:53:47 -0500] "GET /blog/tags/projects HTTP/1.0" 200 28370 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 46.105.14.53 - - [16/Feb/2014:09:53:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 207.241.237.227 - - [16/Feb/2014:09:53:50 -0500] "GET /blog/geekery/soekris-gpio.html HTTP/1.0" 200 9587 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "http://en.wikipedia.org/wiki/Xvfb" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:35 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Win64; x64; Trident/6.0)" 66.249.73.185 - - [16/Feb/2014:09:54:44 -0500] "GET /doc/index.html?org/elasticsearch/action/search/SearchResponse.html HTTP/1.1" 404 294 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 207.241.237.228 - - [16/Feb/2014:09:54:54 -0500] "GET /blog/tags/defcon HTTP/1.0" 200 24142 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 207.241.237.101 - - [16/Feb/2014:09:54:58 -0500] "GET /blog/tags/regex HTTP/1.0" 200 14888 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 87.169.99.232 - - [16/Feb/2014:09:56:12 -0500] "GET /presentations/puppet-at-loggly/puppet-at-loggly.pdf.html HTTP/1.1" 200 24747 "https://www.google.de/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 209.85.238.199 - - [16/Feb/2014:09:56:18 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 3 subscribers; feed-id=14171215010336145331)" 209.85.238.199 - - [16/Feb/2014:09:56:31 -0500] "GET /test.xml HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /blog/geekery/ssl-latency.html HTTP/1.1" 200 17147 "http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CE4QFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fblog%2Fgeekery%2Fssl-latency.html&ei=ZdEAU9mGGuWX1AW09IDoBw&usg=AFQjCNHw6zioJpizqX8Q0YpKKaF4zdCSEg&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:29 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 66.249.73.135 - - [16/Feb/2014:09:57:36 -0500] "GET /blog/geekery/vmware-cpu-performance.html HTTP/1.1" 200 12908 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 46.105.14.53 - - [16/Feb/2014:09:58:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 218.30.103.62 - - [16/Feb/2014:09:59:36 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:41 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:46 -0500] "GET /projects/fex/ HTTP/1.1" 200 14352 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 74.125.40.20 - - [16/Feb/2014:09:59:53 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://suckless.org/rocks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:07 -0500] "GET /projects/xdotool/xdotool.xhtml HTTP/1.1" 304 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 108.174.55.234 - - [16/Feb/2014:10:00:16 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "-" 218.30.103.62 - - [16/Feb/2014:10:00:28 -0500] "GET /blog/geekery/c-vs-python-bdb.html HTTP/1.1" 200 11388 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 121.107.188.202 - - [16/Feb/2014:10:00:28 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:52 -0500] "GET /blog/productivity/better-zsh-xterm-title-fix.html HTTP/1.1" 200 10185 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:14 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:37 -0500] "GET /blog/geekery/puppet-facts-into-mcollective.html HTTP/1.1" 200 9872 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 10756 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 218.30.103.62 - - [16/Feb/2014:10:01:57 -0500] "GET /blog/geekery/jquery-interface-puffer.html%20target= HTTP/1.1" 200 202 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:02:19 -0500] "GET /blog/geekery/ec2-reserved-vs-ondemand.html HTTP/1.1" 200 11834 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 66.249.73.135 - - [16/Feb/2014:10:02:37 -0500] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0” 66.249.73.135 - - [16/Feb/2014:10:03:25 -0500] "GET /blog/tags/bdb HTTP/1.1" 200 23099 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" 107.170.41.69 - - [16/Feb/2014:10:03:31 -0500] "GET /?flav=atom HTTP/1.1" 200 32352 "-" "Feedbin - 1 subscribers" 50.16.19.13 - - [16/Feb/2014:10:03:43 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "http://www.semicomplete.com/blog/tags/puppet?flav=rss20" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 46.105.14.53 - - [16/Feb/2014:10:03:50 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"
www.elastic.co35
Logstash
input { stdin { } }
filter { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}' } }
date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en }
geoip { source => "clientip" }
useragent { source => "agent" target => "useragent" }}
output { stdout { codec => rubydebug } }
www.elastic.co36
Logstash{ "message" => "71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566 \"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "@version" => "1", "@timestamp" => "2011-05-18T08:48:10.000Z", "host" => "bender.local", "clientip" => "71.141.244.242", "ident" => "-", "auth" => "kurt", "timestamp" => "18/May/2011:01:48:10 -0700", "verb" => "GET", "request" => "/admin", "httpversion" => "1.1", "response" => 301, "bytes" => 566, "referrer" => "\"-\"", "agent" => "\"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "geoip" => { "ip" => "71.141.244.242", "country_code2" => “US", "city_name" => "San Francisco", "timezone" => "America/Los_Angeles", "location" => [ [0] -122.4194, [1] 37.7749 ] }, "useragent" => { "name" => "Firefox", "os" => "Windows XP", "os_name" => "Windows XP", "device" => "Other", "major" => "3", "minor" => "6", "patch" => "3" }}
www.elastic.co37
Logstash
input { stdin {} }
filter { grok { match => [ message, "%{COMBINEDAPACHELOG}" ] }}
output { elasticsearch { protocol => “http”
host => “bender” }}
www.elastic.co38
Kibana
www.elastic.co39
Kibana
•Kibana 4 is a total re-architecture from 3
Nodejs + javascript
Zazzier UI
Single binary that serves itself
•Lots more functionality via aggregations
•Extensible - plugins coming real soon
www.elastic.co40
Kibana
www.elastic.co41
Kibana
www.elastic.co42
Kibana
www.elastic.co43
Found - ESaaS
•Fully Managed and Monitored Infrastructure
Automated Backups
HA - Replication and Failover
•GUI Driven, User Friendly*
•Sydney AZ very, very soon
www.elastic.co44
Elastic: Commercial Plugins
•Marvel: Monitor your Cluster
Currently KB3 based front end. v2.0 will be KB4.
•Shield: For Security
ACLs, RBAC via AD or LDAP, SSL, IP filtering, Auditing
•Watcher: Alerting on your data
Email and webhook push notifications
•More coming soon!
www.elastic.co45
Goodies
•Curator: index management https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
•Puppet & Chef modules https://forge.puppetlabs.com/elasticsearch https://github.com/elastic/cookbook-elasticsearch/
• logstash forwarder: low overhead collector https://github.com/elastic/logstash-forwarder
•grokdebugger: log pattern matching http://grokdebug.herokuapp.com/
www.elastic.co46
More Goodies
•Github: https://github.com/elastic
•Docs: http://www.elastic.co/guide/
•Forums: https://discuss.elastic.co
• IRC channels #elasticsearch, #logstash, #kibana, #beats on Freenode
•We’re hiring! [email protected], drop me an email/DM or come say Hi :)
Thanks!
Mark Walkom, Hat Wearer @warkolm