Home >Technology >Splunk conf2014 - Splunk for Data Science

Splunk conf2014 - Splunk for Data Science

Date post:05-Aug-2015
View:696 times
Download:9 times
Share this document with a friend

1. 1 Copyright 2014 Splunk Inc. Tom LaGatta Data Scientist, Splunk Olivier de Garrigues Sr Prof Services Consultant, Splunk SplunkforDataScience 2. 2 Disclaimer During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality describedortoincludeanysuchfeatureorfunctionalityinafuturerelease. 3. 3 3 Key Takeaways Splunkisgreatfor doingDataScience! Splunkcomplements othertoolsinthe DataSciencetoolkit. DataScienceisabout extrac:ngac:onable insightsfromdata. 1 2 3 4. 4 About Us Tom LaGatta, Data Scientist Tom joined Splunk in Spring 2014 as a Data Scientist specializing in Probability and Statistics. Tom is an expert on the mathematics of inference, and he enjoys functional programming in languages like Clojure, Haskell & R.At Splunk, Tom is helping to develop our internal and external Data Science program and curriculum. Tom has a PhD in Mathematics from the University of Arizona, and until recently was a Courant Instructor at the Courant Institute at New York University.TomisbasedinNewYorkCity. Olivier de Garrigues, Senior Professional Services Consultant Olivier is based in London on the EMEA Professional Services team and has helped out more than 40 customers in 10 countries on variousSplunkprojectsinthepastyearandahalf.Priortothis,heworkedasaquantitativeanalystwith extensive use of MATLAB and R. He developed a keen interest in machine learning and enjoys dreaming about how to make Splunk better for data scientists, and helped develop the R ProjectApp. OlivierholdsanMSinMathematicsofFinancefromColumbiaUniversity. 5. 5 SplunkforDataScience 6. 6 What is Data Science? Data Science is about extracting actionable insights from data. Helps people make better decisions. Can be used for automated decision-making. Data Science is cross-functional, and blends techniques & theories from: CS / Programming Math and Statistics Machine Learning Data Mining / Databases Data Visualization Dont be afraid of Data Science! Substantive / Domain Expertise Social Science Communication and Presentation Accounting, Finance and KPIs BusinessAnalytics 7. 7 Data Science & Analytics Teams There is no one size fits all data scientist. Data Science &Analytics teams are made up of people with complementary skill sets. Source: Schutt & ONeil. Doing Data Science. 2013 8. 8 Splunk for Data Science Splunk is great for doing Data Science! Integrate, query & visualize all the data: Platform for machine data Connects with any other data source Easy-to-useAnalytics capabilities. Powerful algorithms out-of-the-box. Sharp visualizations and dashboards. Deliver results to both IT & Business users. Complements other Data Science tools (next slide). 9. 9 Splunk and Data Science Tools Splunk complements other tools in the Data Science toolkit: Hadoop: the workhorse of the Data Science world. Using Hunk, you can integrate Hadoop & HDFS seamlessly into Splunk. R & Python: the preferred languages of Data Science. Execute R & Python scripts in your Splunk queries using the R ProjectApp & SDK for Python. SQL& other RDBMS: valuable stores for customer & product data. Use Splunks DB ConnectApp to mash relational data up with machine data. External tools: export finalized data from Splunk using the ODBC Driver. Tip: do all your data processing in Splunk/Hunk, and export only the final results. D3 Custom Visualizations: sharp dashboards & reports using Splunk. 10. 10 Splunk and Data Science Use Cases Green Use Cases (easy out of the box) Yellow Use Cases (needs tinkering) Trend Forecasting D3 Custom Visualizations A/B Testing Predictive Modeling Root Cause Analysis Sentiment Analysis Anomaly Detection Conversion Funnel/Pathing Market Segmentation More Algorithms via R & Python Topic Modeling Capacity Planning Correlate Data from 2+ Sources Data Munging & Normalization KPIs & Executive Dashboards Splunk is a powerful tool for lots of Data Science use cases: 11. 11 DataScienceUseCases 12. 12 Use Case: Trend Forecasting Trend Forecasting: Given past & realtime data, predict future values & events. Common applications: Forecast revenue & other KPIs Web server traffic & product downloads Customer conversion rates Estimate MTTR & server outages Resource & capacity planning (AWSApp) Security threats (Enterprise SecurityApp) The true course of events can (and will) take only one of many divergent paths. But which one? Be mindful of rare events & black swans! 13. 13 Splunk Solution: predict! predict command: forecast future trajectories of time series. Implements a Kalman filter to identify seasonal trends. Gives an uncertainty envelope as a buffer around the trend. Tip:Always run the predict command on LOTS of past data. Capture low-frequency and high-frequency trends. Remember: the future is always uncertain 14. 14 Splunk Solution: Predict App David Carassos PredictApp: forecast future values of individual events. 8 minute walkthrough: https://www.youtube.com/watch?v=ROvaqJigNFg Implements a Nave Bayes classifier. You have to train models! Train a model to predict any target field using any reference field(s): fields ref1, ref2, ..., target | train my_model from target! Guess target field for incoming events: guess my_model into target Temporal or non-temporal prediction (include _time among reference fields). 15. 15 Concept: Supervised Learning & Classification Supervised learning: use observed training data to classify values of unknown testing data. predict command (Kalman filter): Training data = timechart of past & realtime values. Testing data = time range for future values. PredictApp (Nave Bayes classifier): Training data = events with reference & target fields. Testing data = events with reference fields but not target field. Tip: only deploy models & algorithms after extensive testing & evaluation. More powerful learning algorithms using R ProjectApp or SDK for Python. 16. 16 Demo: Predict App Train a model to predict movie Rating based on MovieID, UserID, Genre,Tag index=movielens Timestamp < 1199188800 UserID=593* | eval original_rating = case(Rating3,"Like") | fields original_rating MovieID UserID Genre Tag | train rating_model from original_rating! Guess Rating for test data based on trained model index=movielens Timestamp > 1199188800 UserID=593* | guess rating_model into guessed_rating | top original_rating guessed_rating! Accuracy of model: correct on 97.6% of values. Tip: always train on LOTS of training data. Evaluate before deploying. 17. 17 Use Case: Sentiment Analysis SentimentAnalysis: the assignment of emotional labels to textual data. Can be simple +1 vs. -1, or more sophisticated: happy, angry, sad, etc. Analyze tweets, emails, news articles, logs or any other textual data! Social data correlates with other factors. Typically done via supervised learning: Train a model on labeled corpus of text. Test the model on incoming text data. Read more about SentimentAnalysis: Chapter14ofBigDataAnalyticsUsingSplunk(pp.255-282). MichaelWilde&DavidCarasso.SocialMedia&SentimentAnalysis..conf2012 3rd 8th 4th 1st 2nd 2011 Irish General Election 17% 1.8% 10% 36% 19% r=.79 18. 18 Splunk Solution: Sentiment Analysis App David Carassos SentimentAnalysisApp assigns binary sentiment values to textual data (logs, tweets, email, etc.). Nave Bayes classifier under the hood. Twitter & IMDB models out of the box. Can guess language of authorship, and heat, a measure of emotional charge. Tip: compare relative sentiment changes across time & groups. How to train your own models: http://answers.splunk.com/answers/59743 19. 19 Demo: Sentiment Analysis App 20. 20 Use Case: Anomaly Detection An anomaly (or outlier) is an event which is vastly dissimilar to other events. Anomaly Detection is one of Splunks most common use cases. Examples: Transactions which occur faster than humanly possible. DDoS attacks from IPaddress ranges. High-value customer purchase patterns. Quick techniques for finding statistical outliers: Non-average outliers: more than 2*stdev from the avg. Non-typical outliers: more than 1.5*IQR above perc75 or below perc25. Tip: save these as eventtypes for automated outlier detection. Once anomalies have been found, dig deeper to discover root causes. 21. 21 Splunk Solution: cluster Anomalies are dissimilar to other events (by definition). We can use clustering algorithms to help us detect anomalies: Non-anomalous events typically form a few large clusters. Anomalous events typically form lots of small clusters. Cluster your data, sort ascending: cluster showcount=true labelonly=true | sort cluster_count cluster_label! Remember: there is no right way to find all anomalies. Explore your data! 22. 22 Concept: Unsupervised Learning & Clustering Aclustering algorithm is any process which groups together similar things (events, people, etc), and separates dissimilar things (events, people, etc). Clustering is unsupervised: choose labels based on patterns in the data. Clustering is in the eye of the beholder: Lots of different clustering algorithms. Lots of different similarity functions. Do not confuse with: Computer cluster: a group of computers working together as a single system. Splunk cluster: a group of Splunk indexers replicating indexes & external data. 23. 23 Demo: cluster! 24. 24 Splunk Solution: Other Commands anomalies: Assigns an unexpectedness score to each event. anomalousvalue: Assigns an anomaly score to events with anomalous values. outlier: Removes or truncates outliers. kmeans: Powerful clustering algorithm. You choose k = # of clusters. 25. 25 Splunk Solution: Prelert (Partner App) ManagesAnomaly Detection directly. Pre-built dashboards, alerts,API. Use cases: Security, ITOps /APM, DevOps Godfrey Sullivan: "beautifully adjacent and complimentary to what Splunk does Can download from SplunkApps. May save you time withAnomaly Detection. Can also be good source of inspiration for your ownAnomaly Detection dashboards. Keep in mind Prelert is a paid app: Cost: $225/month @ 5GB 26. 26 Use Case: Market Segmentation Market Segmentation: group customers according to common needs and priorities, and develop strategies to target them. Market segments are internally homogeneous, and externally heterogeneous. i.e., market segments are clusters of customers. Many reasons for Market Segmentation: Different market segments require different strategies. Customers in same segment have similar product preferences. Different segments, different preferences. Segments should be reasonably stable, to allow for historical analysis (good for Data Science). Use Splunks clustering algorithms to identify and label market segments! 27. 27 DataVisualiza7ons 28. 28 Intro to Data Visualization Data Visualization is the creation and study of the visual representation of data, and is a vital part of Data Science. The goal of data visualization is to communicate information: Visualizations communicate complex ideas with clarity, precision, and efficiency. Transmission speed of the optic nerve is about 9Mb/sec fast image processing. Pattern matching, edge detection. Visualizations pack lots of information into small spaces. More than text alone! 29. 29 Telling Stories with Data Visualizations We process data in linear narratives: even dashboards go top-to-bottom. Visualizations help pierce the monotony of text, number & data streams. Think about the story youre telling: Empathize with the viewer. Whats their takeaway? Agood visualization tells its own story: Island Nation Obtains Favourable Balance ofTrade; Goes OnTo RuleThe World. Weave multiple visualizations together to tell more effective stories. William Playfair (1786) 30. 30 Source: New York Times. May 17, 2012 Splunk 31. 31 Source: New York Times. May 17, 2012 Splunk 32. 32 Tips for Effective Data Visualizations #1 tip: Plot the most important keys on x & y axes. You choose most important. You might need >1 visualization. Manipulate size, color and shape to convey additional information. Annotate, label and add icons Use chart overlay to correlate data sources. Mix histograms & line charts Manipulate numerical scale: linear vs. log scales (previous 2 slides). Read more about Data Visualization: Tableaus whitepaper, VisualAnalysis Best Practices (2013). EdwardTuftes The Visual Display of Quantitative Information (2001). 33. 33 Splunk now supports D3 visualizations with some minor customization. Satoshis talk: I want that cool viz in Splunk! Resources for Custom Visualizations: Splunk Web FrameworkToolkit https://apps.splunk.com/app/1613/ Splunk 6.x Dashboard Examples https://apps.splunk.com/app/1603/ Custom SimpleXMLExtensions http://apps.splunk.com/app/1772/ Lots more D3 visualizations for use: https://github.com/mbostock/d3/wiki/Gallery D3 Custom Visualizations in Splunk 34. 34 Demo: Sankey Chart 35. 35 How-to for Sankey Charts Install the Custom SimpleXMLExtensions app: http://apps.splunk.com/app/1772/ Create your own app, and install Sankey chart components: Drop autodiscover.js in $SPLUNK_HOME/etc/apps//appserver/static Copy & paste /sankeychart/ subfolder into $SPLUNK_HOME/etc/apps// appserver/static/components Restart Splunk. In your dashboard: Include script="autodiscover.js" in or opening tag Insert XMLsnippet from 2- or 3-node Sankey dashboard example Change 2 instances of custom_simplexml_extensions to . Update search and data-options parameters (nodes) in XMLto reflect your data. 36. 36 Know Your Audience Finally, keep in mind your audience: who are they, what questions do they care about, and how do they want to consume the data? Executive: KPIs, charts, tables with icons MarketingAnalyst: KPIs & metrics. Sharp images for their own reports & decks.Tableau. Data Scientist: output clean data to organized data stores (Hunk, HDFS, SQL, NoSQL). Sysadmin: sparklines, gauges for activity & MTTR, tables with highlighted anomalies. Security Ops: maps with detailed overlays, drill down on anomalous events. Bring it back to the business problem & use case! 37. 37 3 Key Takeaways Splunkisgreatfor doingDataScience! Splunkcomplements othertoolsinthe DataSciencetoolkit. DataScienceisabout extrac:ngac:onable insightsfromdata. 1 2 3 38. 38 List of References Good books on Data Science: Schutt & ONeil. Doing Data Science. OReilly 2013 Provost & Fawcett. Data Science for Business. OReilly 2013 Max Shron. Thinking With Data. OReilly 2014 Edward Tufte. The Visual Display of Quantitative Information. Graphics Press 2001 Zumel & Mount. Practical Data Science with R. Manning 2014 Hastie et al. Elements of Statistical Learning. Springer-Verlag 2009 (free PDF!) Using Splunk for Data Science: Zadrozny, Kodali (and Stout). Big Data Analytics Using Splunk. Apress 2013 David Carasso. Exploring Splunk. CITO Research 2012 David Carasso. Data Mining with Splunk. .conf2012 Michael Wilde & David Carasso. Social Media & Sentiment Analysis. .conf2012 Good free references: Tableau. Visual Analysis Best Practices. Tableau 2013 King & Magoulas. 2013 Data Science Salary Survey. OReilly 2013 DJ Patil. Building Data Science Teams. OReilly 2013 Cathy ONeil. On Being A Data Skeptic. OReilly 2013

Popular Tags:
of 38/38
1 Copyright © 2014 Splunk Inc. Tom LaGatta Data Scientist, Splunk Olivier de Garrigues Sr Prof Services Consultant, Splunk Splunk for Data Science
Embed Size (px)