Copyright © 2011, Splunk Inc. Listen to your data.2
Agenda
Getting Started with splunk>
Deployment and Integration(time permitting)
Copyright © 2011, Splunk Inc. Listen to your data.
Install splunk>
Start Splunk• WIN: \Program Files\Splunk\bin\splunk.exe start• Other: /opt/splunk/bin/splunk start
www.splunk.com/download• 32 or 64 bit?• Indexer or Universal Forwarder?
Splunk Home•WIN: \Program Files\Splunk•Other: /opt/splunk (Applications/splunk)
Copyright © 2011, Splunk Inc. Listen to your data.6
splunk> Web Basics
Browser Support• Firefox 2, 3.x, 4.0 (3.5 with splunk> version 4.0.6 and higher)• Internet Explorer 6, 7, 8 and 9• Safari 3, 4, 5 (and up)• Chrome 9 (and up)• be sure to install Flash (versions prior to 4.3.x)
Default on install is http://localhost:8000
Copyright © 2011, Splunk Inc. Listen to your data.
Take A splunk> Tutorial
splunk> Tutorials
• splunk> Home >> Getting Started App
• splunk.com >> Documentation >> User Manual >> Tutorial
Copyright © 2011, Splunk Inc. Listen to your data.8
key : value
Universal Indexing Indexes Unstructured Data
It’s that simple.
Copyright © 2011, Splunk Inc. Listen to your data.9
Every wordEvery valueEvery character … even punctuation … even white spaces
Universal Indexing Indexes Unstructured Data
Copyright © 2011, Splunk Inc. Listen to your data.11
SearchingSearch > *Select Time Range• Historical, custom, or real-timeUsing the timeline• Click events and zoom in and out• Click and drag over events for a specific range
Copyright © 2011, Splunk Inc. Listen to your data.12
Searching (continued)Search for any keyword
Search > error
Use Boolean expressions
Search > error OR failed NOT application
Spaces are implied AND, Operators need to be in caps
Search > Audit Failure = Audit AND Failure
Use quotes to search for a specific string
Search > “Audit Failure”
Use wildcards
Search > 46* OR 49*
Copyright © 2011, Splunk Inc. Listen to your data.13
Searching (continued)
Search > error | head 1Search results are “piped” to the commandCommands for:• Manipulating fields• Formatting• Handling results• ReportingSave a search
Copyright © 2011, Splunk Inc. Listen to your data.14
Search AssistantContextual Help
• advanced typeahead
History• search• commands
Search Reference• short/long description• examples
Copyright © 2011, Splunk Inc. Listen to your data.15
Tagging and Event TypingEventtypes for more human-readable reports
• categorize and make sense of mountains of data• punctuation helps find events with similar patternsSearch > eventtype=failed_login instead ofSearch > “failed login” OR “FAILED LOGIN” OR “Authentication failure” OR “Failed to authenticate user”
Tags are labels• apply ad hoc knowledge• create logical divisions or groups• tag hosts, sources, fields, even eventtypesSearch > tag=web_servers instead ofSearch > host=“apache1.splunk.com” OR host=“apache2.splunk.com” OR host=“apache3.splunk.com”
Copyright © 2011, Splunk Inc. Listen to your data.16
Over 100 Commands!
splunk.com > Documentation > Search Reference
abstract accum addcoltotals addinfo addtotals af analyzefields anomalies anomalousvalue append appendcols ar associate audit autoregress bin bucket chart cluster collect common contingency convert correlate counttable crawl ctable dbinspect dedup delete
delta diff discretize erex eval eventcount eventstats excerpt extract file fillnull folderize format gentimes head highlight iconify
input inputcsv inputlookup iplocation join kmeans kv kvform loadjob localize localop lookup macro makecontinuous makemv maketable map metadata multikv mvcombine mvexpand nomv
outlier outlierfilter outputcsv outputlookup outputtext overlap rangemap rare regex relevancy rename replace reverse run savedsearch savedsplunk script scrub selfjoin sendemail set sichart sirare sistats sitimechart sitop slc stash strcat streamstats sumindex summaryindex tail test timechart top
transaction transam trendline typeahead typelearner typer uniq untable xmlkv xmlunescape xpath xyserieshttp://www.splunk.com/base/Documentation/latest/SearchReference/SearchCheatsheet
Copyright © 2011, Splunk Inc. Listen to your data.17
FieldsDefault fields• host, source, sourcetype, _time, _raw, etc.• View on left panel in search results or all in field picker
Where do fields come from?• Pre-defined by sourcetypes• Automatically extracted key-value pairs• User-defined
Copyright © 2011, Splunk Inc. Listen to your data.18
Extract FieldsInteractive Field Extractor
• generate PCRE (perl compatible regular expression)
• editable regex• preview/save
props.conf
[mysourcetype]REPORT-myclass = myFields
transforms.conf
[myFields]REGEX = ^(\w+)\sFORMAT = myFieldLabel::$1
Configuration File•manual field extraction
•delim-based extractions
Rex Search Command... | rex field=_raw "From: (?<from>.*) To: (?<to>.*)"
Copyright © 2011, Splunk Inc. Listen to your data.19
Saved Searches and AlertingFind Something Interesting?
OR
Copyright © 2011, Splunk Inc. Listen to your data.20
Alerting (continued)
Searches run on a schedule and fire an alert • Example: Run a search for “Failed password” every 15 min
over the last 15 min and alert if the number of events is greater than 10
Searches are running in real-time and fire an alert• Example: Run a search for “Failed password user=john.doe”
in a 1 minute window and alert if an event is found
Copyright © 2011, Splunk Inc. Listen to your data.21
Alerting Actions• Send email• RSS• Execute a script• Track in Alert Manager
Copyright © 2011, Splunk Inc. Listen to your data.22
ReportingBuild reports from the results of any search
Select type of report (Values over time, Top Values, Rare Values) and on which fields to report or perform statistics Choose the type of chart (line, area, column, etc) and
other formatting options
Copyright © 2011, Splunk Inc. Listen to your data.23
Reporting Examples
• Use wizard or reporting commands (timechart, top, etc)• Build real-time reports with real-time searches• Save reports for use on dashboards
Copyright © 2011, Splunk Inc. Listen to your data.24
DashboardsCreate dashboards from search results
Copyright © 2011, Splunk Inc. Listen to your data.26
splunk> Appssplunk> Apps
• Apps create different contexts for your data out of sets of views, dashboards, and configurations
• Splunk Home -> Getting Started App• Splunk Home -> Find more apps• Install an App• You can create your own!
Search is an App!• Summary will show everything you have indexed • Updated in real-time• Click on any source, sourcetype, or host to look at events
Copyright © 2011, Splunk Inc. Listen to your data.
splunk> Supports Diverse Apps/Solutions
Security
IronPort WSA
CDR
Copyright © 2011, Splunk Inc. Listen to your data.
Searches can be managed as asynchronous processes
Jobs can be • Scheduled• Moved to background tasks• Paused, stopped, resumed, finalized• Managed• Archived
Job Management
Copyright © 2011, Splunk Inc. Listen to your data.29
splunk> ManagerNow Manage All of that Cool Stuff You Just Created (and more!)
• Permissions• Saved Searches/Reports• Custom Views• Distributed Splunk• Deployment Server• License Usage….
Copyright © 2011, Splunk Inc. Listen to your data.
splunk> LicensesFree Download Limits Indexing to 500MB/day
• Enterprise Features expire after 60 days• Reverts to Basic Free License
Features Disabled in Free License• Multiple user accounts and role-based access controls• Distributed search• Forwarding to non-Splunk Instances• Deployment management• Scheduled saved searches and alerting• Summary indexing
Add a License
Copyright © 2011, Splunk Inc. Listen to your data.31
Support Through the splunk> Community
Browse and share Apps from Splunk, Partners and the
Community
splunkbase.splunk.com
Splunkbase
Community-driven knowledge
exchange and Q&A
answers.splunk.com
5 tracks, more than 40 sessions, the smartest Splunk users together
www.splunk.com/goto/conference
Copyright © 2011, Splunk Inc. Listen to your data.32
Where to Go for HelpDocumentation– http://www.splunk.com/base/Documentation
Technical Support – http://www.splunk.com/support
Videos– http://www.splunk.com/videos
Education– http://www.splunk.com/goto/education
Professional Services
Copyright © 2011, Splunk Inc. Listen to your data.
The IT Search Company
Your Guide David Lutz
email: [email protected]: @dlux_at_splunk
skype: dluxatsplunkdotcom
Copyright © 2011, Splunk Inc. Listen to your data.
A splunk> Installation Has 4 Functions Searching and Reporting (Search Head)
Indexing and Search Services (Indexer)
Data Collection and Forwarding (Forwarder)
Local and Distributed Management (Deployment Server)
35
A splunk> installation can be one or all of these …
Copyright © 2011, Splunk Inc. Listen to your data.36
Getting Data Into splunk>Agent and Agent-less Approach for Flexibility
perf
shellcode
Mounted File Systems\\hostname\mount
syslogTCP/UDP
WMIEvent Logs Performance
Active Directory
syslog compatible hostsand network devices
Unix, Linux and Windows hosts
Windows hosts Custom apps and scripted API connections
Local File Monitoringlog files, config files
dumps and trace files
Windows InputsEvent Logs
performance countersregistry monitoring
Active Directory monitoring
virtualhost
Windows hosts
Scripted Inputsshell scripts custom
parsers batch loading
Agent-less Data Input Splunk Forwarder
Copyright © 2011, Splunk Inc. Listen to your data.37
Understanding the Universal ForwarderForward data without negatively impacting production performance.
Scripts
Universal Forwarder Deployment
Logs ConfigurationsMessages Metrics
Central Deployment Management
Monitor files, changes and the system registry; capture metrics and status.
Universal Forwarder
Regular (Heavy) Forwarder
Monitor All Supported Inputs
✔ ✔
Routing, Filtering, Cloning
✔ ✔
Splunk Web ✔
Python Libraries ✔
Event Based Routing ✔
Scripted Inputs ✔
Copyright © 2011, Splunk Inc. Listen to your data.38
Horizontal ScalingLoad balanced search and indexing for massive, linear scale out.
Forwarder Auto Load Balancing
Distributed Search
Copyright © 2011, Splunk Inc. Listen to your data.39
Multiple Datacenters
Headquarters
London Hong Kong Tokyo New York
Distributed Search
Index and store locally. Distribute searches to datacenters, networks & geographies.
Copyright © 2011, Splunk Inc. Listen to your data.
Data Redundancy
40
Clone data to multiple index servers to eliminate a single point of failure.
Data CloningForwarding to DR site
Active Hot Standby
---------
Copyright © 2011, Splunk Inc. Listen to your data.
High Availability
41
Combine auto load balancing and cloning for HA at every Splunk tier.
Clone Group 1 : Complete Dataset
Data Cloning & Auto Load Balancing
Distributed Search Distributed Search
Clone Group 2 : Complete Dataset
Shared Storage
Copyright © 2011, Splunk Inc. Listen to your data.42
Service Desk
Event Console
SIEM
Send Data to Other SystemsRoute raw data in real time or send alerts based on searches.
Copyright © 2011, Splunk Inc. Listen to your data.43
Integrate External Data
LDAP, AD Watch Lists
CRM/ERPCMDB
Correlate IP addresses with locations, accounts with regions
Extend search with lookups to external data sources.
Copyright © 2011, Splunk Inc. Listen to your data.44
Integrate Users and Roles
Problem Investigation Problem Investigation Problem Investigation
Save Searches
Share Searches
LDAP, AD Users and Groups
Splunk Flexible Roles
Manage Users
Manage Indexes
Capabilities & Filters
NOT tag=PCI
App=ERP…
Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.
Integrate authentication with LDAP and Active Directory.
Copyright © 2011, Splunk Inc. Listen to your data.45
Centralized Licensing Management
Problem Investigation
Groups, Stacks, and Pools for Enterprise Deployments
Copyright © 2011, Splunk Inc. Listen to your data.
Deployment Monitoring
46
Keep Tabs On Your Splunk Enterprise Deployment
ForwardersIndexersSourcetypesLicenses
Copyright © 2011, Splunk Inc. Listen to your data.
Support Through the Splunk Community
48
Browse and share Apps from Splunk, Partners and the
Community
splunkbase.splunk.com
Splunkbase
Community-driven knowledge
exchange and Q&A
answers.splunk.com
5 tracks, more than 40 sessions, the smartest Splunk users together
www.splunk.com/goto/conference
Copyright © 2011, Splunk Inc. Listen to your data.
Where to Go for HelpDocumentation– http://www.splunk.com/base/Documentation
Technical Support – http://www.splunk.com/support
Videos– http://www.splunk.com/videos
Education– http://www.splunk.com/goto/education
Professional Services
49
Copyright © 2011, Splunk Inc. Listen to your data.
The IT Search Company
Your Guide David Lutz
email: [email protected]: @dlux_at_splunk
skype: dluxatsplunkdotcom