Search Language - Intermediate

Karen Hodges, Sr. Instructor

© Copyright Splunk 20112The 2nd Annual Splunk Worldwide Users’ Conference

Karen Hodges – Senior Instructor – SplunkOver 20 years of experience in software training and education in:- UNIX System Administration- Intergraph GIS Systems- Relational Database Management Systems- BMC Remedy- Mortgage Fraud Detection- Real Property Title Search- Splunk

Your presenter . . .

© Copyright Splunk 20113The 2nd Annual Splunk Worldwide Users’ Conference

Knowledge Objects- Tags- Event types- Saved searches and alerts

Advanced searching techniques- Comparison operators- The search pipeline

Topics

© Copyright Splunk 20114The 2nd Annual Splunk Worldwide Users’ Conference4

Knowledge Objects

© Copyright Splunk 20115The 2nd Annual Splunk Worldwide Users’ Conference

Type in keywords, hit return, get results . . .

Splunk as “Search Engine”

© Copyright Splunk 20116The 2nd Annual Splunk Worldwide Users’ Conference

Splunk allows you to “store” knowledge alongside your IT data

Institutional knowledge - For example: server function or device location

Learned knowledge- For example: identify crash precursors or suspicious activity patterns

You store these in Splunk using Knowledge Objects

So Much More than a “Search Engine”

© Copyright Splunk 20117The 2nd Annual Splunk Worldwide Users’ Conference

Server names aren’t always very helpful!Sometimes they pack too much information into the name

Sometimes they make them reflect their hobbies/obsessions

Scenario – Confusing Server Names

© Copyright Splunk 20118The 2nd Annual Splunk Worldwide Users’ Conference

Tags are metadata you can add to field values

Knowledge Objects – Tags to the Rescue

© Copyright Splunk 20119The 2nd Annual Splunk Worldwide Users’ Conference

Search all hosts tagged as “webfarm”

Using Tags

© Copyright Splunk 201110The 2nd Annual Splunk Worldwide Users’ Conference

IT data is full of strange and confusing messageSome are alarming!

Some are low key, but should be alarming

Scenario – So Many Different Needles and Hays

© Copyright Splunk 201111The 2nd Annual Splunk Worldwide Users’ Conference

Event types are fields based on a search – similar to a saved search

Knowledge Objects – Event Types

© Copyright Splunk 201112The 2nd Annual Splunk Worldwide Users’ Conference

For example: 2 events in linux_secure

Save event types to differentiate these 2 events- pwd_fail_known and pwd_fail_unknown

Event Type Example - Different Events

© Copyright Splunk 201113The 2nd Annual Splunk Worldwide Users’ Conference

For example: 2 different types of firewalls- CheckPoint firewall “action=reject”- Netscreen firewall “action=deny”

Event Type Example – Same Event

© Copyright Splunk 201114The 2nd Annual Splunk Worldwide Users’ Conference14

Using Event Types•Use the eventtype as you would any other field

© Copyright Splunk 201115The 2nd Annual Splunk Worldwide Users’ Conference

Servers and devices run 24/7Hackers, bugs and crashes (oh my!) are lurking 24/7Humans aren’t 24/7 – they need things like sleep, vacations, lunch, or just a few minutes away from staring at a screen in a freezing cold server room!

Scenario – 24/7 Monitoring

© Copyright Splunk 201116The 2nd Annual Splunk Worldwide Users’ Conference

Searches can be run on a schedule and be setup to “do something” based on the results

We call these Alerts

Splunk Alerts Never Sleep!

© Copyright Splunk 201117The 2nd Annual Splunk Worldwide Users’ Conference

Hackers need a user name AND a password to log in to your systems

Public web pages often contain names of CEOs, sales folks, etc. splunk.com is no exception

Alerting Scenario – Public User Logins

© Copyright Splunk 201118The 2nd Annual Splunk Worldwide Users’ Conference

Since only certain users appear on the web page, we can give those users the tag=publicID

We can use the “pwd_fail_known” Event Type we created earlier

Leverage Tagging and Event Types

© Copyright Splunk 201119The 2nd Annual Splunk Worldwide Users’ Conference

Craft the search that searches for login attempts from public users then create the alert

Click next to define alert conditions

Craft Your Search and Create the Alert

© Copyright Splunk 201120The 2nd Annual Splunk Worldwide Users’ Conference

You can specify alert conditions which will trigger the alert

In our case we are looking for four or more login attempts since after that legitimate users are locked out

Alert Conditions

© Copyright Splunk 201121The 2nd Annual Splunk Worldwide Users’ Conference

Can send email, create RSS feed, or trigger shell script

We have opted to have the results included in our email so we can evaluate the severity of the attack easily

Tracking allows us to view fired alerts in the Alert manager

Alert Actions

© Copyright Splunk 201122The 2nd Annual Splunk Worldwide Users’ Conference

Use the Alerts menu item in the main Splunk navigation to display the Alerts manager window.

Click Results to view the events that triggeredthe alert

Click Edit to edit the alert settings

Alert Manager

Failed Logins

Failed Logins

© Copyright Splunk 201123The 2nd Annual Splunk Worldwide Users’ Conference23

Advanced Searching Techniques

© Copyright Splunk 201124The 2nd Annual Splunk Worldwide Users’ Conference

Comparison operators make your searches more exacting

Splunk’s full-featured search language permits you to organize and analyze data in amazing ways!

So Much More than a “Search Engine”- Part II

© Copyright Splunk 201125The 2nd Annual Splunk Worldwide Users’ Conference

Comparison operators!= > < <= >=

Towards More Sophisticated Searches

© Copyright Splunk 201126The 2nd Annual Splunk Worldwide Users’ Conference

Search is a data generating commandYou can organize and analyze data using the search pipeline

The Search Pipeline

sourcetype=syslog ERROR | top user | fields - percent

Fetch events from disk that

match

Summarize into table of top 10 users

Remove column showing

percentage

Intermediateresults table

Intermediate results table

Final results table

DiskDisk

© Copyright Splunk 201127The 2nd Annual Splunk Worldwide Users’ Conference

After the search command use the “|” symbol to pipe your search results to a subsequent command

For example here we are changing the sort order to sort by user name descending – grouping all the logins together

Organize and Analyze Your Data

© Copyright Splunk 201128The 2nd Annual Splunk Worldwide Users’ Conference

We’ve already seen sort, there are many MANY more . . .-dedup removes duplicates

‣ Weeding out duplicate entries makes results easier to use AND keeps statistical operations more pure

-regex allows you filter your results using a regular expression‣ REGEX gurus can filter using all the ?’s and *’s they want!

-transaction allows you to group your events by a certain field and time range‣ See all the web pages your boss visited in the past hour from your proxy

data

Data Processing Commands

© Copyright Splunk 201129The 2nd Annual Splunk Worldwide Users’ Conference

When you type in a command after the | symbol Splunk’s Search Assistant provides an instant mini “man page”

Splunk Makes Using its Search Language Easy

© Copyright Splunk 201130The 2nd Annual Splunk Worldwide Users’ Conference

The table command is useful for visually organizing events

Columns are displayed in the same order of fields entered in the command- Column headers are field names- Rows are field values- Each row represents an event

View Events in a Table

30

© Copyright Splunk 201131The 2nd Annual Splunk Worldwide Users’ Conference

The top command finds the most common values of a given field- Returns top 10 results by default

Automatically returns a count and percentage

Adding limit=# after the top command returns the specified number of results

Top Scenario – Getting Top Site Visitors

31

© Copyright Splunk 201132The 2nd Annual Splunk Worldwide Users’ Conference

count returns the number of occurrences of a given field

The by clause returns a count for each field value of a named field

Stats Scenario – Counting Product Sales

32

© Copyright Splunk 201133The 2nd Annual Splunk Worldwide Users’ Conference

Online trading activity is captured in a log file which includes each trader’s unique identification

Company policy requires that we monitor each trader’s activity in hourly chunks, but the trades are all jumbled up together making it hard to spot patterns in each trader’s trades

Transaction Scenario – Monitor Trading Activity

© Copyright Splunk 201134The 2nd Annual Splunk Worldwide Users’ Conference

Use transaction to group each trade by TradeID

Set your time span to an hour and your max pause to one hour in case some traders only have one or two trades per hour

Use Transaction to Group Your Trades

© Copyright Splunk 201135The 2nd Annual Splunk Worldwide Users’ Conference

Event types and tags are excellent ways to capture existent knowledge as well as knowledge learned from using Splunk

Splunk’s search language includes many powerful commands which allow you to organize and analyze your data easily

Summary

© Copyright Splunk 201136The 2nd Annual Splunk Worldwide Users’ Conference

You’ve just seen some of the many ways Splunk can be used to leverage the intelligence in your IT data

Further your Splunk education with official Splunk trainingUsing Splunk – Gets deeper into basic search, alerts, knowledge objects, quick reports and more…

Searching and Reporting with Splunk – Takes you to the next level leveraging statistical operations and reporting in Splunk

Congratulations!

August 15, 2011

Questions?

Karen Hodges, Sr. Instructor