Setting up High Speed Logging (HSL) & Configuring F5 to work with Splunk

David Perodin - FSE

Agenda

Explain the necessary components for F5’s new Logging framework

Pools, Destinations, Publishers, & etc.

Demonstrate F5 and Splunk integration

Questions

BIG-IP Logging

Prior to 11.3,

Logging done by different systems via different mechanisms

Configuration was totally independent of each other.

V10.1 introduced HSL support by iRules

V11.0 the HTTP Request Logging profile was introduced

V11.3

Logging systems are inter-connected

The linux host processes can now log to remote servers

Logging Overview

System

AFM

High Speed DNS

Publisher Formatted

Destination HSL Dest.

Pool

How is this Better?

Remote Logging available since 11.1

Available before 11.1 via the retired bigpipe CLI

No customization

• Every message sent to every entry in list of remote loggers

11.3 Filters allow separate treatment of individual daemons

11.3 Publishers allow separate treatment of different loggers

Not everyone in an organization is interested in the same logs

• System Logs to Operations

• Firewall Logs to Security Team

• Audit Logs to ???

© F5 Networks, Inc 6

More versatile logging

System

AFM

High Speed DNS

Publisher Formatted

Destination (Splunk)

HSL Dest.

Pool (Splunk)

Formatted Destination (ArcSight)

Formatted Destination

(Syslog)

HSL Dest.

HSL Dest.

Pool (Syslog)

Pool (ArcSight)

What's Left to Do?

Alerting

SNMP Traps

Email

Overview of Common Elements

Pool

A collection of log servers defined by IP address and port

Destination

A Destination is a Pool of log servers

May provide formatting

Publisher

A Publisher is a collection of Destinations

Remote Logging Steps: Pool Creation

1. Create a Pool

2. Create a Destination

3. Create a Formatted Destination

4. Create a Publisher

5. Create tmm_filters

© F5 Networks, Inc 10

Pool Creation - GUI

Remote Logging Steps: Destination

1. Create a Pool

2. Create a Destination

• Create a High Speed Log (HSL) Destination

3. Create a Formatted Destination

4. Create a Publisher

5. Logging Application Steps (varies by Application)

Destination

Destination Creation

A Destination is a Pool of log servers along with a Type

Configuration Elements

• Enter a unique Name

• Select a Type (see next slides)

• Remote High-Speed Log, ArcSight, Splunk or Remote Syslog

Destination Type

Unformatted

• Remote High-Speed Log (aka HSL Destination)

• Select a pool

• Formatted

• Splunk

• Requires an HSL Destination to forward too.

• ArcSight

• Requires an HSL Destination to forward too.

• Syslog

• Select a Syslog format

• And an HSL Destination

Destination Creation

Go to System > Logs > Configuration > Log Destinations

High-Speed Log Destination Creation

Unformatted

Must be create before formatted destinations

•

Formatted Destinations

1. Create a Pool

2. Create a Destination

3. Create a Formatted Destination

• Tied to an HSL Destination

4. Create a Publisher

5. Logging Application Steps (varies by Application)

Remote Syslog Destination Creation

Name your log destination

Select a syslog format

Select a High-Speed Log Destination

• Unformatted Destination you created earlier

Splunk Destination Creation

Similar to create a Remote Syslog destination

Select the Splunk format

Select a High-Speed Log Destination

• Unformatted Destination you created earlier

Remote Logging Steps: Publisher

1. Create a Pool

2. Create a Destination

3. Create a Formatted Destination

4. Create a Publisher

• Using one or more Destinations

5. Create tmm_filters

Log Publisher

A Publisher is a collection of Destinations

Configuration Elements:

Choose a unique name for this Publisher

(Optionally) Enter a Description

Select a Destination from the available choices

Support Details - Uneven Load Balancing

Load balancing across Pools of remote logging servers

BIG-IP follows the connection/session

BIG-IP doe not load balance by message

At low volumes of logging uneven log message counts will be seen.

• For example in testing or performing a POC.

HSL will not make a load balancing decision

Until it runs out of bandwidth to the selected pool member.

Or there is a change in server response

Publisher local-db-publisher

Used by the legacy logging system

Local logging places an I/O load on the BIG-IP

Should not be used, can have a significant impact

Previous Remote Logging Option

This screen introduced in V11.1

Does not load balance

All Syslog servers in the list receive a copy of the message

11.3 System Logging - A New Paradigm

Required: elements described previously

Pool

Destination

Publisher

What is unique is the tmm_filter

tmm_filter

Under System > Logs > Configuration > Log Filers

Can create custom filters

Name

Description (optional)

Severity

• Default is Debug

Source

• List of processes

• Defaults to all

Message ID

Log Publisher

Severity

Filter base on severity

Name (required)

Description (optional)

Severity

Source

Filter base on process

Source

• Select from the list of processes

• Defaults to all

11.3 System Logging

Filter base on Message ID

Message ID

Log Publisher

• Message destination(s)

© F5 Networks, Inc 28

Interaction of Legacy Paradigm and Filters

Log Messages

Filter match

No Filter

Publisher

Syslog (legacy)

© F5 Networks, Inc 29

All Logging Done Off the BIG-IP

Log Messages

Filter match

Publisher

Publisher (none)

Filter match

all debug

Syslog (legacy)

Nothing

unmatched

DANGEROUS DEFAULTS

Beware the default severity 'debug' and default source 'all'

Thank You! Thank You!