The syslog-ng Premium Edition 6 LTSAdministrator Guide

Publication date June 19, 2018

AbstractThis manual is the primary documentation of the syslog-ng Premium Edition 6 LTS

product.

Copyright © 1996-2018 One Identity LLC

Copyright © 2018 One Identity LLC. All rights reserved. This document is protected by copyright and is distributed under licenses restricting its use,copying, distribution, and decompilation. No part of this document may be reproduced in any form by any means without prior written authorization ofOne Identity.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

AIX™, AIX 5L™, AS/400™, BladeCenter™, eServer™, IBM™, the IBM™ logo, IBM System i™, IBM System i5™, IBM System x™, iSeries™,i5/OS™, Netfinity™, NetServer™, OpenPower™, OS/400™, PartnerWorld™, POWER™, ServerGuide™, ServerProven™, and xSeries™ are trademarksor registered trademarks of International Business Machines.

Alliance Log Agent for System i™ is a registered trademark of Patrick Townsend & Associates, Inc.

The Balabit™ name and the Balabit™ logo are registered trademarks of Balabit SA.

Debian™ is a registered trademark of Software in the Public Interest Inc.

Hadoop™ and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

Linux™ is a registered trademark of Linus Torvalds.

MapR™, is a trademark of MapR Technologies, Inc.

Elasticsearch™ and Kibana™ is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Apache Kafka and the Apache Kafka Logo are trademarks of the Apache Software Foundation.

MySQL™ is a registered trademark of Oracle and/or its affiliates.

Oracle™, JD Edwards™, PeopleSoft™, and Siebel™ are registered trademarks of Oracle Corporation and/or its affiliates.

Red Hat™, Inc., Red Hat™Enterprise Linux™ and Red Hat™ Linux™ are trademarks of Red Hat, Inc.

SUSE™ is a trademark of SUSE AG, a Novell business.

Solaris™ is a registered trademark of Oracle and/or its affiliates.

Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of SplunkInc. in the United States and other countries.

The syslog-ng™ name and the syslog-ng™ logo are registered trademarks of Balabit.

Windows™ 95, 98, ME, 2000, XP, Server 2003, Vista, Server 2008, 7, 8, and Server 2012 are registered trademarks of Microsoft Corporation.

For details on FIPS-compliance, see this page.

All other product names mentioned herein are the trademarks of their respective owners.

DISCLAIMER. One Identity is not responsible for any third-party websites mentioned in this document. One Identity does not endorse and is notresponsible or liable for any content, advertising, products, or other material on or available from such sites or resources. One Identity will not beresponsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, orservices that are available on or through any such sites or resources.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (https://www.openssl.org/). This product includescryptographic software written by Eric Young (eay@cryptsoft.com)

This product includes open source software components. For details on the licenses and availability of these software components, see Appendix C,Open source licenses (p. 529).

iisyslog-ng.com

Table of ContentsPreface ............................................................................................................................................... x

1. Summary of contents ................................................................................................................ x2. Target audience and prerequisites ............................................................................................. xi3. Products covered in this guide ................................................................................................. xii4. Typographical conventions ..................................................................................................... xii5. Contact and support information ............................................................................................. xii

5.1. Sales contact .............................................................................................................. xiii5.2. Support contact .......................................................................................................... xiii5.3. Training ..................................................................................................................... xiii

6. About this document ............................................................................................................. xiii6.1. Summary of changes .................................................................................................. xiii6.2. Feedback ................................................................................................................... xvi6.3. Acknowledgments ...................................................................................................... xvi

1. Introduction to syslog-ng ................................................................................................................ 11.1. What syslog-ng is .................................................................................................................. 11.2. What syslog-ng is not ............................................................................................................ 31.3. Why is syslog-ng needed? ...................................................................................................... 31.4. What is new in syslog-ng Premium Edition 6 LTS? ................................................................. 31.5. Who uses syslog-ng? ............................................................................................................. 3

1.5.1. Public references of syslog-ng Premium Edition ........................................................... 31.6. Supported platforms .............................................................................................................. 4

1.6.1. Limitations on Microsoft Windows platforms ............................................................... 51.6.2. Certified packages ...................................................................................................... 6

2. The concepts of syslog-ng ............................................................................................................... 82.1. The philosophy of syslog-ng .................................................................................................. 82.2. Logging with syslog-ng ......................................................................................................... 8

2.2.1. The route of a log message in syslog-ng ....................................................................... 82.3. Modes of operation .............................................................................................................. 10

2.3.1. Client mode .............................................................................................................. 102.3.2. Relay mode .............................................................................................................. 112.3.3. Server mode ............................................................................................................. 11

2.4. Global objects ..................................................................................................................... 122.5. Timezones and daylight saving ............................................................................................. 13

2.5.1. How syslog-ng PE assigns timezone to the message .................................................... 132.5.2. A note on timezones and timestamps .......................................................................... 14

2.6. Versions and releases of syslog-ng PE ................................................................................... 142.7. Licensing ............................................................................................................................ 15

2.7.1. Licensing benefits ..................................................................................................... 152.7.2. Licensing model and modes of operation .................................................................... 152.7.3. Licensing examples .................................................................................................. 17

2.8. GPL and LGPL licenses ....................................................................................................... 182.9. High availability support ...................................................................................................... 182.10. The structure of a log message ............................................................................................ 18

2.10.1. BSD-syslog or legacy-syslog messages .................................................................... 192.10.2. IETF-syslog messages ............................................................................................. 21

iiisyslog-ng.com

2.11. Message representation in syslog-ng PE .............................................................................. 242.12. Structuring macros, metadata, and other value-pairs ............................................................. 25

2.12.1. Specifying data types in value-pairs ......................................................................... 262.13. Things to consider when forwarding messages between syslog-ng PE hosts ........................... 312.14. NFS file system for log files ............................................................................................... 34

3. Installing syslog-ng ....................................................................................................................... 363.1. Prerequisites to installing syslog-ng PE ................................................................................. 363.2. Security-enhanced Linux: grsecurity, SELinux ...................................................................... 373.3. Installing syslog-ng using the .run installer ............................................................................ 37

3.3.1. Installing syslog-ng PE in client or relay mode ........................................................... 383.3.2. Installing syslog-ng PE in server mode ....................................................................... 413.3.3. Installing syslog-ng PE without user-interaction ......................................................... 45

3.4. Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE, AIX) ................................ 453.5. Using syslog-ng PE on SELinux ........................................................................................... 473.6. Installing syslog-ng on Debian-based platforms ..................................................................... 483.7. Installing syslog-ng PE using .pkg installer ........................................................................... 49

3.7.1. Installing syslog-ng PE with user-interaction .............................................................. 493.7.2. Installing syslog-ng PE without user-interaction ......................................................... 503.7.3. Installing syslog-ng PE from a transformed PKG package ........................................... 51

3.8. Installing syslog-ng PE on Windows platforms ...................................................................... 513.9. Installing syslog-ng without user-interaction on Windows ...................................................... 573.10. Managing syslog-ng PE from Puppet .................................................................................. 583.11. Upgrading syslog-ng PE ..................................................................................................... 59

3.11.1. Upgrading from previous syslog-ng PE versions to 6 LTS .......................................... 593.11.2. Upgrading to syslog-ng PE 6 LTS ............................................................................ 593.11.3. Upgrading syslog-ng PE to other package versions .................................................... 603.11.4. Upgrading from syslog-ng PE to syslog-ng OSE ....................................................... 613.11.5. Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE .......... 61

3.12. Uninstalling syslog-ng PE .................................................................................................. 613.13. Configuring Microsoft SQL Server to accept logs from syslog-ng ......................................... 62

4. The syslog-ng PE quick-start guide .............................................................................................. 684.1. Configuring syslog-ng on client hosts ................................................................................... 684.2. Configuring syslog-ng on server hosts .................................................................................. 704.3. Configuring syslog-ng relays ................................................................................................ 72

4.3.1. Configuring syslog-ng on relay hosts ......................................................................... 724.3.2. How relaying log messages works ............................................................................. 73

5. The syslog-ng PE configuration file .............................................................................................. 755.1. Location of the syslog-ng configuration file .......................................................................... 755.2. The configuration syntax in detail ......................................................................................... 755.3. Notes about the configuration syntax .................................................................................... 775.4. Global and environmental variables ...................................................................................... 785.5. Logging configuration changes ............................................................................................. 795.6. Modules in syslog-ng PE ..................................................................................................... 79

5.6.1. Loading modules ...................................................................................................... 805.7. Managing complex syslog-ng configurations ......................................................................... 80

5.7.1. Including configuration files ...................................................................................... 805.7.2. Reusing configuration blocks .................................................................................... 82

6. Collecting log messages — sources and source drivers .................................................................. 85

ivsyslog-ng.com

6.1. How sources work ............................................................................................................... 856.2. Collecting messages from Windows eventlog sources ............................................................ 87

6.2.1. eventlog() source options .......................................................................................... 886.2.2. Limitations of using the EVT API on Windows Vista or newer .................................... 89

6.3. Collecting internal messages ................................................................................................ 906.3.1. internal() source options ............................................................................................ 91

6.4. Collecting messages from text files ....................................................................................... 926.4.1. Notes on reading kernel messages .............................................................................. 936.4.2. File sources and the RFC5424 message format ........................................................... 946.4.3. file() source options .................................................................................................. 94

6.5. Collecting messages using the RFC3164 protocol (network() driver) ..................................... 1016.5.1. network() source options ......................................................................................... 103

6.6. Collecting messages from named pipes ............................................................................... 1136.6.1. pipe() source options ............................................................................................... 114

6.7. Receiving messages from external applications .................................................................... 1186.7.1. program() source options ......................................................................................... 119

6.8. Collecting messages from tables or relational database ......................................................... 1236.8.1. Supported SQL sources by platform ......................................................................... 1246.8.2. sql() source options ................................................................................................. 1256.8.3. Customizing SQL queries ........................................................................................ 134

6.9. Collecting messages on Sun Solaris .................................................................................... 1356.9.1. sun-streams() source options .................................................................................... 136

6.10. Collecting messages using the IETF syslog protocol (syslog() driver) .................................. 1406.10.1. syslog() source options .......................................................................................... 141

6.11. Collecting the system-specific log messages of a platform .................................................. 1516.12. Collecting messages from the systemd-journal system log storage ....................................... 152

6.12.1. systemd-journal() source options ............................................................................ 1546.13. Collecting systemd messages using a socket ...................................................................... 1566.14. Collecting messages from remote hosts using the BSD syslog protocol ............................... 156

6.14.1. tcp(), tcp6(), udp() and udp6() source options — OBSOLETE ................................. 1576.15. Collecting messages from UNIX domain sockets ............................................................... 158

6.15.1. unix-stream() and unix-dgram() source options ....................................................... 1597. Sending and storing log messages — destinations and destination drivers .................................. 167

7.1. Sending messages directly to Elasticsearch version 1.x ........................................................ 1687.1.1. Prerequisites ........................................................................................................... 1707.1.2. How syslog-ng PE interacts with Elasticsearch ......................................................... 1717.1.3. Client modes .......................................................................................................... 1717.1.4. Elasticsearch destination options .............................................................................. 172

7.2. Sending messages directly to Elasticsearch version 2.0 or higher .......................................... 1797.2.1. Prerequisites ........................................................................................................... 1817.2.2. How syslog-ng PE interacts with Elasticsearch ......................................................... 1817.2.3. Client modes .......................................................................................................... 1827.2.4. Elasticsearch destination options .............................................................................. 182

7.3. Storing messages in plain-text files ..................................................................................... 1907.3.1. file() destination options .......................................................................................... 191

7.4. Storing messages on the Hadoop Distributed File System (HDFS) ........................................ 1987.4.1. Prerequisites ........................................................................................................... 1997.4.2. How syslog-ng PE interacts with HDFS ................................................................... 200

vsyslog-ng.com

7.4.3. Storing messages with MapR-FS ............................................................................. 2007.4.4. HDSF destination options ........................................................................................ 201

7.5. Publishing messages to Apache Kafka ................................................................................ 2077.5.1. Prerequisites ........................................................................................................... 2087.5.2. How syslog-ng PE interacts with Apache Kafka ........................................................ 2097.5.3. Kafka destination options ........................................................................................ 209

7.6. Storing messages in encrypted files ..................................................................................... 2157.6.1. Displaying the contents of logstore files ................................................................... 2177.6.2. Journal files ............................................................................................................ 2177.6.3. logstore() destination options ................................................................................... 219

7.7. Storing messages in a MongoDB database ........................................................................... 2257.7.1. How syslog-ng PE connects the MongoDB server ..................................................... 2267.7.2. mongodb() destination options ................................................................................. 227

7.8. Sending messages to a remote log server using the RFC3164 protocol (network() driver) ....... 2347.8.1. network() destination options ................................................................................... 235

7.9. Sending messages to named pipes ....................................................................................... 2467.9.1. pipe() destination options ........................................................................................ 246

7.10. Sending messages to external applications ......................................................................... 2517.10.1. program() destination options ................................................................................ 252

7.11. Generating SMTP messages (e-mail) from logs .................................................................. 2597.11.1. smtp() destination options ...................................................................................... 2607.11.2. Error handling ....................................................................................................... 267

7.12. Sending SNMP traps ........................................................................................................ 2677.12.1. Converting Cisco syslog messages to "clogMessageGenerated" SNMP traps ............ 2697.12.2. snmp() destination options ..................................................................................... 271

7.13. Storing messages in an SQL database ................................................................................ 2747.13.1. Using the sql() driver with an Oracle database ......................................................... 2767.13.2. Using the sql() driver with a Microsoft SQL database .............................................. 2777.13.3. The way syslog-ng interacts with the database ........................................................ 2787.13.4. MySQL-specific interaction methods ..................................................................... 2797.13.5. MsSQL-specific interaction methods ...................................................................... 2797.13.6. Supported SQL destinations by platform ................................................................. 2797.13.7. sql() destination options ......................................................................................... 280

7.14. Sending messages to a remote log server using the IETF-syslog protocol ............................ 2887.14.1. syslog() destination options ................................................................................... 289

7.15. Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp()drivers) .................................................................................................................................... 300

7.15.1. tcp(), tcp6(), udp(), and udp6() destination options .................................................. 3007.16. Sending messages to UNIX domain sockets ...................................................................... 301

7.16.1. unix-stream() and unix-dgram() destination options ................................................. 3027.17. Sending messages to a user terminal — usertty() destination ............................................... 309

8. Routing messages: log paths, reliability, and filters ..................................................................... 3118.1. Log paths .......................................................................................................................... 311

8.1.1. Embedded log statements ........................................................................................ 3128.1.2. Log path flags ......................................................................................................... 314

8.2. Managing incoming and outgoing messages with flow-control ............................................. 3168.2.1. Flow-control and multiple destinations ..................................................................... 3188.2.2. Configuring flow-control ......................................................................................... 319

visyslog-ng.com

8.3. Using disk-based and memory buffering ............................................................................. 3208.3.1. Enabling reliable disk-based buffering ...................................................................... 3228.3.2. Enabling normal disk-based buffering ...................................................................... 3238.3.3. Enabling memory buffering ..................................................................................... 323

8.4. Client-side failover ............................................................................................................ 3248.5. Filters ............................................................................................................................... 324

8.5.1. Using filters ............................................................................................................ 3258.5.2. Combining filters with boolean operators ................................................................. 3258.5.3. Comparing macro values in filters ............................................................................ 3268.5.4. Using wildcards, special characters, and regular expressions in filters ........................ 3278.5.5. Tagging messages ................................................................................................... 3288.5.6. Filter functions ....................................................................................................... 328

8.6. Dropping messages ............................................................................................................ 3339. Global options of syslog-ng PE ................................................................................................... 334

9.1. Configuring global syslog-ng options .................................................................................. 3349.2. Global options ................................................................................................................... 334

10. TLS-encrypted message transfer .............................................................................................. 35010.1. Secure logging using TLS ................................................................................................ 35010.2. Encrypting log messages with TLS ................................................................................... 351

10.2.1. Configuring TLS on the syslog-ng clients ............................................................... 35110.2.2. Configuring TLS on the syslog-ng server ................................................................ 352

10.3. Mutual authentication using TLS ...................................................................................... 35310.3.1. Configuring TLS on the syslog-ng clients ............................................................... 35410.3.2. Configuring TLS on the syslog-ng server ................................................................ 355

10.4. TLS options .................................................................................................................... 35611. FIPS-compliant syslog-ng ......................................................................................................... 362

11.1. Installing FIPS-compliant syslog-ng PE ............................................................................. 36211.2. Limitations of the FIPS-compliant syslog-ng PE ................................................................ 36211.3. Legal Notice of FIPS Compliance of Syslog-ng Premium Edition ....................................... 363

12. Reliable Log Transfer Protocol™ ............................................................................................ 36412.1. Logging using RLTP™ ................................................................................................... 364

12.1.1. How RLTP™ connections work ............................................................................. 36512.1.2. Using RLTP™ in a client-relay-server scenario ....................................................... 365

12.2. RLTP™ options ............................................................................................................... 36612.3. Examples for using RLTP™ ............................................................................................ 367

13. Reliability and minimizing the loss of log messages ................................................................... 36913.1. Introduction ..................................................................................................................... 36913.2. Flow control, no disk buffering, no RLTP™ ...................................................................... 36913.3. Flow control, normal disk buffering, no RLTP™ ................................................................ 37113.4. Flow control, reliable disk buffering, no RLTP™ ............................................................... 37313.5. Flow control, reliable disk buffering, RLTP™ .................................................................... 37513.6. Deciding which loss prevention mechanism to apply .......................................................... 377

14. Manipulating messages ............................................................................................................. 37814.1. Customizing message format ............................................................................................ 378

14.1.1. Formatting messages, filenames, directories, and tablenames ................................... 37814.1.2. Templates and macros ........................................................................................... 37914.1.3. Date-related macros .............................................................................................. 38114.1.4. Hard vs. soft macros .............................................................................................. 382

viisyslog-ng.com

14.1.5. Macros of syslog-ng PE ......................................................................................... 38314.1.6. Using template functions ....................................................................................... 39114.1.7. Template functions of syslog-ng PE ........................................................................ 392

14.2. Modifying messages ........................................................................................................ 39814.2.1. Replacing message parts ........................................................................................ 39914.2.2. Setting message fields to specific values ................................................................. 40014.2.3. Creating custom SDATA fields .............................................................................. 40114.2.4. Setting multiple message fields to specific values .................................................... 40114.2.5. Conditional rewrites .............................................................................................. 402

14.3. Regular expressions ......................................................................................................... 40314.3.1. Types and options of regular expressions ................................................................ 40414.3.2. Optimizing regular expressions .............................................................................. 406

15. Parsing and segmenting structured messages ............................................................................ 40715.1. Parsing messages with comma-separated and similar values ............................................... 407

15.1.1. Options of CSV parsers ......................................................................................... 40915.2. Parsing key=value pairs ................................................................................................. 412

15.2.1. Options of key=value parsers .............................................................................. 41315.3. The JSON parser ............................................................................................................. 414

15.3.1. Options of JSON parsers ....................................................................................... 41516. Processing message content with a pattern database ................................................................. 418

16.1. Classifying log messages .................................................................................................. 41816.1.1. The structure of the pattern database ....................................................................... 41916.1.2. How pattern matching works ................................................................................. 42016.1.3. Artificial ignorance ............................................................................................... 420

16.2. Using pattern databases .................................................................................................... 42116.2.1. Using parser results in filters and templates ............................................................ 42216.2.2. Downloading sample pattern databases ................................................................... 423

16.3. Correlating log messages .................................................................................................. 42416.3.1. Referencing earlier messages of the context ............................................................ 425

16.4. Triggering actions for identified messages ......................................................................... 42616.4.1. Conditional actions ............................................................................................... 42716.4.2. External actions .................................................................................................... 42816.4.3. Actions and message correlation ............................................................................ 429

16.5. Creating pattern databases ................................................................................................ 43116.5.1. Using pattern parsers ............................................................................................. 43116.5.2. The syslog-ng pattern database format .................................................................... 433

17. Statistics and metrics of syslog-ng ............................................................................................. 44618. Multithreading and scaling in syslog-ng PE .............................................................................. 449

18.1. Multithreading concepts of syslog-ng PE ........................................................................... 44918.2. Configuring multithreading .............................................................................................. 45018.3. Optimizing multithreaded performance ............................................................................. 451

19. Troubleshooting syslog-ng ......................................................................................................... 45419.1. Possible causes of losing log messages .............................................................................. 45419.2. Creating syslog-ng core files ............................................................................................ 45619.3. Collecting debugging information with strace, truss, or tusc ................................................ 45719.4. Running a failure script .................................................................................................... 45719.5. Stopping syslog-ng .......................................................................................................... 45919.6. Reporting bugs and finding help ....................................................................................... 459

viiisyslog-ng.com

19.7. Recover data from orphaned diskbuffer files ...................................................................... 45920. Best practices and examples ...................................................................................................... 460

20.1. General recommendations ................................................................................................ 46020.2. Handling large message load ............................................................................................ 46020.3. Using name resolution in syslog-ng ................................................................................... 461

20.3.1. Resolving hostnames locally .................................................................................. 46120.4. Collecting logs from chroot .............................................................................................. 46220.5. Configuring log rotation ................................................................................................... 463

Appendix A. The syslog-ng manual pages ...................................................................................... 465dqtool ...................................................................................................................................... 466loggen ..................................................................................................................................... 468lgstool ..................................................................................................................................... 472pdbtool .................................................................................................................................... 479persist-tool .............................................................................................................................. 485syslog-debun ........................................................................................................................... 490syslog-ng ................................................................................................................................. 493syslog-ng.conf ......................................................................................................................... 497syslog-ng-ctl ............................................................................................................................ 503syslog-ng-query ....................................................................................................................... 508syslog-windebun.ps1 ................................................................................................................ 511

Appendix B. END USER LICENSE AGREEMENT FOR BALABIT PRODUCT (EULA) ............ 513Appendix C. Open source licenses .................................................................................................. 529

C.1. GNU General Public License v2 ........................................................................................ 529C.1.1. Preamble ............................................................................................................... 529C.1.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION ANDMODIFICATION ............................................................................................................ 530C.1.3. How to Apply These Terms to Your New Programs .................................................. 534

C.2. GNU Lesser General Public License v2.1 ........................................................................... 535C.2.1. Preamble ............................................................................................................... 535C.2.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION ANDMODIFICATION ............................................................................................................ 537C.2.3. How to Apply These Terms to Your New Libraries ................................................... 542

C.3. License attributions ........................................................................................................... 543Appendix D. Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License.......................................................................................................................................................... 544Glossary .......................................................................................................................................... 549List of syslog-ng PE parameters ........................................................................................................ 553Index ............................................................................................................................................... 561

ixsyslog-ng.com

Preface

Welcome to the syslog-ng Premium Edition 6 LTS Administrator Guide!

This document describes how to configure and manage syslog-ng. Background information for the technologyand concepts used by the product is also discussed.

1. Summary of contents

Chapter 1, Introduction to syslog-ng (p. 1) describes the main functionality and purpose of syslog-ng PE.

Chapter 2, The concepts of syslog-ng (p. 8) discusses the technical concepts and philosophies behind syslog-ngPE.

Chapter 3, Installing syslog-ng (p. 36) describes how to install syslog-ng PE on various UNIX-based platformsusing the precompiled binaries.

Chapter 4, The syslog-ng PE quick-start guide (p. 68) provides a briefly explains how to perform the mostcommon log collecting tasks with syslog-ng PE.

Chapter 5, The syslog-ng PE configuration file (p. 75) discusses the configuration file format and syntax indetail, and explains how to manage large-scale configurations using included files and reusable configurationsnippets.

Chapter 6, Collecting log messages — sources and source drivers (p. 85) explains how to collect and receivelog messages from various sources.

Chapter 7, Sending and storing log messages — destinations and destination drivers (p. 167) describes thedifferent methods to store and forward log messages.

Chapter 8, Routing messages: log paths, reliability, and filters (p. 311) explains how to route and sort logmessages, and how to use filters to select specific messages.

Chapter 9, Global options of syslog-ng PE (p. 334) lists the global options of syslog-ng PE and explains howto use them.

Chapter 10, TLS-encrypted message transfer (p. 350) shows how to secure and authenticate log transport usingTLS encryption.

Chapter 12, Reliable Log Transfer Protocol™ (p. 364) describes the reliable log transport that prevents messageloss.

Chapter 13, Reliability and minimizing the loss of log messages (p. 369) describes how to use flow control, diskbuffering, and Reliable Log Transfer Protocol™ to minimize or completely prevent the loss of log messages.

Chapter 14, Manipulating messages (p. 378) describes how to customize message format using templates andmacros, how to rewrite and modify messages, and how to use regular expressions.

xsyslog-ng.com

Summary of contents

Chapter 15, Parsing and segmenting structured messages (p. 407) describes how to segment and processstructured messages like comma-separated values.

Chapter 16, Processing message content with a pattern database (p. 418) explains how to identify and processlog messages using a pattern database.

Chapter 17, Statistics and metrics of syslog-ng (p. 446) details the available statistics that syslog-ng PE collectsabout the processed log messages.

Chapter 18, Multithreading and scaling in syslog-ng PE (p. 449) describes how to configure syslog-ng PE touse multiple processors, and how to optimize its performance.

Chapter 19, Troubleshooting syslog-ng (p. 454) offers tips to solving problems.

Chapter 20, Best practices and examples (p. 460) gives recommendations to configure special features ofsyslog-ng PE.

Appendix A, The syslog-ng manual pages (p. 465) contains the manual pages of the syslog-ng PE application.

Appendix B, END USER LICENSE AGREEMENT FOR BALABIT PRODUCT (EULA) (p. 513) includes thetext of the End-User License Agreement applicable to syslog-ng Premium Edition.

Appendix C, Open source licenses (p. 529) includes the text of the licenses applicable to syslog-ng PremiumEdition.

Appendix D, Creative Commons Attribution Non-commercial NoDerivatives (by-nc-nd) License (p. 544) includesthe text of the Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License applicableto The syslog-ng Premium Edition 6 LTS Administrator Guide.

Glossary (p. 549) defines the important terms used in this guide.

List of syslog-ng PE parameters (p. 553) provides cross-references to the definitions of options, parameters,and macros available in syslog-ng PE.

The Index provides cross-references to important terms used in this guide.

2. Target audience and prerequisites

This guide is intended for system administrators and consultants responsible for designing and maintaininglogging solutions and log centers. It is also useful for IT decision makers looking for a tool to implementcentralized logging in heterogeneous environments.

The following skills and knowledge are necessary for a successful syslog-ng administrator:

■ At least basic system administration knowledge.

■ An understanding of networks, TCP/IP protocols, and general network terminology.

■ Working knowledge of the UNIX or Linux operating system.

■ In-depth knowledge of the logging process of various platforms and applications.

■ An understanding of the legacy syslog (BSD-syslog) protocol) and the new syslog (IETF-syslog)protocol) standard.

xisyslog-ng.com

Target audience and prerequisites

3. Products covered in this guide

This guide describes the use of the following products:

■ syslog-ng Premium Edition (syslog-ng PE) 6.0.1 and later

4. Typographical conventions

Before you start using this guide, it is important to understand the terms and typographical conventions usedin the documentation. For more information on specialized terms and abbreviations used in the documentation,see theGlossary at the end of this document.

The following kinds of text formatting and icons identify special information in the document.

TipTips provide best practices and recommendations.

NoteNotes provide additional information on a topic, and emphasize important facts and considerations.

WarningWarnings mark situations where loss of data or misconfiguration of the device is possible if the instructions are not obeyed.

Command Commands you have to execute.

Emphasis Reference items, additional readings.

/path/to/file File names.

Parameters Parameter and attribute names.

Label GUI output messages or dialog labels.

Menu A submenu or menu item in the menu bar.

Button Buttons in dialog windows.

5. Contact and support information

This product is developed and maintained by Balabit-Europe Ltd.. We are located in Budapest, Hungary. Ouraddress is:

Balabit-Europe Ltd.2 Alíz StreetH-1117

xiisyslog-ng.com

Products covered in this guide

Budapest, HungaryTel: +36 1 398-6700Fax: +36 1 208-0875E-mail: <info@balabit.com>Web: https://www.balabit.com/

5.1. Sales contact

You can directly contact our Sales Team with sales related topics.

5.2. Support contact

In case you experience a problem that is not covered in this guide, see the syslog-ng FAQ or the syslog-ngmailing list.

Product support, including 7x24 online support is available in various packages. For support options, visit thefollowing page.

To access the Balabit Online Support System (BOSS), sign up for an account at MyDownloads and requestaccess to the BalaBit Online Support System (BOSS). Online support is available 24 hours a day.

BOSS is available only for registered users with a valid support package.

Support e-mail address:<support@balabit.com>.

Support hotline: +36 1 398 6700 (available from 9 AM to 5 PM CET on weekdays)

5.3. Training

Balabit-Europe Ltd. holds courses on using its products for new and experienced users. For dates, details, andapplication forms, visit the https://my.balabit.com/training/ webpage.

6. About this document

This guide is a work-in-progress document with new versions appearing periodically.

The latest version of this document can be downloaded from the syslog-ng Documentation page.

6.1. Summary of changes

6.1.1. Version 5 F6 - 6 LTS

Changes in product:

■ The syslog-ng PE application can now separate a message consisting of whitespace orcomma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. Youcan also specify other separator character instead of the equal sign, for example, colon (:) to parseMySQL log messages. For details, see Section 15.2, Parsing key=value pairs (p. 412).

xiiisyslog-ng.com

Sales contact

■ The OpenSSL application used in syslog-ng PE has been updated to version 1.0.2, so you can usenew, stronger ciphers to protect the communication between your syslog-ng PE clients and servers.For an updated list of supported ciphers, see Section cipher-suite() (p. 358).

■ You can now specify the curves that are permitted in the connection using the curve-list() option,and also import Diffie-Hellman parameters from a file using the dhparam-file() option. Fordetails, see Section 10.4, TLS options (p. 356).

■ A new file source driver option, force-directory-polling(), has been introduced, which canbe used when the filename specified as the source file contains a wildcard to force the polling matchinglogifles. For details, see Section 6.4, Collecting messages from text files (p. 92) and Sectionforce-directory-polling() (p. 96).

Changes in documentation:

■ Extended the list of internal() source options with options host-override(), log-iw-size(),normalize-hostnames(), program-override(), and use-fqdn(). For details, see Sectionhost-override() (p. 91), Section log-iw-size() (p. 91), Section normalize-hostnames() (p. 91), Sectionprogram-override() (p. 91), and Section use-fqdn() (p. 92).

■ Added information about max-uid-query().

■ Added warning about the requirement to delete the persist file once the dir() option ofdisk-buffer() has been modified or a new one has been added. For more information, seeChapter7, Sending and storing log messages — destinations and destination drivers (p. 167).

■ Clarified information on the usage of the cert-subject() TLS option in Sectioncert-subject() (p. 357).

■ Reworked Section 18.3, Optimizing multithreaded performance (p. 451) to make information moreaccessible.

■ Editorial corrections.

6.1.2. Version 5 F3 - 5 F4

Changes in product:

■ Elasticsearch 2.x destination and Shield support is now available in the product. For details, seeSection 7.2, Sending messages directly to Elasticsearch version 2.0 or higher (p. 179).

■ A new template function, format-cef-extension is available to format name-value pairs asArcSight Common Event Format extensions. For details, see Section format-cef-extension (p. 392).

■ The Debian 6 (squeeze) platform is no longer supported.

■ You can now specify the location where syslog-ng PE stores the disk-buffer files using the dir()option of disk-buffer().

xivsyslog-ng.com

Summary of changes

■ Value-pairs now have a new option to select every value-pair that has a name beginning with aspecified prefix, but remove the prefix when formatting the message. For details, see Sectionvalue-pairs() (p. 27).

Changes in documentation:

■ Editorial corrections.

6.1.3. Version 5 F2 - 5 F3

Changes in product:

■ Section 7.4, Storing messages on the Hadoop Distributed File System (HDFS) (p. 198) has been addedto the document.

■ CSV-parsers can use strings as delimiters. For details, see Section delimiters() (p. 410).

Changes in documentation:

■ The tcp(), tcp6(), udp(), udp6() source and destination drivers have been deprecated, as all oftheir functionality can be achieved with the network() driver. For help on migrating to thenetwork() driver, see Procedure 6.14.1.1, Change an old source driver to the network()driver (p. 157) and Procedure 7.15.1.1, Change an old destination driver to the network()driver (p. 300).

■ The beginning of Chapter 19, Troubleshooting syslog-ng (p. 454) has been extended with basictroubleshooting information.

■ Section 1.6, Supported platforms (p. 4) has been updated.

■ Other editorial corrections.

6.1.4. Version 5 F1 - 5 F2

Changes in product:

■ Section netmask6() (p. 332) has been added to the document.

■ Section 3.2, Security-enhanced Linux: grsecurity, SELinux (p. 37) has been added to the document.

■ New examples have been added to Section hash (p. 395).

■ The use-rcptid() global option has been deprecated. Section use-uniqid() (p. 349) has been addedto the document.

■ The assume-utf8 source flag has been documented.

■ Multiple message fields can be modified using the groupset rewrite rule. For details, see Section14.2.4, Setting multiple message fields to specific values (p. 401).

■ Platforms CentOS 7, Ubuntu 14.04 LTS (Trusty Tahr), and Red Hat ES 7 are now supported.

■ Section 7.11, Generating SMTPmessages (e-mail) from logs (p. 259) has been added to the document.

■ Section 6.12, Collecting messages from the systemd-journal system log storage (p. 152) has beenadded to the document.

xvsyslog-ng.com

Summary of changes

■ Log rotation with syslog-ng PE has been described in Section 20.5, Configuring log rotation (p. 463).

■ The inlist() filter has been added to Section 8.5.6, Filter functions (p. 328).

■ The option time-sleep() is now deprecated.

6.1.5. Version 5 LTS - 5 F1

Changes in product:

■ Section 7.7, Storing messages in a MongoDB database (p. 225) has been added to the document.

■ A new parser is available to parse JSON-formatted messages. For details, see Section 15.3, The JSONparser (p. 414).

■ Section format-json (p. 393) has been added to the document.

Changes in documentation:

■ The retry_sql_inserts option has been renamed to retries to increase consistency.

■ Section on-error() (p. 343) can be set locally for MongoDB destinations as well.

■ Section timestamp-freq() (p. 347) has been added toChapter 9, Global options of syslog-ng PE (p. 334).

■ The FILE_NAME and SOURCE macros have been added to Section 14.1.5, Macros of syslog-ngPE (p. 383).

■ Section 2.12, Structuring macros, metadata, and other value-pairs (p. 25) has been added to thedocument.

■ Section on-error() (p. 343) has been added to the document.

■ Section 1.6, Supported platforms (p. 4) has been updated.

■ Chapter 12, Reliable Log Transfer Protocol™ (p. 364) has been expanded, including severalclarifications and improvements.

■ Option api() has been renamed to event-api() in Section 6.2.1, eventlog() source options (p. 88).

6.2. Feedback

Any feedback is greatly appreciated, especially on what else this document should cover. General comments,errors found in the text, and any suggestions about how to improve the documentation is welcome atdocumentation@balabit.com.

6.3. Acknowledgments

BalaBit would like to express its gratitude to the syslog-ng users and the syslog-ng community for their invaluablehelp and support.

xvisyslog-ng.com

Feedback

Chapter 1. Introduction to syslog-ng

This chapter introduces the syslog-ng Premium Edition application in a non-technical manner, discussing howand why is it useful, and the benefits it offers to an existing IT infrastructure.

1.1. What syslog-ng is

The syslog-ng Premium Edition (syslog-ng PE) application is a flexible and highly scalable system loggingapplication that is ideal for creating centralized and trusted logging solutions. Among others, syslog-ng PEallows you the following.

Secure and reliable log transferThe syslog-ng PE application enables you to send the log messages of your hosts to remote servers using thelatest protocol standards. You can collect and store your log data centrally on dedicated log servers. Transferlog messages using the RLTP™ protocol ensures that no messages are lost.

Disk-based message buffering. To minimize the risk of losing important log messages, the syslog-ng PEapplication can store messages on the local hard disk if the central log server or the network connection becomesunavailable. The syslog-ng application automatically sends the stored messages to the server when the connectionis reestablished, in the same order the messages were received. The disk buffer is persistent – no messages arelost even if syslog-ng is restarted.

Secure logging using TLS. Log messages may contain sensitive information that should not be accessed bythird parties. Therefore, syslog-ng PE supports the Transport Layer Security (TLS) protocol to encrypt thecommunication. TLS also allows you to authenticate your clients and the logserver using X.509 certificates.

Flexible data extraction and processingMost log messages are inherently unstructured, which makes them difficult to process. To overcome thisproblem, syslog-ng PE comes with a set of built-in parsers, which you can combine to build very complexthings.

Filter and classify. The syslog-ng PE application can sort the incoming log messages based on their contentand various parameters like the source host, application, and priority. You can create directories, files, anddatabase tables dynamically using macros. Complex filtering using regular expressions and boolean operatorsoffers almost unlimited flexibility to forward only the important log messages to the selected destinations.

Parse and rewrite. The syslog-ng PE application can segment log messages to named fields or columns,and also modify the values of these fields. You can process JSON messages, key-value pairs, and more.

To get the most information out of your log data, syslog-ng PE allows you to correlate log messages andaggregate the extracted information into a single message. You can also use external information to enrich yourlog data.

1syslog-ng.com

What syslog-ng is

Big data clustersThe log data that your organization has to process, store, and review increases daily, so many organizationsuse big data solutions for their logs. To accomodate this huge amount of data, syslog-ng PE natively supportsstoring log messages in HDFS files and Elasticsearch clusters.

Message queue supportLarge organizations increasingly rely on queuing infrastructure to transfer their data. syslog-ng PE supportsApache Kafka, and the Simple Text Oriented Messaging Protocol (STOMP).

SQL, NoSQL, and monitoringStoring your log messages in a database allows you to easily search and query the messages and interoperatewith log analyzing applications. The syslog-ng application supports the following databases: MongoDB, MSSQL,MySQL, Oracle, PostgreSQL, and SQLite.

Wide protocol and platform supportsyslog protocol standards. syslog-ng not only supports legacy BSD syslog (RFC3164) and the enhancedRFC5424 protocols, but also JavaScript Object Notation (JSON) and journald message formats.

Heterogeneous environments. The syslog-ng PE application is the ideal choice to collect logs in massivelyheterogeneous environments using several different operating systems and hardware platforms, including Linux,Unix, BSD, Sun Solaris, HP-UX, Tru64, and AIX.

IPv4 and IPv6 support. The syslog-ng application can operate in both IPv4 and IPv6 network environments,and can receive and send messages to both types of networks.

Client-side failoverWhen transferring messages to a remote server, the syslog-ng PE clients can be configured to send the logmessages to secondary servers if the primary server becomes unaccessible.

Encrypted and timestamped log storageThe syslog-ng PE application can store log messages securely in encrypted, compressed, and timestampedbinary files. Timestamps can be requested from an external Timestamping Authority (TSA).

Excellent performanceDepending on the exact syslog-ng PE configuration, environment, and other parameters, syslog-ng PE is capableof processing:

■ Over 590,000 messages per second (over 220 MB of data per second) when receiving messages frommultiple connections and storing them in text files.

■ Over 560,000 messages per second (over 210 MB of data per second) when receiving messages frommultiple connections and storing them in logstore files (that is, encrypted files).

■ Over 565,000 messages per second (over 210 MB of data per second) when receiving messages frommultiple secure (TLS-encrypted) connections and storing them in text files.

2syslog-ng.com

Big data clusters

1.2. What syslog-ng is not

The syslog-ng application is not log analysis software. It can filter log messages and select only the onesmatching certain criteria. It can even convert the messages and restructure them to a predefined format, or parsethe messages and segment them into different fields. But syslog-ng cannot interpret and analyze the meaningbehind the messages, or recognize patterns in the occurrence of different messages.

1.3. Why is syslog-ng needed?

Log messages contain information about the events happening on the hosts. Monitoring system events is essentialfor security and system health monitoring reasons.

The original syslog protocol separates messages based on the priority of the message and the facility sendingthe message. These two parameters alone are often inadequate to consistently classify messages, as manyapplications might use the same facility — and the facility itself is not even included in the log message. Tomake things worse, many log messages contain unimportant information. The syslog-ng application helps youto select only the really interesting messages, and forward them to a central server.

Company policies or other regulations often require log messages to be archived. Storing the important messagesin a central location greatly simplifies this process.

For details on how can you use syslog-ng PE to comply with various regulations, see the Regulatory complianceand system logging whitepaper.

1.4. What is new in syslog-ng Premium Edition 6 LTS?

■ For details on the news and highlights of syslog-ng Premium Edition 6 LTS, see the What is new insyslog-ng Premium Edition 6 LTS.

■ For details on changes in The syslog-ng Premium Edition 6 LTS Administrator Guide, see Section6.1.2, Version 5 F3 - 5 F4 (p. xiv).

1.5. Who uses syslog-ng?

The syslog-ng application is used worldwide by companies and institutions who collect and manage the logsof several hosts, and want to store them in a centralized, organized way. Using syslog-ng is particularlyadvantageous for:

■ Internet Service Providers

■ Financial institutions and companies requiring policy compliance

■ Server, web, and application hosting companies

■ Datacenters

■ Wide area network (WAN) operators

■ Server farm administrators.

1.5.1. Public references of syslog-ng Premium Edition

Among others, the following companies decided to use syslog-ng PE in their production environment:

3syslog-ng.com

What syslog-ng is not

■ Air France

■ Coop Denmark

■ DataPath, Inc. (Read Case Study)

■ Facebook

■ Hush Communications Canada Inc.

■ Tecnocom Espana Solutions, S.L. (Read Case Study)

1.6. Supported platforms

The syslog-ng Premium Edition application is officially supported on the following platforms. Note that thefollowing table is for general reference only, and is not always accurate about the supported platforms andoptions available for specific platforms. The latest version of this table is available athttps://syslog-ng.com/log-management-software/supported-platforms. Unless explicitly noted otherwise, thesubsequent releases of the platform (for example, Windows Server 2008 R2 and its service packs in case ofWindows Server 2008) are also supported.

IA64PowerPCSPARC64SPARCx86_64x86-✔----AIX 7.1

----✔✔CentOS 5

----✔✔CentOS 6

----✔-CentOS 7

----✔✔Debian 7(wheezy)

----✔-Debian 8(jessie)

----✔-FreeBSD 10

✔-----HP-UX 11i v3

----✔✔Oracle Linux 5

----✔✔Oracle Linux 6

----✔-Oracle Linux 7

----✔✔openSUSE 11

----✔✔Red Hat EL 5

----✔✔Red Hat EL 6

----✔-Red Hat EL 7

----✔✔SLES 11

----✔-SLES 12

--✔✔✔-Solaris 10

--✔-✔-Solaris 11

4syslog-ng.com

Supported platforms

IA64PowerPCSPARC64SPARCx86_64x86----✔✔Ubuntu 12.04

LTS (PrecisePangolin)

----✔✔Ubuntu 14.04LTS (TrustyTahr)

----✔-Ubuntu 16.04LTS (XenialXerus)

----✔✔W i n d o w sServer 2008

----✔-W i n d o w sServer 2012

----✔-W i n d o w sServer 2016

----✔✔W i n d o w sVista

----✔✔Windows 7

----✔✔Windows 8

----✔✔Windows 10Table 1.1. Platforms supported by syslog-ng Premium Edition

WarningCertain sources and destinations are not supported on every platform (particularly, the sql() source and destination, andthe hdfs() destination). For details, see the description of the specific source and destination.

The central syslog-ng PE server can be installed on Microsoft Windows platforms as well, in this case you canconfigure syslog-ng PE using a configuration file, like on any other platform. However, if you want only toforward eventlog and other log messages from Windows to your central logserver, you can use the syslog-ngAgent for Windows application. The syslog-ng Agent for Windows can be managed centrally from a domaincontroller, and can be configured from a graphical interface as well. The syslog-ng Agent for Windows applicationis available as part of syslog-ng Premium Edition.

For details about the syslog-ng Agent for Windows application, see The syslog-ng Agent for Windows 6 LTSAdministrator Guide.

1.6.1. Limitations on Microsoft Windows platforms

The following features and options of syslog-ng PE are not available on Microsoft Windows platforms.

■ IPv6 is not supported, only IPv4

5syslog-ng.com

Limitations on Microsoft Windows platforms

■ Java-based destinations, like Elasticsearch, Hadoop Distributed File System (HDFS), or ApacheKafka.

■ The mongodb() destination

■ The pipe() source

■ The pipe() destination

■ The program() source

■ The program() destination

■ The smtp() destination

■ The snmp() destination

■ The sql() source

■ The sql() destination

■ The sun-streams() source

■ The unix-dgram() source

■ The unix-dgram() destination

■ The unix-stream() source

■ The unix-stream() destination

1.6.2. Certified packages

Starting from version 4.0, syslog-ng Premium Edition is Novell Ready certified for the following platforms:

6syslog-ng.com

Certified packages

■ SUSE Linux Enterprise Server 10 on the x86 and x86_64 AMD64 & Intel EM64T architectures

■ SUSE Linux Enterprise Server 11 on the x86 and x86_64 AMD64 & Intel EM64T architectures

Starting from version 4.0, syslog-ng Premium Edition is RedHat Ready certified for the following platforms:

■ Red Hat Enterprise Linux 2.1 on the x86 architecture

■ Red Hat Enterprise Linux 3 on the x86_64 AMD64 & Intel EM64T architecture

■ Red Hat Enterprise Linux 4 on the x86 and x86_64 AMD64 & Intel EM64T architectures

■ Red Hat Enterprise Linux 5 on the x86 and x86_64 AMD64 & Intel EM64T architectures

■ Red Hat Enterprise Linux 6 on the x86 and x86_64 AMD64 & Intel EM64T architectures

Starting from version 5.4, syslog-ng Premium Edition is MapR certified.

7syslog-ng.com

Certified packages

Chapter 2. The concepts of syslog-ng

This chapter discusses the technical concepts of syslog-ng.

2.1. The philosophy of syslog-ng

Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is tocollect the log messages of several devices on a single, central log server. The different devices — calledsyslog-ng clients — all run syslog-ng, and collect the log messages from the various applications, files, andother sources. The clients send all important log messages to the remote syslog-ng server, which sorts andstores them.

2.2. Logging with syslog-ng

The syslog-ng application reads incoming messages and forwards them to the selected destinations. Thesyslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects, log paths define what syslog-ng does with a message,connecting the sources to the destinations. A log path consists of one or more sources and one or moredestinations: messages arriving from a source are sent to every destination listed in the log path. A log pathdefined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selectingonly messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messagessatisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segmentmessages into different fields to help processing the messages, while rewrite rules modify the messages byadding, replacing, or removing parts of the messages.

2.2.1. Procedure – The route of a log message in syslog-ng

Purpose:

The following procedure illustrates the route of a log message from its source on the syslog-ng client to its finaldestination on the central syslog-ng server.

8syslog-ng.com

The philosophy of syslog-ng

Figure 2.1. The route of a log message

Steps:

Step 1. A device or application sends a log message to a source on the syslog-ng client. For example, anApache web server running on Linux enters a message into the /var/log/apache file.

Step 2. The syslog-ng client running on the web server reads the message from its /var/log/apache source.

Step 3. The syslog-ng client processes the first log statement that includes the /var/log/apache source.

Step 4. The syslog-ng client performs optional operations (message filtering, parsing, and rewriting) on themessage, for example, it compares the message to the filters of the log statement (if any). If the messagecomplies with all filter rules, syslog-ng sends the message to the destinations set in the log statement,for example, to the remote syslog-ng server.

WarningMessage filtering, parsing, and rewriting is performed in the order that the operations appear in the logstatement.

NoteThe syslog-ng client sends a message to all matching destinations by default. As a result, a message may besent to a destination more than once, if the destination is used in multiple log statements. To prevent suchsituations, use the final flag in the destination statements. For details, see Table 8.1, Log statementflags (p. 314).

9syslog-ng.com

Logging with syslog-ng

Step 5. The syslog-ng client processes the next log statement that includes the /var/log/apache source,repeating Steps 3-4.

Step 6. The message sent by the syslog-ng client arrives from a source set in the syslog-ng server.

Step 7. The syslog-ng server reads the message from its source and processes the first log statement thatincludes that source.

Step 8. The syslog-ng server performs optional operations (message filtering, parsing, and rewriting) on themessage, for example, it compares the message to the filters of the log statement (if any). If the messagecomplies with all filter rules, syslog-ng sends the message to the destinations set in the log statement.

WarningMessage filtering, parsing, and rewriting is performed in the order that the operations appear in the logstatement.

Step 9. The syslog-ng server processes the next log statement, repeating Steps 7-9.

NoteThe syslog-ng application can stop reading messages from its sources if the destinations cannot process thesent messages. This feature is called flow-control and is detailed in Section 8.2, Managing incoming andoutgoing messages with flow-control (p. 316).

2.3. Modes of operation

The syslog-ng Premium Edition application has three distinct operation scenarios: Client, Server, and Relay.The syslog-ng PE application running on a host determines the mode of operation automatically based on thelicense and the configuration file.

2.3.1. Client mode

Figure 2.2. Client-mode operation

In client mode, syslog-ng collects the local logs generated by the host and forwards them through a networkconnection to the central syslog-ng server or to a relay. Clients often also log the messages locally into files.

No license file is required to run syslog-ng in client mode.

10syslog-ng.com

Modes of operation

2.3.2. Relay mode

Figure 2.3. Relay-mode operation

In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to thecentral syslog-ng server using a network connection. Relays also log the messages from the relay host into alocal file, or forward these messages to the central syslog-ng server.

You cannot use the following destinations in relay mode: mongodb(), pipe(), sql(). The file() andlogstore() (and in case of running an syslog-ng PE relay on Microsoft Windows, eventlog()) destinationswork only for local messages that are generated on the relay.

No license file is required to run syslog-ng in relay mode.

2.3.3. Server mode

Figure 2.4. Server-mode operation

In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clientsand relays over the network, and stores them locally in files, or passes them to other applications, for examplelog analyzers.

Running syslog-ng Premium Edition in server mode requires a license file. The license determines how manyindividual hosts can connect to the server. For details on how syslog-ng PE calculates the number of hosts, seeSection 2.7, Licensing (p. 15).

11syslog-ng.com

Relay mode

2.4. Global objects

The syslog-ng application uses the following objects:

■ Source driver: A communication method used to receive log messages. For example, syslog-ng canreceive messages from a remote host via TCP/IP, or read the messages of a local application froma file. For details on source drivers, see Chapter 6, Collecting log messages — sources and sourcedrivers (p. 85).

■ Source: A named collection of configured source drivers.

■ Destination driver: A communication method used to send log messages. For example, syslog-ngcan send messages to a remote host via TCP/IP, or write the messages into a file or database. Fordetails on destination drivers, see Chapter 7, Sending and storing log messages — destinations anddestination drivers (p. 167).

■ Destination: A named collection of configured destination drivers.

■ Filter: An expression to select messages. For example, a simple filter can select the messages receivedfrom a specific host. For details, see Section 14.1, Customizing message format (p. 378).

■ Macro: An identifier that refers to a part of the log message. For example, the ${HOST}macro returnsthe name of the host that sent the message. Macros are often used in templates and filenames. Fordetails, see Section 14.1, Customizing message format (p. 378).

■ Parser: Parsers are objects that parse the incoming messages, or parts of a message. For example,the csv-parser() can segment messages into separate columns at a predefined separator character(for example a comma). Every column has a unique name that can be used as a macro. For details,see Chapter 15, Parsing and segmenting structured messages (p. 407) and Chapter 16, Processingmessage content with a pattern database (p. 418).

■ Rewrite rule: A rule modifies a part of the message, for example, replaces a string, or sets a field toa specified value. For details, see Section 14.2, Modifying messages (p. 398).

■ Log paths: A combination of sources, destinations, and other objects like filters, parsers, and rewriterules. The syslog-ng application sends messages arriving from the sources of the log paths to thedefined destinations, and performs filtering, parsing, and rewriting of the messages. Log paths arealso called log statements. Log statements can include other (embedded) log statements and junctionsto create complex log paths. For details, see Chapter 8, Routing messages: log paths, reliability, andfilters (p. 311).

■ Template: A template is a set of macros that can be used to restructure log messages or automaticallygenerate file names. For example, a template can add the hostname and the date to the beginning ofevery log message. For details, see Section 14.1, Customizing message format (p. 378).

■ Option: Options set global parameters of syslog-ng, like the parameters of name resolution andtimezone handling. For details, see Chapter 9, Global options of syslog-ng PE (p. 334).

12syslog-ng.com

Global objects

For details on the above objects, see Section 5.2, The configuration syntax in detail (p. 75).

2.5. Timezones and daylight saving

The syslog-ng application receives the timezone and daylight saving information from the operating system itis installed on. If the operating system handles daylight saving correctly, so does syslog-ng.

The syslog-ng application supports messages originating from different timezones. The original syslog protocol(RFC3164) does not include timezone information, but syslog-ng provides a solution by extending the syslogprotocol to include the timezone in the log messages. The syslog-ng application also enables administrators tosupply timezone information for legacy devices which do not support the protocol extension.

2.5.1. Procedure – How syslog-ng PE assigns timezone to the message

When syslog-ng PE receives a message, it assigns timezone information to the message using the followingalgorithm.

Step 1. The sender application (for example the syslog-ng client) or host specifies the timezone of the messages.If the incoming message includes a timezone it is associated with the message. Otherwise, the localtimezone is assumed.

Step 2. Specify the time-zone() parameter for the source driver that reads the message. This timezone willbe associated with the messages only if no timezone is specified within the message itself. Each sourcedefaults to the value of the recv-time-zone() global option. It is not possible to override only thetimezone information of the incoming message, but setting the keep-timestamp() option to no

allows syslog-ng PE to replace the full timestamp (timezone included) with the time the message wasreceived.

NoteWhen processing a message that does not contain timezone information, the syslog-ng PE application willuse the timezone and daylight-saving that was effective when the timestamp was generated. For example,the current time is 2011-03-11 (March 11, 2011) in the EU/Budapest timezone. When daylight-saving isactive (summertime), the offset is +02:00. When daylight-saving is inactive (wintertime) the timezone offsetis +01:00. If the timestamp of an incoming message is 2011-01-01, the timezone associated with themessage will be +01:00, but the timestamp will be converted, because 2011-01-01 meant winter time whendaylight saving is not active but the current timezone is +02:00.

Step 3. Specify the timezone in the destination driver using the time-zone() parameter. Each destinationdriver might have an associated timezone value: syslog-ng converts message timestamps to this timezonebefore sending the message to its destination (file or network socket). Each destination defaults to thevalue of the send-time-zone() global option.

NoteA message can be sent to multiple destination zones. The syslog-ng application converts the timezoneinformation properly for every individual destination zone.

13syslog-ng.com

Timezones and daylight saving

WarningIf syslog-ng PE sends the message is to the destination using the legacy-syslog protocol (RFC3164) whichdoes not support timezone information in its timestamps, the timezone information cannot be encapsulatedinto the sent timestamp, so syslog-ng PE will convert the hour:min values based on the explicitly specifiedtimezone.

Step 4. If the timezone is not specified, local timezone is used.

Step 5. When macro expansions are used in the destination filenames, the local timezone is used. (Also, if thetimestamp of the received message does not contain the year of the message, syslog-ng PE uses thelocal year.)

2.5.2. A note on timezones and timestamps

If the clients run syslog-ng, then use the ISO timestamp, because it includes timezone information. That wayyou do not need to adjust the recv-time-zone() parameter of syslog-ng.

If you want syslog-ng to output timestamps in Unix (POSIX) time format, use the S_UNIXTIME and R_UNIXTIMEmacros. You do not need to change any of the timezone related parameters, because the timestamp informationof incoming messages is converted to Unix time internally, and Unix time is a timezone-independent timerepresentation. (Actually, Unix time measures the number of seconds elapsed since midnight of CoordinatedUniversal Time (UTC) January 1, 1970, but does not count leap seconds.)

2.6. Versions and releases of syslog-ng PE

As of June 2011, the following release policy applies to syslog-ng Premium Edition:

■ Long Term Supported or LTS releases (for example, syslog-ng Agent 4 LTS) are supported for 3years after their original publication date and for 1 year after the next LTS release is published(whichever date is later). The second digit of the revisions of such releases is 0 (for example, syslog-ngPE 4.0.1). Maintenance releases to LTS releases contain only bugfixes and security updates.

■ Feature releases (for example, syslog-ng Agent 4 F1) are supported for 6 months after their originalpublication date and for 2 months after succeeding Feature or LTS Release is published (whicheverdate is later). Feature releases contain enhancements and new features, presumably 1-3 new featureper release. Only the last of the feature releases is supported (for example when a new feature releasecomes out, the last one becomes unsupported).For a full description on stable and feature releases, see the Balabit version policy.

WarningDowngrading from a feature release to an earlier (and thus unsupported) feature release, or to the previousLTS release is officially not supported, but usually works as long as your syslog-ng PE configuration fileis appropriate for the old syslog-ng PE version. However, persistent data like the position of the lastprocessed message in a file source will be probably lost.

Logstore files created with a newer version of syslog-ng PE might not be readable with an older versionof syslog-ng PE.

14syslog-ng.com

A note on timezones and timestamps

2.7. Licensing

2.7.1. Licensing benefits

Buying a syslog-ng Premium Edition (syslog-ng PE) license permits you to perform the following:

■ Install one instance of the syslog-ng PE application in server mode to a single host. This host actsas the central log server of the network. You have to install the license file only on this host.

■ Install the syslog-ng PE application in relay or client mode on host computers within your organization(on any supported platform). You cannot redistribute the application to third parties. The total numberof hosts permitted to run syslog-ng in relay or client mode is limited by the syslog-ng PE license.The client and relay hosts may use any operating system supported by syslog-ng PE. For details, seethe Supported platforms in syslog-ng Premium Edition page.

The syslog-ng Premium Edition license determines the number of individual hosts (also called log source hosts)that can send log messages to syslog-ng PE.

License grants and legal restrictions are fully described in the General End User License Agreement (EULA).Note that the EULA and the syslog-ng Premium Edition Product Usage Terms apply only to scenarios wherethe Licensee (the organization who has purchased the product) is the end user of the product. In any otherscenario — for example, if you want to offer services provided by syslog-ng Premium Edition to your customersin an OEM or a Managed Service Provider (MSP) scenario — you have to negotiate the exact terms andconditions with One Identity.

2.7.2. Licensing model and modes of operation

A Log Source Host (LSH) is any host, server, or device (including virtual machines, active or passive networkingdevices, syslog-ng clients and relays, and so on) that is capable of sending log messages. Log Source Hosts areidentified by their IP addresses, so virtual machines and vhosts are separately counted.

The syslog-ng Premium Edition application has three distinct modes of operation: Client, Relay, and Server.

■ In Client mode syslog-ng Premium Edition collects local logs generated by the host it is running on,and forwards them through a network connection to the central syslog-ng PE server, a relay, oranother network destination. If you install the syslog-ng Premium Edition application in Client modeon a host, it counts as a Log Source Host, even if it does not send log messages to a syslog-ngPremium Edition server.

■ In Relay mode syslog-ng Premium Edition receives logs through the network from Log Source Hostsand forwards them to the central syslog-ng PE server, a relay, or another network destination. If youinstall the syslog-ng Premium Edition application in Relay mode on a host, it counts as a Log SourceHost, even if it does not send log messages to a syslog-ng Premium Edition server.Relays cannot store the received log messages in local files, except for the log messages of the relayhost. Naturally, relays can use disk-based buffering for every message.

■ In Server mode syslog-ng Premium Edition acts as a central log-collecting server that receivesmessages through a network connection, and stores them locally, or forwards them to other destinationsor external systems (for example, a SIEM or a database). Installing the syslog-ng Premium Edition

15syslog-ng.com

Licensing

application in Server mode requires a license file, this license file determines the number of LogSource Hosts that can send log messages to the syslog-ng Premium Edition server.Note that the number of source hosts is important, not the number of hosts that directly sends messagesto syslog-ng Premium Edition: every host that send messages to the server (directly or using a relay)counts as a Log Source Host.

Server modeRelay modeClient mode✔✔✔Collect the local logs of the

host

✔✔✔Forward local logs over thenetwork

✔✔✔Store local messages inlocal files

✔✔noReceive logs over thenetwork

✔✔noForward received logs overthe network

✔nonoStore received logs in localfiles

✔nonoForward logs using specialdestinations (for example,databases)

✔nonoRequires license fileTable 2.1. Modes of operation in syslog-ng PE

2.7.2.1. Notes about counting the licensed hosts

WarningIf the actual IP address of the host differs from the IP address received by looking up its IP address fromits hostname in the DNS, the syslog-ng server counts them as two different hosts.

■

■ The chain-hostnames() option of syslog-ng can interfere with the way syslog-ng PE counts the logsource hosts, causing syslog-ng to think there are more hosts logging to the central server, especially if theclients sends a hostname in the message that is different from its real hostname (as resolved from DNS).Disable the chain-hostnames() option on your log source hosts to avoid any problems related to licensecounting.

■ If the number of Log Source Hosts reaches the license limit, the syslog-ng PE server will not acceptconnections from additional hosts. The messages sent by additional hosts will be dropped, even if the clientuses a reliable transport method (for example, RLTP).

■ If the no-parse flag is set in a message source on the syslog-ng PE server, syslog-ng PE assumes that themessage arrived from the host (that is, from the last hop) that sent the message to syslog-ng PE, andinformation about the original sender is lost.

16syslog-ng.com

Licensing model and modes of operation

2.7.3. Licensing examples

Example 2.1. A simple exampleScenario:

You want to install syslog-ng PE in server mode on a log server.■

■ 45 servers with syslog-ng PE installed in client mode send logs to the syslog-ng PE log server.

■ 45 networks devices without syslog-ng PE installed send logs to the syslog-ng PE log server.

License requirements: You need a syslog-ng Premium Edition license for at least 100 Log Source Host (LSH) asthere are 90 LSHs (45+45=90) in this scenario.

Example 2.2. High Availability (HA) clusterScenario:

You want to install syslog-ng PE in server mode on two hosts that run as an active-passive high-availabilitycluster.

■

■ 45 servers with syslog-ng PE installed in client mode send logs to the syslog-ng PE log server.

■ 45 networks devices without syslog-ng PE installed send logs to the syslog-ng PE log server.

License requirements: You need a syslog-ng Premium Edition license for at least 100 Log Source Host (LSH) asthere are 90 LSHs (45+45=90) in this scenario. You also need a High Availability (HA) license for the passive log server.

Example 2.3. Using alternative log servers with syslog-ng PE clientsScenario:

You want to install syslog-ng PE in server mode on a log server.■

■ 45 servers with syslog-ng PE installed in client mode send logs to the syslog-ng PE log server.

■ 45 networks devices without syslog-ng PE installed send logs to the syslog-ng PE log server.

■ 100 servers with syslog-ng PE installed send log messages to a log server without syslog-ng PE installed.

License requirements: You need a syslog-ng Premium Edition license for at least 200 LSHs as there are 190 LSHs(45+45 that send logs to a syslog-ng PE log server, and another 100 that run syslog-ng PE, 45+45+100=190) in thisscenario.

Example 2.4. Using syslog-ng PE relaysScenario:

You want to install syslog-ng PE in server mode on a log server.■

■ 45 servers with syslog-ng PE installed in client mode send logs directly to the syslog-ng PE log server.

■ 5 servers with syslog-ng PE installed in relay mode send logs to the syslog-ng PE log server.

■ Every syslog-ng PE relay receives logs from 9 networks devices without syslog-ng PE installed (a total of45 devices).

■ 100 servers with syslog-ng PE installed send log messages to a log server without syslog-ng PE installed.

License requirements: You need syslog-ng Premium Edition license for at least 200 LSH as there are 195 LSHs(45+5+(5*9)+100=195) in this scenario.

Example 2.5. Multiple facilitiesYou have two facilities (for example data centers or server farms). Facility 1 has 75 AIX servers and 20 MicrosoftWindows hosts, Facility 2 has 5 HP-UX servers and 40 Debian servers. That is 140 hosts altogether.

17syslog-ng.com

Licensing examples

NoteIf, for example, the 40 Debian servers at Facility 2 are each running 3 virtual hosts, thenthe total number of hosts at Facility 2 is 125, and the license sizes in the following examplesshould be calculated accordingly.

■ Scenario: The log messages are collected to a single, central syslog-ng PE log server.License requirements: You need a syslog-ng Premium Edition license for 150 LSH as there are 140LSHs (75+20+5+40) in this scenario.

■ Scenario: Each facility has its own syslog-ng PE log server, and there is no central log server.License requirements: You need two separate licenses: a license for at least 95 LSHs (75+20) at Facility1, and a license for at least 45 LSHs (5+40) at Facility 2. You need a license for 100 LSHs at Facility 1,and a license for 50 LSHs at Facility 2.

■ Scenario: The log messages are collected to a single, central syslog-ng PE log server. Facility 1 and 2each have a syslog-ng PE relay that forwards the log messages to the central syslog-ng PE log server.License requirements: You need a syslog-ng Premium Edition license for 150 LSH as there are 142LSHs (1+75+20+1+5+40) in this scenario (since the relays are also counted as an LSH).

■ Scenario: Each facility to has its own local syslog-ng PE log server, and there is also a central syslog-ngPE log server that collects every log message independently from the two local log servers.License requirements: You need three separate licenses. A syslog-ng Premium Edition a license for atleast 95 LSHs (75+20) at Facility 1, a license for at least 45 LSHs (5+40) at Facility 2, and also a licensefor at least 147 LSHs for the central syslog-ng Premium Edition log server (assuming that you want tocollect the logs of the local log servers as well).

2.8. GPL and LGPL licenses

Starting with version 4 F1, the syslog-ng Premium Edition application is based on the syslog-ng Open SourceEdition application, and includes elements that are licensed under the LGPL or GPL licenses. You can downloadthe core of syslog-ng PE here. The components located under the /lib directory are licensed under the GNULesser General Public License Version 2.1 license, while the rest of the codebase is licensed under the GNUGeneral Public License Version 2 license. External libraries and other dependencies used by syslog-ng PE havetheir own licenses, typically GPL, LGPL, MIT, or BSD.

Appendix C, Open source licenses (p. 529) includes the text of the licenses applicable to syslog-ng PremiumEdition.

2.9. High availability support

Multiple syslog-ng servers can be run in fail-over mode. The syslog-ng application does not include any internalsupport for this, as clustering support must be implemented on the operating system level. A tool that can beused to create UNIX clusters is Heartbeat (for details, see this page).

One Identity also has a log server appliance called syslog-ng Store Box that supports high-availability. Fordetails, see the syslog-ng Store Box Product Page.

2.10. The structure of a log message

The following sections describe the structure of log messages. Currently there are two standard syslog messageformats:

18syslog-ng.com

GPL and LGPL licenses

■ The old standard described in RFC 3164 (also called the BSD-syslog or the legacy-syslog protocol):see Section 2.10.1, BSD-syslog or legacy-syslog messages (p. 19)

■ The new standard described in RFC 5424 (also called the IETF-syslog protocol): see Section 2.10.2,IETF-syslog messages (p. 21)

■ How messages are represented in syslog-ng PE: see Section 2.11, Message representation in syslog-ngPE (p. 24).

2.10.1. BSD-syslog or legacy-syslog messages

This section describes the format of a syslog message, according to the legacy-syslog or BSD-syslog protocol.A syslog message consists of the following parts:

■ PRI

■ HEADER

■ MSG

The total message cannot be longer than 1024 bytes.

The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The messagecorresponds to the following format: <priority>timestamp hostname application: message. The different partsof the message are explained in the following sections.

NoteThe syslog-ng application supports longer messages as well. For details, see the log-msg-size() option in Section 9.2,Global options (p. 334). However, it is not recommended to enable messages larger than the packet size when using UDPdestinations.

2.10.1.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message.Facility represents the part of the system sending the message, while severity marks its importance. The Priorityvalue is calculated by first multiplying the Facility number by 8 and then adding the numerical value of theSeverity. The possible facility and severity values are presented below.

NoteFacility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numericalvalues as well.

FacilityNumerical Codekernel messages0

user-level messages1

mail system2

19syslog-ng.com

BSD-syslog or legacy-syslog messages

FacilityNumerical Codesystem daemons3

security/authorization messages4

messages generated internally by syslogd5

line printer subsystem6

network news subsystem7

UUCP subsystem8

clock daemon9

security/authorization messages10

FTP daemon11

NTP subsystem12

log audit13

log alert14

clock daemon15

locally used facilities (local0-local7)16-23Table 2.2. syslog Message Facilities

The following table lists the severity values.

SeverityNumerical CodeEmergency: system is unusable0

Alert: action must be taken immediately1

Critical: critical conditions2

Error: error conditions3

Warning: warning conditions4

Notice: normal but significant condition5

Informational: informational messages6

Debug: debug-level messages7Table 2.3. syslog Message Severities

2.10.1.2. The HEADER message part

The HEADER part contains a timestamp and the hostname (without the domain name) or the IP address of thedevice. The timestamp field is the local time in the Mmm dd hh:mm:ss format, where:

■ Mmm is the English abbreviation of the month: Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct,Nov, Dec.

■ dd is the day of the month on two digits. If the day of the month is less than 10, the first digit isreplaced with a space. (For example Aug 7.)

20syslog-ng.com

BSD-syslog or legacy-syslog messages

■ hh:mm:ss is the local time. The hour (hh) is represented in a 24-hour format. Valid entries are between00 and 23, inclusive. The minute (mm) and second (ss) entries are between 00 and 59 inclusive.

NoteThe syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. For details,see the ts-format() option in Section 9.2, Global options (p. 334).

2.10.1.3. The MSG message part

The MSG part contains the name of the program or process that generated the message, and the text of themessage itself. The MSG part is usually in the following format: program[pid]: message text.

2.10.2. IETF-syslog messages

This section describes the format of a syslog message, according to the IETF-syslog protocol. A syslog messageconsists of the following parts:

■ HEADER (includes the PRI as well)

■ STRUCTURED-DATA

■ MSG

The following is a sample syslog message:

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed

for lonvick on /dev/pts/8

The message corresponds to the following format:

<priority>VERSION ISOTIMESTAMP HOSTNAME APPLICATION PID MESSAGEID STRUCTURED-DATA

MSG

In this example, the Facility has the value of 4, severity is 2, so PRI is 34. The VERSION is 1. The messagewas created on 11 October 2003 at 10:14:15pm UTC, 3 milliseconds into the next second. The message originatedfrom a host that identifies itself as "mymachine.example.com". The APP-NAME is "su" and the PROCID isunknown. The MSGID is "ID47". The MSG is "'su root' failed for lonvick...", encoded in UTF-8. The encodingis defined by theBOM. There is no STRUCTURED-DATA present in the message, this is indicated by "-" inthe STRUCTURED-DATA field. The MSG is "'su root' failed for lonvick...".

The HEADER part of the message must be in plain ASCII format, the parameter values of theSTRUCTURED-DATA part must be in UTF-8, while the MSG part should be in UTF-8. The different partsof the message are explained in the following sections.

Source: https://tools.ietf.org/html/rfc5424The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

21syslog-ng.com

IETF-syslog messages

2.10.2.1. The PRI message part

The PRI part of the syslog message (known as Priority value) represents the Facility and Severity of the message.Facility represents the part of the system sending the message, while severity marks its importance. The Priorityvalue is calculated by first multiplying the Facility number by 8 and then adding the numerical value of theSeverity. The possible facility and severity values are presented below.

NoteFacility codes may slightly vary between different platforms. The syslog-ng application accepts facility codes as numericalvalues as well.

FacilityNumerical Codekernel messages0

user-level messages1

mail system2

system daemons3

security/authorization messages4

messages generated internally by syslogd5

line printer subsystem6

network news subsystem7

UUCP subsystem8

clock daemon9

security/authorization messages10

FTP daemon11

NTP subsystem12

log audit13

log alert14

clock daemon15

locally used facilities (local0-local7)16-23Table 2.4. syslog Message Facilities

The following table lists the severity values.

SeverityNumerical CodeEmergency: system is unusable0

Alert: action must be taken immediately1

Critical: critical conditions2

Error: error conditions3

22syslog-ng.com

IETF-syslog messages

SeverityNumerical CodeWarning: warning conditions4

Notice: normal but significant condition5

Informational: informational messages6

Debug: debug-level messages7Table 2.5. syslog Message Severities

2.10.2.2. The HEADER message part

The HEADER part contains the following elements:

■ VERSION: Version number of the syslog protocol standard. Currently this can only be 1.

■ ISOTIMESTAMP: The time when the message was generated in the ISO 8601 compatible standardtimestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example:2006-06-13T15:58:00.123+01:00.

■ HOSTNAME: The machine that originally sent the message.

■ APPLICATION: The device or application that generated the message

■ PID: The process name or process ID of the syslog application that sent the message. It is notnecessarily the process ID of the application that generated the message.

■ MESSAGEID: The ID number of the message.

NoteThe syslog-ng application supports other timestamp formats as well, like ISO, or the PIX extended format. The timestampused in the IETF-syslog protocol is derived from RFC3339, which is based on ISO8601. For details, see the ts-format()option in Section 9.2, Global options (p. 334).

The syslog-ng PE application will truncate the following fields:

■ If APP-NAME is longer than 48 characters it will be truncated to 48 characters.

■ If PROC-ID is longer than 128 characters it will be truncated to 128 characters.

■ If MSGID is longer than 32 characters it will be truncated to 32 characters.

■ If HOSTNAME is longer than 255 characters it will be truncated to 255 characters.

2.10.2.3. The STRUCTURED-DATA message part

The STRUCTURED-DATA message part may contain meta- information about the syslog message, orapplication-specific information such as traffic counters or IP addresses. STRUCTURED-DATA consists ofdata blocks enclosed in brackets ([]). Every block includes the ID of the block, and one or more name=valuepairs. The syslog-ng application automatically parses the STRUCTURED-DATA part of syslog messages,which can be referenced in macros (for details, see Section 14.1.5, Macros of syslog-ng PE (p. 383)). An exampleSTRUCTURED-DATA block looks like:

23syslog-ng.com

IETF-syslog messages

[exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0

class="high"]

2.10.2.4. The MSG message part

The MSG part contains the text of the message itself. The encoding of the text must be UTF-8 if theBOMcharacteris present in the message. If the message does not contain the BOM character, the encoding is treated as unknown.Usually messages arriving from legacy sources do not include the BOM character. CRLF characters will notbe removed from the message.

2.11. Message representation in syslog-ng PE

When the syslog-ng PE application receives a message, it automatically parses the message. The syslog-ng PEapplication can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or theRFC5424 (IETF-syslog) message formats. If syslog-ng PE cannot parse a message, it results in an error.

TipIn case you need to relay messages that cannot be parsed without any modifications or changes, use the flags(no-parse)option in the source definition, and a template containing only the ${MSG} macro in the destination definition.

To parse non-syslog messages, for example, JSON, CSV, or other messages, you can use the built-in parsers of syslog-ngPE. For details, see Chapter 15, Parsing and segmenting structured messages (p. 407).

A parsed syslog message has the following parts.

■ Timestamps. Two timestamps are associated with every message: one is the timestamp containedwithin the message (that is, when the sender sent the message), the other is the time when syslog-ngPE has actually received the message.

■ Severity. The severity of the message.

■ Facility. The facility that sent the message.

■ Tags. Custom text labels added to the message that are mainly used for filtering. None of thecurrent message transport protocols adds tags to the log messages. Tags can be added to the logmessage only within syslog-ng PE. The syslog-ng PE application automatically adds the id of thesource as a tag to the incoming messages. Other tags can be added to the message by the patterndatabase, or using the tags() option of the source.

■ IP address of the sender. The IP address of the host that sent the message. Note that the IP addressof the sender is a hard macro and cannot be modified within syslog-ng PE but the associated hostnamecan be modified, for example, using rewrite rules.

■ Hardmacros. Hard macros contain data that is directly derived from the log message, for example,the ${MONTH} macro derives its value from the timestamp. The most important consideration withhard macros is that they are read-only, meaning they cannot be modified using rewrite rules or othermeans.

■ Soft macros. Soft macros (sometimes also called name-value pairs) are either built-in macrosautomatically generated from the log message (for example, ${HOST}), or custom user-created

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

24syslog-ng.com

Message representation in syslog-ng PE

macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields ofRFC5424-formatted log messages become soft macros as well. In contrast with hard macros, softmacros are writable and can be modified within syslog-ng PE, for example, using rewrite rules.

NoteIt is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST}macro from the message using a column of a CSV-parser.

The data extracted from the log messages using named pattern parsers in the pattern database are also softmacros.

TipFor the list of hard and soft macros, see Section 14.1.4, Hard vs. soft macros (p. 382).

Message size and encodingInternally, syslog-ng PE represents every message as UTF-8. The maximal length of the log messages is limitedby the log-msg-size() option: if a message is longer than this value, syslog-ng PE truncates the message atthe location it reaches the log-msg-size() value, and discards the rest of the message.

When encoding is set in a source (using the encoding() option) and the message is longer (in bytes) thanlog-msg-size() in UTF-8 representation, syslog-ng PE splits the message at an undefined location (becausethe conversion between different encodings is not trivial).

2.12. Structuring macros, metadata, and other value-pairs

Available in syslog-ng PE 5.1 and later.

The syslog-ng PE application allows you to select and construct name-value pairs from any information alreadyavailable about the log message, or extracted from the message itself. You can directly use this structuredinformation, for example, in the following places:

■ format-welf() template function

■ mongodb() destination

■ or in other destinations using the format-json() template function.

When using value-pairs, there are three ways to specify which information (that is, macros or othername-value pairs) to include in the selection.

■ Select groups of macros using the scope() parameter, and optionally remove certain macros fromthe group using the exclude() parameter.

■ List specific macros to include using the key() parameter.

25syslog-ng.com

Message size and encoding

■ Define new name-value pairs to include using the pair() parameter.

These parameters are detailed in Section value-pairs() (p. 27).

2.12.1. Specifying data types in value-pairs

By default, syslog-ng PE handles every data as strings. However, certain destinations and data formats (forexample, SQL, MongoDB, JSON) support other types of data as well, for example, numbers or dates. Thesyslog-ng PE application allows you to specify the data type in templates (this is also called type-hinting). Ifthe destination driver supports data types, it converts the incoming data to the specified data type. For example,this allows you to store integer numbers as numbers in MongoDB, instead of strings.

WarningHazard of data loss! If syslog-ng PE cannot convert the data into the specified type, an error occurs, and syslog-ng PEdrops the message by default. To change how syslog-ng PE handles data-conversion errors, see Section on-error() (p. 343).

To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"),for example: int("$PID").

Currently the mongodb() destination and the format-json template function supports data types.

Example 2.6. Using type-hintingThe following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDBdatabase. The DATE and PID parts are stored as numbers instead of strings.

mongodb(

value-pairs(pair("date", datetime("$UNIXTIME"))

pair("pid", int64("$PID"))

pair("program", "$PROGRAM"))

pair("message", "$MESSAGE"))

)

);

The following example formats the same fields into JSON.

$(format-json date=datetime("$UNIXTIME") pid=int64("$PID") program="$PROGRAM"

message="$MESSAGE")

The syslog-ng PE application currently supports the following data-types.

■ boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted totrue, anything that begins with an f or 0 is converted to false.

■ datetime: Use it only with UNIX timestamps, anything else will likely result in an error. Thismeans that currently you can use only the $UNIXTIME macro for this purpose.

■ literal: The data as a literal string, without adding any quotes or escape characters.

■ int or int32: 32-bit integer.

26syslog-ng.com

Specifying data types in value-pairs

■ int64: 64-bit integer.

■ string: The data as a string.

value-pairs()parameter list of the value-pairs() optionType:empty stringDefault:

Description: The value-pairs() option allows you to select specific information about a message easilyusing predefined macro groups. The selected information is represented as name-value pairs and can be usedformatted to JSON format, or directly used in a mongodb() destination.

Example 2.7. Using the value-pairs() optionThe following example selects every available information about the log message, except for the date-related macros(R_* and S_*), selects the .SDATA.meta.sequenceIdmacro, and defines a new value-pair called MSGHDR that containsthe program name and PID of the application that sent the log message.

value-pairs(

scope(nv_pairs core syslog all_macros selected_macros everything)

exclude("R_*")

exclude("S_*")

key(".SDATA.meta.sequenceId")

pair("MSGHDR" "$PROGRAM[$PID]: ")

)

The following example selects the same information as the previous example, but converts it into JSON format.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \

--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \

--pair MSGHDR="$PROGRAM[$PID]: ")

NoteEvery macro is included in the selection only once, but redundant information may appear if multiple macros include thesame information (for example, including several date-related macros in the selection).

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

1. scope()

2. exclude()

3. key()

4. pair()

5. subkeys()

27syslog-ng.com

value-pairs()

exclude()Space-separated list of macros to remove from the selection created using the scope() option.Type:empty stringDefault:

Description: This option removes the specified macros from the selection. Use it to remove unneeded macrosselected using the scope() parameter.

For example, the following example removes the SDATA macros from the selection.

value-pairs(

scope(rfc5424 selected_macros)

exclude(".SDATA*")

)

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()Space-separated list of macros to be included in selectionType:empty stringDefault:

Description: This option selects the specified macros. The selected macros will be included as MACRONAME= MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?)

to select multiple macros. For example:

value-pairs(

scope(rfc3164)

key("HOST"))

pair()name value pairs in "<NAME>" "<VALUE>" formatType:empty stringDefault:

Description: This option defines a new name-value pair to be included in the message. The value part caninclude macros, templates, and template functions as well. For example:

value-pairs(

scope(rfc3164)

pair("TIME" "$HOUR:$MIN")

pair("MSGHDR" "$PROGRAM[$PID]: "))

28syslog-ng.com

value-pairs()

rekey()<pattern-to-select-names>, <list of transformations>Type:empty stringDefault:

Description: This option allows you to manipulate and modify the name of the value-pairs. You can definetransformations, which are are applied to the selected name-value pairs. The first parameter of the rekey()option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformationsare applied to every key of the scope. For details on globs, see Section glob (p. 405).

■ If rekey() is used within a key() option, the name-value pairs specified in the glob of the key()option are transformed.

■ If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

add-prefix("<my-prefix>") Adds the specified prefix to every name. For example, rekey(add-prefix("my-prefix."))

replace-prefix("<prefix-to-replace>",

"<new-prefix>")

Replaces a substring at the beginning of the key with anotherstring. Only prefixes can be replaced. For example,replace-prefix(".class", ",patterndb") changesthe beginning tag .class to .patterndb

shift("<number>") Cuts the specified number of characters from the beginning ofthe name.

Example 2.8. Using the rekey() optionThe following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters fromthe names, and adds a new prefix (events.).

value-pairs(

key(".cee.*"

rekey(

shift(4)

add-prefix("events.")

)

)

)

The rekey() option can be used with the format-json template-function as well, using the following syntax:

$(format-json --rekey .cee.* --add-prefix events.)

29syslog-ng.com

value-pairs()

scope()space-separated list of macro groups to include in selectionType:empty stringDefault:

Description: This option selects predefined groups of macros. The following groups are available:

■ nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that startwith a dot (.) character. Macros starting with a dot character are generated within syslog-ng PE andare not originally part of the message, therefore are not included in this group.

■ dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with adot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with adot character are generated within syslog-ng PE and are not originally part of the message.

■ all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs anddot-nv-pairs.

■ rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format:$FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

■ rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY,$PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $DATE, and the metadata from thestructured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with.SDATA..

The rfc5424 group does not contain any metadata about the message, only information that waspresent in the original message. To include the most commonly used metadata (for example, the$SOURCEIP macro), use the selected-macros group instead.

■ all-macros: Include every hard macro. This group is mainly useful for debugging, as it containsredundant information (for example, the date-related macros include the date-related informationseveral times in various formats).

■ selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadataabout the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

■ sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, thatis, every macro that starts with .SDATA.

■ everything: Include every hard and soft macros. This group is mainly useful for debugging, as itcontains redundant information (for example, the date-related macros include the date-relatedinformation several times in various formats).

For example:

value-pairs(

scope(rfc3164 selected-macros))

30syslog-ng.com

value-pairs()

subkeys()name value pairs to selectType:empty stringDefault:

Description: This option selects every value-pair that has a name beginning with the specified prefix, butremoves the prefix when formatting the message. For example:

value-pairs(

scope(rfc3164)

subkeys(".cef.")

)

2.13. Things to consider when forwarding messages between syslog-ng PE hosts

When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server,you can use different protocols and options. Every combination has its advantages and disadvantages. The mostimportant thing is to use matching protocols and options, so the server handles the incoming log messagesproperly.

In syslog-ng PE you can change many aspects of the network communication. First of all, there is the structureof the messages itself. Currently, syslog-ng PE supports two standard syslog protocols: the BSD (RFC3164)and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing aroundthe messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng PE. There aretwo drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for thesyslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with thenetwork(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCPand UDP, and optionally also higher-level transport protocols: TLS (over TCP, and RLTP (optionally usingTLS). To complicate things a bit more, you can configure the network() driver (corresponding to the BSD(RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used inRFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and optionsas sources and as destinations:

1. syslog(transport(tcp))

2. syslog(transport(udp))

3. syslog(transport(rltp))

4. syslog(transport(tls))

31syslog-ng.com

Things to consider when forwarding messages between syslog-ng PE hosts

5. syslog(transport(rltp(tls-required(yes)))

6. network(transport(tcp))

7. network(transport(udp))

8. network(transport(rltp))

9. network(transport(tls))

10. network(transport(rltp(tls-required(yes)))

11. network(transport(tcp) flag(syslog-protocol))

12. network(transport(udp) flag(syslog-protocol))

13. network(transport(rltp)flag(syslog-protocol))

14. network(transport(tls) flag(syslog-protocol))

15. network(transport(rltp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your syslog-ng PE client and the source of yoursyslog-ng PE server, everything should work as expected. Unfortunately there are some other combinations,that seem to work, but result in losing parts of the messages. The following table show the combinations:

151413121110987654321S \ D----!----!----✔1

---!----!----✔-2

--!----!----✔--3

-!----!----✔---4

!----!----✔----5

----✔?----✔-----6

---✔?----✔----✔?-7

--✔?----✔----✔?--8

-✔?----✔--------9

✔?----✔----✔?----10

----✔----!----!11

---✔----!----!-12

--✔----!----!--13

-✔----!----!---14

32syslog-ng.com

Things to consider when forwarding messages between syslog-ng PE hosts

151413121110987654321S \ D✔----!----!----15

Table 2.6. Source-destination driver combinations

■ - This method does not work. The logs will not get to the server.

■ ✔ This method works.

■ ! This method has some visible drawbacks. The logs go through, but some of the values aremissing/misplaced/and so on.

■ ✔? This method seems to work, but it is not recommended because this can change in a future release.

The configuration options of the syslog-ng Agent for Windows application are much more limited. Theconfigurable elements are:

■ Protocol: (Legacy BSD Syslog Protocol, Syslog Protocol, Snare Protocol)

■ Use SSL option

■ Use syslog-ng proprietary Reliable Log Transfer Protocol (RLTP) option

The protocol corresponds to the network() and syslog() drivers of syslog-ng PE, except that the syslog-ngPE server does not support the Snare protocol.

The Use syslog-ng proprietary Reliable Log Transfer Protocol (RLTP) option is equivalent to thetransport(rltp) option.

The Use SSL option is the Windows equivalent of the transport(tls) option if RLTP is not set, and to thetransport(rltp(tls-required(yes)) otherwise.

Therefore with syslog-ng Agent the following cases are possible:

Corresponding syslog-ng PE server settingThe syslog-ng Agent for Windows settingsyslog(transport(tcp))Protocol: Syslog Protocolsyslog(transport(rltp))Protocol: Syslog Protocol, Use RLTP on

syslog(transport(tls))Protocol: Syslog Protocol, Use SSL on

syslog(transport(rltp(tls-required(yes)))Protocol: Syslog Protocol, Use RLTP on, Use SSLon

network(transport(tcp))Protocol: Legacy BSD Syslog Protocolnetwork(transport(rltp))Protocol: Legacy BSD Syslog Protocol, Use RLTP

on

network(transport(tls))Protocol: Legacy BSD Syslog Protocol, Use SSL on

33syslog-ng.com

Things to consider when forwarding messages between syslog-ng PE hosts

Corresponding syslog-ng PE server settingThe syslog-ng Agent for Windows settingnetwork(transport(rltp(tls-required(yes)))Protocol: Legacy BSD Syslog Protocol, Use RLTP

on, Use SSL onTable 2.7. Source-destination driver combinations in syslog-ng Agent for Windows

2.14. NFS file system for log files

Using the NFS network file system can lead to problems if NFS connection is not stable, therefore One Identitydoes neither recommend nor officially support such scenarios. If you can avoid it, do not store log files on NFS.If the NFS connection is stable and reliable, syslog-ng PE can read and write files on mounted NFS partitionsas a normal file source or destination. Read this section carefully before using syslog-ng PE and NFS-mountedlog files.

RisksIf there is any issue with the NFS connection (for example, connection loss, the NFS server stops), syslog-ngPE can stop working. These NFS issues can be related to the operating system, and can also vary dependingon its patch level and kernel version. The possible effects include the following:

■ syslog-ng PE freezes, does not respond, does not process logs, is unable to stop or reload, and youcan stop it only using the kill -9 command

■ syslog-ng PE is not able to start, and hangs during startup

■ Message loss or message duplication

■ Message becomes corrupt (it is not lost, but the message or some parts of it contain garbage)

■ When using the logstore() destination, the logstore file becomes corrupt

■ On some RHEL-based systems (possibly depending on the kernel version too), NFS returns NULLcharacters when reading a file that another process is writing at the very same moment.

Limitations of using syslog-ng PE with NFS■ Do not use the logstore() destination to store files on an NFS-mounted partition

■ To use wildcards in the file source if your log files are on an NFS file system, set theforce-directory-polling() option to yes to detect newly created files. Note that wildcard filesources are available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, andare not yet available in syslog-ng PE version 7.

■ Since One Identity does not officially support scenarios where you use syslog-ng PE together withNFS, One Identity will handle support requests and bugs related to such scenarios only if you canreproduce the issue independently from NFS.

Recommendations for using NFS with syslog-ng PEIf you cannot avoid using NFS with syslog-ng PE note the following points.

■ USE at least NFS v4 (or newer if available)

34syslog-ng.com

NFS file system for log files

■ USE the soft mount option (-o soft) to mount the partition

■ USE the TCP mount option (-o tcp) to mount the partition

■ DO NOT install syslog-ng PE on an NFS-mounted partition

■ DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PEon an NFS-mounted partition

■ DO NOT use logstore on an NFS-mounted partition, it can easily become corrupted

35syslog-ng.com

Recommendations for using NFS with syslog-ng PE

Chapter 3. Installing syslog-ng

This chapter explains how to install syslog-ng Premium Edition on the supported platforms using the precompiledbinary files.

■ The syslog-ng PE application features a unified installer package with identical look on everysupported Linux and UNIX platforms. The generic installer, as well as installing platform-specific(for example, RPM) is described in the following sections.

■ For details on installing syslog-ng PE on Microsoft Windows, see Procedure 3.8, Installing syslog-ngPE on Windows platforms (p. 51).

■ For details on installing the syslog-ng Agent for Windows application, see Chapter 2, Installing thesyslog-ng Agent in The syslog-ng Agent for Windows 6 LTS Administrator Guide.

■ For details on installing FIPS-compliant syslog-ng PE, see Chapter 11, FIPS-compliantsyslog-ng (p. 362).

■ If you want to manage your syslog-ng PE hosts usingPuppet, seeProcedure 3.10, Managing syslog-ngPE from Puppet (p. 58).

The syslog-ng PE binaries include all required libraries and dependencies of syslog-ng PE, only the ncurseslibrary is required as an external dependency (syslog-ng PE itself does not use the ncurses library, it is requiredonly during the installation). The components are installed into the /opt/syslog-ng directory. It canautomatically re-use existing configuration and license files, and also generate a simple configurationautomatically into the /opt/syslog-ng/etc/syslog-ng.conf file.

NoteThere are two versions of every binary release. The one with the compact suffix does not include SQL support. If youare installing syslog-ng PE in client or relay mode, or you do not use the sql() source or destination, use the compactbinaries. That way no unnecessary components are installed to your system.

The syslog-ng PE application can be installed interactively following the on-screen instructions as describedin Section 3.3, Installing syslog-ng using the .run installer (p. 37), and also without user interaction using thesilent installation option — see Section 3.3.3, Installing syslog-ng PE without user-interaction (p. 45).

3.1. Prerequisites to installing syslog-ng PE

■ The binary installer packages of syslog-ng Premium Edition include every required dependency formost platforms, only the ncurses library is required as an external dependency (syslog-ng PE itselfdoes not use the ncurses library, it is required only during the installation).

NoteThere are two versions of every binary release. The one with the compact suffix does not include SQLsupport. If you are installing syslog-ng PE in client or relay mode, or you do not use the sql() source ordestination, use the compact binaries. That way no unnecessary components are installed to your system.

36syslog-ng.com

Prerequisites to installing syslog-ng PE

■ For Java-based destinations (for example, Elasticsearch, Apache Kafka, HDFS), Java must be installedon the host where you use such destinations. Typically, this is the host where you are runningsyslog-ng PE in server mode.

■ DO NOT install syslog-ng PE on an NFS-mounted partition

■ DO NOT store the runtime files (for example, the configuration or the persist file) of syslog-ng PEon an NFS-mounted partition

The following platforms require the following patches for syslog-ng PE:

HP-UX

NoteTo install syslog-ng PE version 4 F1 on HP-UX (PARISC), the following patches must be installed on the host:PHCO_24402, PHCO_38279, PHKL_31918, PHSS_30049.

The patch kits are available at http://www.hp.com/ for customers with valid support contract.

Solaris

Generated .pkg packagesArchitectureOSclient, setupSPARCSolaris 10

client, setupSPARC64Solaris 10

client, setupamd64Solaris 10Table 3.1. Supported platforms and generated packages

3.2. Security-enhanced Linux: grsecurity, SELinux

Security-enhanced Linux solutions such as grsecurity or SELinux can interfere with the operation of syslog-ngPE. The syslog-ng PE application supports these security enhancements as follows:

■ grsecurity: Version syslog-ng PE 5 F2 and later can be run on hosts using grsecurity, with thefollowing limitations: using the Oracle SQL source and destination is not supported.

■ SELinux: Version syslog-ng PE 5 F2 and later properly supports SELinux on Red Hat EnterpriseLinux 6.5 and newer platforms. The CentOS platforms corresponding to the supported RHEL versionsare supported as well. For details, see Procedure 3.5, Using syslog-ng PE on SELinux (p. 47).

3.3. Installing syslog-ng using the .run installer

WarningIf you already had syslog-ng Open Source Edition (OSE) installed on the host, and are upgrading to syslog-ng PremiumEdition, make sure that the ${SYSLOGNG_OPTIONS} environmental variable does not contain a -p <path-to-pid-file>

option. If it does, remove this option from the environmental variable, because it can prevent syslog-ng PE from stoppingproperly. Typically, the environmental variable is set in the files /etc/default/syslog-ng or/etc/sysconfig/syslog-ng, depending on the operating system you use.

37syslog-ng.com

HP-UX

This section describes how to install the syslog-ng PE application interactively using the binary installer. Theinstaller has a simple interface: use the TAB or the arrow keys of your keyboard to navigate between the options,and Enter to select an option.

■ To install syslog-ng PE on clients or relays, complete Procedure 3.3.1, Installing syslog-ng PE inclient or relay mode (p. 38).

■ To install syslog-ng PE on your central log server, complete Procedure 3.3.2, Installing syslog-ngPE in server mode (p. 41).

■ To install syslog-ng PE without any user-interaction, complete Section 3.3.3, Installing syslog-ngPE without user-interaction (p. 45).

NoteThe installer stops the running syslogd application if it is running, but its components are not removed. The/etc/init.d/sysklogd init script is automatically renamed to /etc/init.d/sysklogd.backup. Rename this fileto its original name if you want to remove syslog-ng or restart the syslogd package.

3.3.1. Procedure – Installing syslog-ng PE in client or relay mode

Purpose:

Complete the following steps to install syslog-ng Premium Edition on clients or relays. For details on thedifferent operation modes of syslog-ng PE, see Section 2.3, Modes of operation (p. 10).

Steps:

NoteThe native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to writeinto files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file tosyslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng'sinitscript.

Step 1. Login to MyDownloads and download the syslog-ng PE installer package.

Step 2. Enable the executable attribute for the installer using the chmod +x

syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root usingthe ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that theexact name of the file depends on the operating system and platform.) Wait until the package isuncompressed and the welcome screen appears, then select Continue.

Figure 3.1. The welcome screen

38syslog-ng.com

Installing syslog-ng using the .run installer

Step 3. Accepting the EULA: You can install syslog-ng PE only if you understand and accept the terms of theEnd-User License Agreement (EULA). The full text of the EULA can be displayed during installationby selecting the Show EULA option, and is also available in this guide for convenience at AppendixB, END USER LICENSE AGREEMENT FOR BALABIT PRODUCT (EULA) (p. 513). Select Acceptto accept the EULA and continue the installation.If you do not accept the terms of the EULA for some reason, selectReject to cancel installing syslog-ngPE.

Step 4. Detecting platform and operating system: The installer attempts to automatically detect your opratingsystem and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abortthe installation, and verify that your platform is supported. For a list of supported platforms, see Section1.6, Supported platforms (p. 4). If your platform is supported but not detected correctly, contact yourlocal distributor, reseller, or the BalaBit Support Team. For contact details, see Section 5, Contact andsupport information (p. xii).

Figure 3.2. Platform detection

Step 5. Installation path: Enter the path to install syslog-ng PE to. This is useful if you intend to install syslog-ngPE without registering it as a service, or if it cannot be installed to the default location because ofpolicy compliance reasons. If no path is given, syslog-ng PE is installed to the default folder.

Figure 3.3. Installation path

NoteWhen installing syslog-ng PE to an alternative path on AIX, HP-UX, or Solaris platforms, set theCHARSETALIASDIR environmental variable to the lib subdirectory of the installation path. That way syslog-ngPE can find the charset.alias file.

Step 6. Registering as syslog service: Select Register to register syslog-ng PE as the syslog service. This willstop and disable the default syslog service of the system.

39syslog-ng.com

Installing syslog-ng using the .run installer

Figure 3.4. Registering as syslog service

Step 7. Locating the license: Since you are installing syslog-ng PE in client or relay mode, simply select OK.For details on the different operation modes of syslog-ng PE, see Section 2.3, Modes ofoperation (p. 10).

Step 8. Upgrading: The syslog-ng PE installer can automatically detect if you have previously installed aversion of syslog-ng PE on your system. To use the configuration file of this previous installation,select Yes. To ignore the old configuration file and create a new one, select No.Note that if you decide to use your existing configuration file, the installer automatically checks it forsyntax error and displays a list of warnings and errors if it finds any problems.

Figure 3.5. Upgrading syslog-ng

Step 9. Generating a new configuration file: The installer displays some questions to generate a newconfiguration file.

Step a. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, andSYSLOG messages on every interface will be automatically accepted.

Figure 3.6. Accepting remote messages

Step b. Remote destinations: Enter the IP address or hostname of your log server or relay andselect OK.

40syslog-ng.com

Installing syslog-ng using the .run installer

Figure 3.7. Forwarding messages to the log server

NoteAccepting remote messages and forwarding them to a log server means that syslog-ng PE will start in relaymode.

Step 10. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin

directories to your search PATH environment variable. That way you can use syslog-ng PE and itsrelated tools without having to specify the full pathname. Add the following line to your shell profile:

PATH=/opt/syslog-ng/bin:$PATH

Step 11. Optional step for SELinux-enabled systems: Complete Procedure 3.5, Using syslog-ng PE onSELinux (p. 47).

3.3.2. Procedure – Installing syslog-ng PE in server mode

Purpose:

Complete the following steps to install syslog-ng PE on log servers. For details on the different operation modesof syslog-ng PE, see Section 2.3, Modes of operation (p. 10).

Steps:

NoteThe native logrotation tools do not send a SIGHUP to syslog-ng after rotating the log files, causing syslog-ng to writeinto files already rotated. To solve this problem, the syslog-ng init script links the /var/run/syslog.pid file tosyslog-ng's pid. Also, on Linux, the install.sh script symlinks the initscript of the original syslog daemon to syslog-ng'sinitscript.

Step 1. Login toMyDownloads and download the syslog-ng PE installer package and your syslog-ng PremiumEdition license file (license.txt). The license will be required to run syslog-ng PE in server mode(see Section 2.3.3, Server mode (p. 11)) and is needed when you are installing syslog-ng PE on yourcentral log server.

Step 2. Enable the executable attribute for the installer using the chmod +x

syslog-ng-<edition>-<version>-<OS>-<platform>.run, then start the installer as root usingthe ./syslog-ng-<edition>-<version>-<OS>-<platform>.run command. (Note that the

41syslog-ng.com

Installing syslog-ng using the .run installer

exact name of the file depends on the operating system and platform.) Wait until the package isuncompressed and the welcome screen appears, then select Continue.

Figure 3.8. The welcome screen

Step 3. Accepting the EULA: You can install syslog-ng PE only if you understand and accept the terms of theEnd-User License Agreement (EULA). The full text of the EULA can be displayed during installationby selecting the Show EULA option, and is also available in this guide for convenience at AppendixB, END USER LICENSE AGREEMENT FOR BALABIT PRODUCT (EULA) (p. 513). Select Acceptto accept the EULA and continue the installation.If you do not accept the terms of the EULA for some reason, selectReject to cancel installing syslog-ngPE.

Step 4. Detecting platform and operating system: The installer attempts to automatically detect your opratingsystem and platform. If the displayed information is correct, select Yes. Otherwise select Exit to abortthe installation, and verify that your platform is supported. For a list of supported platforms, see Section1.6, Supported platforms (p. 4). If your platform is supported but not detected correctly, contact yourlocal distributor, reseller, or the BalaBit Support Team. For contact details, see Section 5, Contact andsupport information (p. xii).

Figure 3.9. Platform detection

Step 5. Installation path: Enter the path to install syslog-ng PE to. This is useful if you intend to install syslog-ngPE without registering it as a service, or if it cannot be installed to the default location because ofpolicy compliance reasons. If no path is given, syslog-ng PE is installed to the default folder.

Figure 3.10. Installation path

42syslog-ng.com

Installing syslog-ng using the .run installer

NoteWhen installing syslog-ng PE to an alternative path on AIX, HP-UX, or Solaris platforms, set theCHARSETALIASDIR environmental variable to the lib subdirectory of the installation path. That way syslog-ngPE can find the charset.alias file.

Step 6. Registering as syslog service: Select Register to register syslog-ng PE as the syslog service. This willstop and disable the default syslog service of the system.

Figure 3.11. Registering as syslog service

Step 7. Locating the license: Enter the path to your license file (license.txt) and select OK. Typically thisis required only for your central log server.If you are upgrading an existing configuration that already has a license file, the installer automaticallydetects it.

Figure 3.12. Platform detection

Step 8. Upgrading: The syslog-ng PE installer can automatically detect if you have previously installed aversion of syslog-ng PE on your system. To use the configuration file of this previous installation,select Yes. To ignore the old configuration file and create a new one, select No.Note that if you decide to use your existing configuration file, the installer automatically checks it forsyntax error and displays a list of warnings and errors if it finds any problems.

43syslog-ng.com

Installing syslog-ng using the .run installer

Figure 3.13. Upgrading syslog-ng

Step 9. Generating a new configuration file: The installer displays some questions to generate a newconfiguration file.

Step a. Remote sources: Select Yes to accept log messages from the network. TCP, UDP, andSYSLOG messages on every interface will be automatically accepted.

Figure 3.14. Accepting remote messages

Step b. Remote destinations: Enter the IP address or hostname of your log server or relay andselect OK.

Figure 3.15. Forwarding messages to the log server

NoteAccepting remote messages and forwarding them to a log server means that syslog-ng PE will start in relaymode.

Step 10. After the installation is finished, add the /opt/syslog-ng/bin and /opt/syslog-ng/sbin

directories to your search PATH environment variable. That way you can use syslog-ng PE and itsrelated tools without having to specify the full pathname. Add the following line to your shell profile:

44syslog-ng.com

Installing syslog-ng using the .run installer

PATH=/opt/syslog-ng/bin:$PATH

Step 11. Optional step for SELinux-enabled systems: Complete Procedure 3.5, Using syslog-ng PE onSELinux (p. 47).

3.3.3. Installing syslog-ng PE without user-interaction

The syslog-ng PE application can be installed in silent mode without any user-interaction by specifying therequired parameters from the command line. Answers to every question of the installer can be set in advanceusing command-line parameters.

./syslog-ng-premium-edition-<version>.run -- --silent [options]

WarningThe -- characters between the executable and the parameters are mandatory, like in the following example:./syslog-ng-premium-edition-3.0.1b-solaris-10-sparc-client.run -- --silent --accept-eula

-l /var/tmp/license.txt

To display the list of parameters, execute the ./syslog-ng-premium-edition-<version>.run -- --h

command. Currently the following options are available:

■ --accept-eula or -a: Accept the EULA.

■ --license-file <file> or -l <file>: Path to the license file.

■ --upgrade | -u: Perform automatic upgrade — use the configuration file from an existing installation.

■ --remote <destination host>: Send logs to the specified remote server. Not available when performingan upgrade.

■ --network: Accept messages from the network. Not available when performing an upgrade.

■ --configuration <file>: Use the specified configuration file.

■ --list-installed: List information about all installed syslog-ngs.

■ --path <path>: Set installation path.

■ --register: Force service registration.

■ --no-register: Prevent service registration.

3.4. Procedure – Installing syslog-ng PE on RPM-based platforms (Red Hat, SUSE,AIX)

Purpose:

To install syslog-ng PE on operating systems that use the Red Hat Package Manager (RPM), complete thefollowing steps. Installing syslog-ng PE automatically replaces the original syslog service. The followingsupported operating systems use RPM:

■ CentOS

■ openSUSE

■ Red Hat Enterprise Linux

45syslog-ng.com

Installing syslog-ng PE without user-interaction

■ Red Hat Enterprise Server

■ SUSE Linux Enterprise Server

WarningIf you already had syslog-ng Open Source Edition (OSE) installed on the host, and are upgrading to syslog-ng PremiumEdition, make sure that the ${SYSLOGNG_OPTIONS} environmental variable does not contain a -p <path-to-pid-file>

option. If it does, remove this option from the environmental variable, because it can prevent syslog-ng PE from stoppingproperly. Typically, the environmental variable is set in the files /etc/default/syslog-ng or/etc/sysconfig/syslog-ng, depending on the operating system you use.

Steps:

Step 1. Login to MyDownloads and download the syslog-ng RPM package for your system.

Step 2. If the host already uses syslog-ng PE for logging, execute the following command as root.Otherwise, skip this step.

■

rpm -U syslog-ng-premium-edition-<version>-<OS>-<arch>.rpm

The syslog-ng Premium Edition application and all its dependencies will be installed, andthe configuration of the existing syslog-ng PE installation will be used.

NoteIf you are upgrading from syslog-ng version 2.1, note that the location of the configurationfile has been moved to /opt/syslog-ng/etc/syslog-ng.conf

■ Execute the following command as root:

rpm -i syslog-ng-premium-edition-<version>-<OS>-<arch>.rpm

The syslog-ng PE application and all its dependencies will be installed.

Step 3. WarningWhen performing an upgrade, the package manager might automatically execute the post-uninstall script ofthe upgraded package, stopping syslog-ng PE and starting syslogd. If this happens, stop syslogd and startsyslog-ng PE by issuing the following commands:

/etc/init.d/syslogd stop

/etc/init.d/syslog-ng start

This behavior has been detected on CentOS 4 systems, but may occur on other rpm-based platforms as well.

Step 4. Edit the syslog-ng PE configuration file as needed. If you want to run syslog-ng PE in server mode,copy the license file to the /opt/syslog-ng/etc/ directory.For information on configuring syslog-ng PE, see the Chapter 4, The syslog-ng PE quick-startguide (p. 68).

46syslog-ng.com

Installing syslog-ng PE without user-interaction

Step 5. Optional step for SELinux-enabled systems: Complete Procedure 3.5, Using syslog-ng PE onSELinux (p. 47).

Step 6. Optional step for AIX systems: To redirect the messages of the AIX Error log into syslog, create a file(for example /tmp/syslog-ng.add) with the following contents:

errnotify:

en_name = "syslog1"

en_persistenceflg = 1

en_method = "logger Msg from Error Log: `errpt -l $1 | grep -v 'ERROR_ID

TIMESTAMP'`"

Then execute the following command as root: odmadd /tmp/syslog-ng.add.

3.5. Procedure – Using syslog-ng PE on SELinux

Purpose:

Version syslog-ng PE 5 F2 and later properly supports SELinux on Red Hat Enterprise Linux 6.5 and newerplatforms. Version 5 F5 and later also supports SELinux on Red Hat Enterprise Linux 5, as well as on 6.0-6.4.The CentOS platforms corresponding to the supported RHEL versions, and Oracle Linux 7 are supported aswell. To use syslog-ng PE on a SELinux-enabled host, complete the following steps.

NoteThe following steps install SELinux policy module that enables syslog-ng PE to properly run with its default configurationand default installation path (/opt/syslog-ng) on a SELinux-enabled host. If you configure syslog-ng PE to performan operation that is outside the permissions of this policy module (for example, to bind to a non-standard port, use aprogram destination or source, or to write logfiles in a non-standard directory), you have to modify and recompile thepolicy module. If you need help with that, contact the BalaBit Support Team.

Prerequisites:

■ T h e f o l l o w i n g p a c k a g e s m u s t b e a v a i l a b l e o n t h e h o s t :policycoreutils,policycoreutils-devel, policycoreutils-python. If they are notalready installed, issue the following command: yum install policycoreutils

policycoreutils-devel policycoreutils-python

■ On RHEL 6.5, update the following packages at least to the indicated versions. These packages areavailable in the Red Hat repositories and are installed by default on RHEL 6.6. You can update themwith the yum update selinux-policy command.

• selinux-policy-3.7.19-231.el6.noarch > 3.7.19-260.el6.noarch

• selinux-policy-targeted-3.7.19-231.el6.noarch > 3.7.19-260.el6.noarch

■ The syslog-ng PE application must be installed on the host. For details, see Chapter 3, Installingsyslog-ng (p. 36).

Steps:

Expected result:

47syslog-ng.com

Installing syslog-ng PE without user-interaction

The syslog-ng PE application is installed and properly running under SELinux. If syslog-ng PE does not start,or displays permission errors, execute the syslog_ng.sh.

Step 1. D o w n l o a d t h e s y s l o g - n g P E S E L i n u x p o l i c y m o d u l e f r o mhttps://my.balabit.com/downloads/syslog-ng/syslog-ng-pe-5F2_and_higher-SELinux

Step 2. Uncompress the file and run the ./syslog_ng.sh script to compile and load the SELinux rules forsyslog-ng PE.

Step 3. Restart syslog-ng PE using the following command.

WarningThe SELinux policy works only if syslog-ng PE is started by the init daemon.

■ On RHEL6: service syslog-ng restart

■ On RHEL7: systemctl restart syslog-ng

If you do not use the service or the systemctl to start syslog-ng PE execute the syslog_ng.shscript again after starting syslog-ng PE. This is required to correct the settings of the files related tosyslog-ng PE (most notably /dev/log and the files under /opt/syslog-ng). The settings can becomeincorrect if the privileges of the process that started syslog-ng PE are different from the privileges ofthe service or the systemctl process.

Step 4. Optional Step: The syslog-ng PE application can create coredumps, but this is disabled by default.You can enable coredumps with the setsebool -P daemons_dump_core 1 command.Note that his command enables every daemons on your system to create core dumps, not just syslog-ngPE. There is no way to enable per-application core dumps in SELinux.

3.6. Procedure – Installing syslog-ng on Debian-based platforms

Purpose:

To install syslog-ng on operating systems that use the Debian Software Package (deb) format, complete thefollowing steps. The following supported operating systems use this format:

■ Debian Wheezy

■ Debian Jessie

WarningIf you already had syslog-ng Open Source Edition (OSE) installed on the host, and are upgrading to syslog-ng PremiumEdition, make sure that the ${SYSLOGNG_OPTIONS} environmental variable does not contain a -p <path-to-pid-file>

option. If it does, remove this option from the environmental variable, because it can prevent syslog-ng PE from stoppingproperly. Typically, the environmental variable is set in the files /etc/default/syslog-ng or/etc/sysconfig/syslog-ng, depending on the operating system you use.

Steps:

48syslog-ng.com

Installing syslog-ng PE without user-interaction

Step 1. Login to MyDownloads and download the syslog-ng PE DEB package for your system.

Step 2. Issue the following command as root:dpkg -i syslog-ng-premium-edition-<version>-<OS>-<arch>.deb

Step 3. Answer the configuration questions of syslog-ng PE. These are described in detail in Section 3.3,Installing syslog-ng using the .run installer (p. 37).For information on configuring syslog-ng PE, see the Chapter 4, The syslog-ng PE quick-startguide (p. 68).

3.7. Installing syslog-ng PE using .pkg installer

The syslog-ng PE application can be installed with user-interaction or in silent mode.

The generic syslog service of the operating system is stopped and deregistered automatically by the installerbefore installation.

WarningRemove any other syslog service before starting the installation. Failing to do so might have unpredictable results.

The syslog-ng PE application is compatible with the standard Solaris logrotate mechanism by default.

■ To install syslog-ng PE with user-interaction, complete Procedure 3.7.1, Installing syslog-ng PEwith user-interaction (p. 49).

■ To install syslog-ng PE without any user-interaction, complete Procedure 3.7.2, Installing syslog-ngPE without user-interaction (p. 50).

■ To install syslog-ng PE from a transformed PKG package, complete Procedure 3.7.3, Installingsyslog-ng PE from a transformed PKG package (p. 51).

3.7.1. Procedure – Installing syslog-ng PE with user-interaction

Purpose:

The syslog-ng PE application can be installed in an interactive way.

Steps:

Step 1. Unpack the package with the following command:

gunzip syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg.gz

Step 2. Run the following command:

pkgadd -d syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

49syslog-ng.com

Installing syslog-ng PE using .pkg installer

Step 3. Answer the questions with y or n. Besides these, the following commands are available: ? displaysthe help, q exits the installer. The installer generates a default configuration file for syslog-ng PE basedon the answers (if an old configuration file is not used):

■ the path of the license file

■ whether an old configuration file is to be used (if the answer is yes, the rest of the questionswill be skipped)

■ if a new configuration file is requested, whether syslog-ng PE will send or receive logs overthe network

Step 4. The installer finishes the installation and the replaces the default syslog service with syslog-ng PE.

3.7.2. Procedure – Installing syslog-ng PE without user-interaction

Purpose:

The syslog-ng PE application can be installed in silent mode without any user-interaction by specifying therequired parameters from the command line. Answers to every question of the installer can be set in advanceusing command-line parameters.

Steps:

Step 1. Generate a response file that contains the answers to the questions.

pkgask -r <responsefile> -d

syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

For example:

pkgask -r response_file -d syslog-ng-premium-edition-4.2.1-solaris10-amd64.pkg

BBsyslng

WarningThe pkgask will fail if the response file already exists.

Step 2. Run the pkgadd command with the added response file answering y to all questions.

yes | pkgadd -r <responsefile> -d

syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg BBsyslng

Warningpkgask runs the request script of the package under the current condition of the operating system. Theinstallation might fail if something changes (another syslog-ng PE package has been installed, configurationfile has been removed), because the response file's contents will be invalid.

50syslog-ng.com

Installing syslog-ng PE using .pkg installer

3.7.3. Procedure – Installing syslog-ng PE from a transformed PKG package

Purpose:

The syslog-ng PE application can be installed from a normal file structure. To perform this, transform the .pkgpackage.

Steps:

Step 1. To transform the .pkg package, execute the following command:

pkgtrans syslog-ng-premium-edition-<version>-<OS>-<architecture>.pkg

<outdirectory>

In this case, the .pkg package will be extracted to the <outdirectory> directory.

Example 3.1. Extracting syslog-ng PE from a transformed PKG package

pkgtrans syslog-ng-premium-edition-4.2.0-solaris-9-sparc.pkg /tmp/out

This will extract the files to the /tmp/out/BBsyslng directory.

Step 2. To install syslog-ng PE, execute the following command:

pkgadd <options> -d <outdirectory> BBsyslng

In this case, <options> stand for the pkgadd options, for example -r <responsefile>.

3.8. Procedure – Installing syslog-ng PE on Windows platforms

Purpose:

Complete the following steps to install syslog-ng PE in server mode on Microsoft Windows platforms. Fordetails on the different operation modes of syslog-ng PE, see Section 2.3, Modes of operation (p. 10).

Steps:

Step 1. Login to MyDownloads and download the syslog-ng installer package and your syslog-ng PremiumEdition license. The license will be required to run syslog-ng in server mode (see Section 2.3.3, Servermode (p. 11)) and is needed when you are installing syslog-ng on your central log server.

Step 2. Start the installer. The Welcome screen is displayed. Click Next.

51syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.16. The Welcome screen of the syslog-ng PE installer on Microsoft Windows

Step 3. Accepting the EULA: You can install syslog-ng PE only if you understand and accept the terms of theEnd-User License Agreement (EULA). The full text of the EULA can be displayed during installationby selecting the Show EULA option, and is also available in this guide for convenience at AppendixB, END USER LICENSE AGREEMENT FOR BALABIT PRODUCT (EULA) (p. 513). Select Acceptto accept the EULA and continue the installation.If you do not accept the terms of the EULA for some reason, selectReject to cancel installing syslog-ngPE.

Step 4. Select Install syslog-ng Premium Edition and click Next.

NoteIf you do not want to install syslog-ng PE, just copy its files to a host so that you can start it manually withoutregistering it as a service, select Unpack files without installation.

52syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.17. Select how to install syslog-ng PE on Microsoft Windows

Step 5. Installation path: Enter the path to install syslog-ng PE to. This is useful if you intend to install syslog-ngPE without registering it as a service, or if it cannot be installed to the default location because ofpolicy compliance reasons. If no path is given, syslog-ng PE is installed to the default folder.

53syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.18. Select the installation folder for syslog-ng PE on Microsoft Windows

Step 6. Locating the license: Enter the path to your license file and click Next.If you are upgrading an existing configuration that already has a license file, the installer automaticallydetects it.

WarningThe syslog-ng PE application will not use the specified file directly, but copy it under the %INSTALLDIR%\etc\folder.

54syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.19. Selecting the license file on Microsoft Windows

Step 7. Locating the configuration file: Enter the path to your configuration file and click Next.If you are upgrading an existing configuration that already has a configuration file, the installerautomatically detects it.

WarningThe syslog-ng PE application will not use the specified file directly, but copy it to%INSTALLDIR%\etc\syslog-ng.conf. To modify the configuration of syslog-ng PE, edit the%INSTALLDIR%\etc\syslog-ng.conf file. By default, syslog-ng PE is installed into the C:\Program

Files\syslog-ng directory.

55syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.20. Selecting the configuration file on Microsoft Windows

Step 8. Select Register as a service.

56syslog-ng.com

Installing syslog-ng PE using .pkg installer

Figure 3.21. Registering and starting the syslog-ng PE service

Step 9. Select Start the syslog-ng PE service and click Install.Expected result:

The syslog-ng PE application is installed and the service is started.

3.9. Installing syslog-ng without user-interaction on Windows

The syslog-ng Premium Edition application can be installed in silent mode as well, without requiring any userinteraction. The various installer options can be specified as command-line options. Using the /S option isrequired. The following options are available:

/A Start the syslog-ng PE service (/A=yes) when the installationis finished. The syslog-ng PE service will start only if a validconfiguration file is installed as well.

NoteOnly one syslog-ng can be installed as service.

57syslog-ng.com

Installing syslog-ng without user-interaction on Windows

/D=<path> Install the syslog-ng PE into the specified folder(%INSTALLDIR%).

WarningIf you use the /D option, make sure that this is the lastoption in the command-line. For example:syslog-ng-<version>-setup.exe /S

/D=c:\syslog-ng\

/F=<path> Enter the path to the syslog-ng PE configuration file. The installerwill copy the specified configuration file to%INSTALLDIR%\etc\syslog-ng.conf.

WarningThe syslog-ng PE application will not use the specifiedf i l e d i r e c t l y , b u t c o p y i t t o%INSTALLDIR%\etc\syslog-ng.conf. To modifythe configuration of syslog-ng PE, edit the%INSTALLDIR%\etc\syslog-ng.conf file. By default,syslog-ng PE is installed into the C:\Program

Files\syslog-ng directory.

/L=<path> Enter the path to the license file. The installer will copy thespecified license file to %INSTALLDIR%\etc\license.txt.

/M Add entries about syslog-ng PE to the Start menu.

/P Only unpack syslog-ng PE into the specified destination folder.If this option is enabled, (/P=yes), the installer will ignore otheroptions (except for /D that specifies the destination folderh).

/R Register the syslog-ng PE service (/R=yes).

/S Start the installer in silent mode. This option is required for thesilent installation.

3.10. Procedure – Managing syslog-ng PE from Puppet

Purpose:

To simplify the management of large-scale syslog-ng PE deployments, you can centrally manage your syslog-ngPE hosts from Puppet. The syslog-ng Premium Edition Puppet module (syslog_ng) allows you to performthe following tasks.

■ Install syslog-ng PE from a package repository.

■ Upgrade syslog-ng PE to a newer version.

■ Delete syslog-ng PE from a host.

■ Update the syslog-ng PE configuration file of your hosts from a central repository.

58syslog-ng.com

Installing syslog-ng without user-interaction on Windows

■ Create backup of your syslog-ng PE configuration files. You can redistribute these backups to yourhosts if a rollback is needed.

The Puppet module supports the following platforms: Red Hat Enterprise Linux (RHEL), Oracle Linux, CentOS,Ubuntu, and Debian. Other Linux platforms based on .deb and .rpm packages might also work, but are nottested.

To manage your syslog-ng PE clients from Puppet, complete the following steps.

Prerequisites:

To use the syslog_ng Puppet module, the following prerequisites must be met.

■ A Puppet architecture installed and configured

■ The Puppet master and slaves nodes are configured and connected

■ The filebucket feature of Puppet is enabled and configured

■ Package repositories are added to the slave nodes (where syslog-ng PE can be installed from)

Steps:

Step 1. Login to MyDownloads and download the syslog-ng PE installer package for your platform.

Step 2. Add the syslog-ng PE installer package to your local package repositories.

Step 3. Download the syslog_ng Puppet module from the Balabit website.

Step 4. Copy the module to your Puppet master, then build and install it. For details, see the Readme.md fileof the module.

3.11. Upgrading syslog-ng PE

This section describes the possible upgrade paths of syslog-ng PE.

■ To see the possible upgrade paths, see Section 3.11.1, Upgrading from previous syslog-ng PE versionsto 6 LTS (p. 59).

■ To upgrade an existing syslog-ng PE installation, see Procedure 3.11.2, Upgrading to syslog-ng PE6 LTS (p. 59).

3.11.1. Upgrading from previous syslog-ng PE versions to 6 LTS

Upgrading is supported from the following syslog-ng PE versions:

■ syslog-ng PE 5 LTS (5.0.x)

■ syslog-ng PE 5 F6 (5.6.x)

To upgrade an existing syslog-ng PE installation, see Procedure 3.11.2, Upgrading to syslog-ng PE 6 LTS (p. 59).

3.11.2. Procedure – Upgrading to syslog-ng PE 6 LTS

Purpose:

To upgrade to syslog-ng PE 6 LTS, complete the following steps:

59syslog-ng.com

Upgrading syslog-ng PE

Steps:

Step 1. Download the new installer package. Use the same package type as you used for the installation (forexample, use the .run package for the upgrade if you have originally installed syslog-ng PE using a.run installer).

Step 2. Install syslog-ng PE and check the warnings. Upgrade the respective parts of your configuration ifneeded.

Step 3. Set the version of the configuration file to 6.0.

3.11.3. Upgrading syslog-ng PE to other package versions

This scenario is not supported and will fail with the following error messages.

Upgrading from platform-specific package to .runUpgrading from rpm package to .run package.

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed

Upgrading from deb package to .run package.

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed

Upgrading from pkg package to .run package.

Unsupported. Installation stops and the following error message is displayed:

Incompatible syslog-ng package already installed to <syslog-ng path>

Upgrading from .run to a platform-specific packageThis scenario is not supported and will fail with the following error messages. To replace a .run package witha platform-specific package, create a backup of your configuration and persist files, uninstall the .run packageusing the --purge option, then install the platform-specific package.

Upgrading from .run package to rpm package.

Unsupported. Installation stops and the following error message is displayed:

Incompatible standalone (.run) installer of syslog-ng Premium Edition

WarningHazard of data loss! Installing rpm package syslog-ng PE on AIX platform is possible even if the upgrade conditions arenot met, since the rpm package installs before checking the upgrade conditions and therefore no error message is displayed.This might result in overwriting the old configuration file.

Upgrading from .run package to deb package.

60syslog-ng.com

Upgrading syslog-ng PE to other package versions

Unsupported. Installation stops and the following error message is displayed:

Errors were encountered while processing

Upgrading from .run package to pkg package.

Unsupported. Installation stops and the following error message is displayed:

Please remove the conflicting package before installing this package. Installation

aborted.

3.11.4. Upgrading from syslog-ng PE to syslog-ng OSE

Upgrading from syslog-ng PE to syslog-ng OSE is unsupported since it counts as downgrading.

3.11.5. Upgrading from complete syslog-ng PE to client setup version of syslog-ng PE

The installer displays the following message if you try to upgrade from complete syslog-ng PE to client setupsyslog-ng PE with .run package.

This version of syslog-ng Premium Edition doesn't support storing messages in SQL

servers, while the installed one did.

3.12. Uninstalling syslog-ng PE

If you need to uninstall syslog-ng PE for some reason, you have the following options:

■ If you have installed syslog-ng PE using the .run installer: Execute the uninstall.sh script locatedat /opt/syslog-ng/bin/uninstall.sh. The uninstall script will automatically restore the syslogdaemon used before installing syslog-ng. To completely remove syslog-ng PE, including theconfiguration files, use the uninstall.sh --purge command.

■ If you have installed syslog-ng PE from a .deb package: Execute the dpkg -r

syslog-ng-premium-edition command to remove syslog-ng, or the dpkg -P

syslog-ng-premium-edition command to remove syslog-ng PE and the configuration files aswell. Note that removing syslog-ng PE does not restore the syslog daemon used before syslog-ng.

■ If you have installed syslog-ng PE from an .rpm package: Execute the rpm -e

syslog-ng-premium-edition command to remove syslog-ng PE. Note that removing syslog-ngPE does not restore the syslog daemon used before syslog-ng PE.

■ If you have installed syslog-ng PE from a .pkg package: Execute the pkgrm BBsyslng commandto remove syslog-ng PE. Note that removing syslog-ng PE does not restore the syslog daemon usedbefore syslog-ng.For automatic uninstall (answering y to all questions): Execute the yes | pkgrm BBsyslng

command.

The following files have to be deleted manually:

• <syslog-ng path>/etc/syslog-ng.conf

• <syslog-ng path>/var/syslog-ng.persist

61syslog-ng.com

Upgrading from syslog-ng PE to syslog-ng OSE

• <syslog-ng path>/var/syslog-ng-00000.qf

• anything else under the <syslog-ng path>/var directory

■ On Microsoft Windows, run the uninstall.exe file from the installation directory of syslog-ngPE.To uninstall syslog-ng PE without user interaction, use the /S option. By default, the uninstallerdoes not delete the configuration and license files and the status information (the etc and var

folders). To delete every file, use the /C=yes option, that will delete the configuration file as well:uninstall.exe /S /C=yes

3.13. Procedure – Configuring Microsoft SQL Server to accept logs from syslog-ng

Purpose:

Complete the following steps to configure your Microsoft SQL Server to enable remote logins and accept logmessages from syslog-ng.

Steps:

Step 1. Start the SQL Server Management Studio application. Select Start > Programs > Microsoft SQLServer 2005 > SQL Server Management Studio.

Step 2. Create a new database.

Step a. Figure 3.22. Creating a new MSSQL database 1.

In the Object Explorer, right-click on the Databases entry and select New Database.

62syslog-ng.com

Uninstalling syslog-ng PE

Step b. Figure 3.23. Creating a new MSSQL database 2.

Enter the name of the new database (for example syslogng) into the Database namefield and click OK.

Step 3. Create a new database user and associate it with the new database.

63syslog-ng.com

Uninstalling syslog-ng PE

Step a. Figure 3.24. Creating a new MSSQL user 1.

In the Object Explorer, select Security, right-click on the Logins entry, then selectNew Login.

64syslog-ng.com

Uninstalling syslog-ng PE

Step b. Figure 3.25. Creating a new MSSQL user 2.

Enter a name (for example syslog-ng) for the user into the Login name field.

Step c. Select the SQL Server Authentication option and enter a password for the user.

Step d. In the Default database field, select the database created in Step 2 (for examplesyslogng).

Step e. In the Default language field, select the language of log messages that you want tostore in the database, then click OK.

WarningIncorrect language settings may result in the database converting the messages to adifferent character-encoding format. That way the log messages may become unreadable,causing information loss.

Step f. In the Object Explorer, select Security > Logins, then right-click on the new logincreated in the previous step, and select Properties.

65syslog-ng.com

Uninstalling syslog-ng PE

Step g. Figure 3.26. Associating database with the new user

Select User Mapping. In the Users mapped to this login option, check the linecorresponding to the new login (for example syslogng). In the Database rolemembership field, check the db_owner and public options.

66syslog-ng.com

Uninstalling syslog-ng PE

Step 4. Figure 3.27. Associating database with the new user

Enable remote logins for SQL users.

In the Object Explorer right-click on your database server, and select Properties > Security, and setthe Server Authentication option to SQL Server and Windows Authentication mode.

67syslog-ng.com

Uninstalling syslog-ng PE

Chapter 4. The syslog-ng PE quick-start guide

This chapter provides a very brief introduction into configuring the syslog-ng PE application. For details onthe format of the configuration file and how to configure sources, destinations, and other features, refer to thesubsequent chapters.

■ To configure syslog-ng PE as a client that sends log messages to a central log server, see Procedure4.1, Configuring syslog-ng on client hosts (p. 68).

■ To configure syslog-ng PE as a server that receives log messages from client hosts, see Procedure4.2, Configuring syslog-ng on server hosts (p. 70).

■ To configure syslog-ng PE as a relay that receives log messages from client hosts and forwards themto a central log server, see Procedure 4.2, Configuring syslog-ng on server hosts (p. 70).

4.1. Procedure – Configuring syslog-ng on client hosts

Purpose:

To configure syslog-ng on a client host, complete the following steps.

Steps:

Step 1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operatingsystems, see Chapter 3, Installing syslog-ng (p. 36).

Step 2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, syslog-ngPE automatically collects the log messages that use the native system logging method of the platform,for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list ofmessages that are collected automatically, see Section 6.11, Collecting the system-specific log messagesof a platform (p. 151).

Add sources to collect the messages from your log files. File sources look like this:

source s_myfilesource {

file("/var/log/myapplication.log" follow-freq(1)); };

Name every source uniquely. For details on configuring file sources, see Section 6.4, Collectingmessages from text files (p. 92).

TipMany applications send log messages to logfiles by default (for example, the Roundcube webmail client, orthe ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommendedto reconfigure the application that way.

NoteThe default configuration file of syslog-ng PE collects platform-specific log messages and the internal logmessages of syslog-ng PE.

68syslog-ng.com

source s_local {

system();

internal();

};

Step 3. Create a network destination that points directly to the syslog-ng server, or to a local relay. The networkdestination greatly depends on the protocol that your log server or relay accepts messages. Manysystems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

destination d_network { network("10.1.2.3" transport("udp")); };

However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

destination d_network { syslog("10.1.2.3" transport("tcp")); };

Step 4. Create a log statement connecting the local sources to the syslog-ng server or relay. For example:

log {

source(s_local); destination(d_network); };

Step 5. If the logs will also be stored locally on the host, create local file destinations.

NoteThe default configuration of syslog-ng PE places the collected messages into the /var/log/messages file:

destination d_local {

file("/var/log/messages"); };

Step 6. Create a log statement connecting the local sources to the file destination.

NoteThe default configuration of syslog-ng PE has only one log statement:

log {

source(s_local); destination(d_local); };

Step 7. Set filters, macros and other features and options (for example TLS encryption) as necessary.

Example 4.1. The default configuration file of syslog-ng PEThe following is a simple configuration file that collects local log messages to the /var/log/messages

file.

@version: 6.0

@include "scl.conf"

source s_local { system(); internal(); };

destination d_local {

file("/var/log/messages"); };

log { source(s_local); destination(d_local); };

69syslog-ng.com

Example 4.2. A simple configuration for clientsThe following configuration file collects local log messages and the log messages of syslog-ng PE, andforwards them to a log server using the IETF-syslog protocol.

@version: 6.0

@include "scl.conf"

source s_local { system(); internal(); };

destination d_syslog_tcp {

syslog("192.168.1.1" transport("tcp") port(2010)); };

log { source(s_local);destination(d_syslog_tcp); };

If you experience difficulties, see Chapter 19, Troubleshooting syslog-ng (p. 454) for tips on solvingcommon problems.

4.2. Procedure – Configuring syslog-ng on server hosts

Purpose:

To configure syslog-ng on a server host, complete the following steps.

Steps:

Step 1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operatingsystems, see Chapter 3, Installing syslog-ng (p. 36).

Step 2. Starting with version 3.2, syslog-ng PE automatically collects the log messages that use the nativesystem logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klogon FreeBSD. For a complete list of messages that are collected automatically, see Section 6.11,Collecting the system-specific log messages of a platform (p. 151).

Step 3. Configure the network sources that collect the log messages sent by the clients and relays. How thenetwork sources should be configured depends also on the capabilities of your client hosts: many oldernetworking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

source s_network { syslog(ip(10.1.2.3) transport("udp")); };

However, if possible, use the much more reliable TCP transport:

source s_network { syslog(ip(10.1.2.3) transport("tcp")); };

For other options, see Section 6.10, Collecting messages using the IETF syslog protocol (syslog()driver) (p. 140) and Section 6.14, Collecting messages from remote hosts using the BSD syslogprotocol (p. 156).

NoteStarting with syslog-ng PE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164)and IETF-syslog (RFC 5424-26) messages.

Step 4. Create local destinations that will store the log messages, for example file- or program destinations.The default configuration of syslog-ng PE places the collected messages into the /var/log/messagesfile:

70syslog-ng.com

destination d_local {

file("/var/log/messages"); };

If you want to create separate logfiles for every client host, use the ${HOST} macro when specifyingthe filename, for example:

destination d_local {

file("/var/log/messages_${HOST}"); };

For details on further macros and how to use them, see Chapter 14, Manipulating messages (p. 378).

Step 5. Create a log statement connecting the sources to the local destinations.

log {

source(s_local); source(s_network); destination(d_local); };

Step 6. Set filters, options (for example TLS encryption) and other advanced features as necessary.

NoteBy default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, notthe host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server,use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option canbe set individually for every source if needed.

If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay todo the name resolution.

Example 4.3. A simple configuration for serversThe following is a simple configuration file for syslog-ng Premium Edition that collects incoming log messagesand stores them in a text file.

@version: 6.0

@include "scl.conf"

options {

time-reap(30);

mark-freq(10);

keep-hostname(yes);

};

source s_local { system(); internal(); };

source s_network {

syslog(transport(tcp));

};

destination d_logs {

file(

"/var/log/syslog-ng/logs.txt"

owner("root")

group("root")

perm(0777)

); };

log { source(s_local); source(s_network); destination(d_logs); };

If you experience difficulties, see Chapter 19, Troubleshooting syslog-ng (p. 454) for tips on solvingcommon problems.

71syslog-ng.com

4.3. Configuring syslog-ng relays

This section describes how to configure syslog-ng PE as a relay.

4.3.1. Procedure – Configuring syslog-ng on relay hosts

Purpose:

To configure syslog-ng on a relay host, complete the following steps:

Steps:

Step 1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operatingsystems, see Chapter 3, Installing syslog-ng (p. 36).

Step 2. Configure the network sources that collect the log messages sent by the clients.

Step 3. Create a network destination that points to the syslog-ng server.

Step 4. Create a log statement connecting the network sources to the syslog-ng server.

Step 5. Configure the local sources that collect the log messages of the relay host.

Step 6. Create a log statement connecting the local sources to the syslog-ng server.

Step 7. Enable the keep-hostname() and disable the chain-hostnames() options. (For details on howthese options work, see Section chain-hostnames() (p. 334).)

NoteIt is recommended to use these options on your syslog-ng PE server as well.

Step 8. Set filters and options (for example TLS encryption) as necessary.

NoteBy default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, notthe host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server,use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option canbe set individually for every source if needed.

If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay todo the name resolution.

Example 4.4. A simple configuration for relaysThe following is a simple configuration file that collects local and incoming log messages and forwards themto a logserver using the IETF-syslog protocol.

@version: 6.0

@include "scl.conf"

options {

time-reap(30);

mark-freq(10);

keep-hostname(yes);

chain-hostnames(no);

};

source s_local { system(); internal(); };

72syslog-ng.com

Configuring syslog-ng relays

source s_network {

syslog(transport(tcp));

};

destination d_syslog_tcp {

syslog("192.168.1.5" transport("tcp") port(2010));

};

log { source(s_local); source(s_network);

destination(d_syslog_tcp);

};

4.3.2. How relaying log messages works

Depending on your exact needs about relaying log messages, there are many scenarios and syslog-ng PE optionsthat influence how the log message will look like on the log server. Some of the most common cases aresummarized in the following example.

Consider the following example: client-host > syslog-ng-relay > syslog-ng-server, where the IP address ofclient-host is 192.168.1.2. The client-host device sends a syslog message to syslog-ng-relay.Depending on the settings of syslog-ng-relay, the following can happen.

■ By default, the keep-hostname() option is disabled, so syslog-ng-relay writes the IP addressof the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discardingany IP address or hostname that was originally in the message.

■ If the keep-hostname() option is enabled on syslog-ng-relay, but name resolution is disabled(the use-dns() option is set to no), syslog-ng-relay uses the HOST field of the message as-is,which is probably 192.168.1.2.

■ To resolve the 192.168.1.2 IP address to a hostname on syslog-ng-relay using a DNS server,use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configuredand reverse DNS lookup is available for the 192.168.1.2 address, syslog-ng PE will rewrite theHOST field of the log message to client-host.

NoteIt is also possible to resolve IP addresses locally, without relying on the DNS server. For details on localname resolution, see Procedure 20.3.1, Resolving hostnames locally (p. 461).

■ The above points apply to the syslog-ng PE server (syslog-ng-server) as well, so ifsyslog-ng-relay is configured properly, use the keep-hostname(yes) option onsyslog-ng-server to retain the proper HOST field. Setting keep-hostname(no) onsyslog-ng-server would result in syslog-ng PE rewriting the HOST field to the address of thehost that sent the message to syslog-ng-server, which is syslog-ng-relay in this case.

■ If you cannot or do not want to resolve the 192.168.1.2 IP address on syslog-ng-relay, butwant to store your log messages on syslog-ng-server using the IP address of the original host

73syslog-ng.com

How relaying log messages works

(that is, client-host), you can enable the spoof-source() option on syslog-ng-relay.However, spoof-source() works only under the following conditions:

• The syslog-ng PE binary has been compiled with the --enable-spoof-source option.

• The log messages are sent using the highly unreliable UDP transport protocol. (Extremelyunrecommended.)

74syslog-ng.com

How relaying log messages works

Chapter 5. The syslog-ng PE configuration file

5.1. Location of the syslog-ng configuration file

The syslog-ng application is configured by editing the syslog-ng.conf file. Use any regular text editorapplication to modify the file.

■ On Linux and UNIX systems, the syslog-ng.conf and license.txt files are located in the/opt/syslog-ng/etc/ directory.

■ On Microsoft Windows, they are located in the C:\Program Files\syslog-ng\etc\ folder.

NoteEarlier versions of syslog-ng PE stored the configuration and license files under different directories, depending on theplatform, typically under /etc/syslog-ng/.

NoteOn Microsoft Windows platforms the syslog-ng Agent for Windows stores its configuration in the system registry or inan XML file, and can be configured from a graphical interface. However, syslog-ng PE uses a plain-text configurationfile when running on Microsoft Windows. For details about the syslog-ng Agent for Windows, see The syslog-ng Agentfor Windows 6 LTS Administrator Guide.

5.2. The configuration syntax in detail

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. Forsyslog-ng version 6 LTS, this line looks like:

@version: 6.0

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain theversion information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets theconfiguration and sends warnings about the parts of the configuration that should be updated. Version 3.0 andlater will correctly operate with configuration files of version 2.x, but the default values of certain parametershave changed since 3.0.

Example 5.1. A simple configuration fileThe following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and themessages from /dev/log into the /var/log/messages_syslog-ng.log file.

@version: 6.0

source s_local { unix-dgram("/dev/log"); internal(); };

destination d_file { file("/var/log/messages_syslog-ng.log"); };

log { source(s_local); destination(d_file); };

As a syslog-ng user described on a mailing list:

75syslog-ng.com

Location of the syslog-ng configuration file

The syslog-ng's config file format was written by programmers for programmers to beunderstood by programmers. That may not have been the stated intent, but it is how thingsturned out. The syntax is exactly that of C, all the way down to braces and statement terminators.

—Alan McKinnon

■ The main body of the configuration file consists of object definitions: sources, destinations, log pathsdefine which log message are received and where they are sent. All identifiers, option names andattributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objectsmust be defined before they are referenced in another statement. Object definitions (also calledstatements) have the following syntax:

object_type object_id {<options>};

• Type of the object: One of source, destination, log, filter, parser, rewrite rule, ortemplate.

• Identifier of the object: A unique name identifying the object. When using a reserved word as anidentifier, enclose the identifier in quotation marks.

TipUse identifiers that refer to the type of the object they identify. For example, prefix source objects withs_, destinations with d_, and so on.

• Parameters: The parameters of the object, enclosed in braces {parameters}.

• Semicolon: Object definitions end with a semicolon (;).For example, the following line defines a source and calls it s_internal.

source s_internal { internal(); };

The object can be later referenced in other statements using its ID, for example, the previous sourceis used as a parameter of the following log statement:

log { source(s_internal); destination(d_file); };

■ The parameters and options within a statement are similar to function calls of the C programminglanguage: the name of the option followed by a list of its parameters enclosed within brackets andterminated with a semicolon.

option(parameter1, parameter2); option2(parameter1, parameter2);

For example, the file() driver in the following source statement has three options: the filename(/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() optionalso has a parameter, while the flags() option has two parameters.

source s_tail { file("/var/log/apache/access.log"

follow-freq(1) flags(no-parse, validate-utf8)); };

76syslog-ng.com

The configuration syntax in detail

Objects may have required and optional parameters. Required parameters are positional, meaningthat they must be specified in a defined order. Optional parameters can be specified in any orderusing the option(value) format. If a parameter (optional or required) is not specified, its defaultvalue is used. The parameters and their default values are listed in the reference section of theparticular object.

Example 5.2. Using required and optional parametersThe unix-stream() source driver has a single required argument: the name of the socket to listen on.Optional parameters follow the socket name in any order, so the following source definitions have thesame effect:

source s_demo_stream1 {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

source s_demo_stream2 {

unix-stream("<path-to-socket>" group(log) max-connections(10)); };

■ Some options are global options, or can be set globally, for example, whether syslog-ng PE shoulduse DNS resolution to resolve IP addresses. Global options are detailed in Chapter 9, Global optionsof syslog-ng PE (p. 334).

options { use-dns(no); };

■ All identifiers, attributes, and any other strings used in the syslog-ng configuration file are casesensitive.

■ Objects can be used before definition.

■ To add comments to the configuration file, start a line with # and write your comments. These linesare ignored by syslog-ng.

# Comment: This is a stream source

source s_demo_stream {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

TipBefore activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng--syntax-only command.

To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload

command.

5.3. Notes about the configuration syntax

When you are editing the syslog-ng configuration file, note the following points:

■ The configuration file can contain a maximum of 6665 source / destination / log elements.

■ When writing the names of options and parameters (or other reserved words), the hyphen (-) andunderscore (_) characters are equivalent, for example max-connections(10) andmax_connections(10) are both correct.

77syslog-ng.com

Notes about the configuration syntax

■ Numbers can be prefixed with + or - to indicate positive or negative values. Numbers beginningwith zero (0) or 0x are treated as octal or hexadecimal numbers, respectively.

■ You can use commas (,) to separate options or other parameters for readability, syslog-ng completelyignores them. The following declarations are equivalent:

source s_demo_stream {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

source s_demo_stream {

unix-stream("<path-to-socket>", max-connections(10), group(log));

};

■ When enclosing object IDs (for example the name of a destination) between double-quotes("mydestination"), the ID can include whitespace as well, for example:

source "s demo stream" {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

NoteOn Microsoft Windows platforms, enclose paths and filenames between single quotes, for example,'C:\temp\logs\mylogs.log'

■ For notes on using regular expressions, see Section 14.3, Regular expressions (p. 403).

5.4. Global and environmental variables

Starting with syslog-ng PE version 4 F1, it is possible to define global variables in the configuration file. Globalvariables are actually name-value pairs. When syslog-ng processes the configuration file during startup, itautomatically replaces `name` with value. To define a global variable, use the following syntax:

@define name "value"

The value can be any string, but special characters must be escaped.To use the variable, insert the name of thevariable enclosed between backticks (`, similarly to using variables in Linux or UNIX shells) anywhere in theconfiguration file.

The value of the global variable can be also specified using the following methods:

■ Without any quotes, as long as the value does not contain any spaces or special characters. In otherword, it contains only the following characters: a-zA-Z0-9_..

■ Between apostrophes, in case the value does not contain apostrophes.

■ Between double quotes, in which case special characters must be escaped using backslashes (\).

TipThe environmental variables of the host are automatically imported and can be used as global variables.

78syslog-ng.com

Global and environmental variables

Example 5.3. Using global variablesFor example, if an application is creating multiple log files in a directory, you can store the path in a global variable, anduse it in your source definitions.

@define mypath "/opt/myapp/logs"

source s_myapp_1 { file("`mypath`/access.log" follow-freq(1)); };

source s_myapp_2 { file("`mypath`/error.log" follow-freq(1)); };

source s_myapp_3 { file("`mypath`/debug.log" follow-freq(1)); };

The syslog-ng PE application will interpret this as:

@define mypath "/opt/myapp/logs"

source s_myapp_1 { file("/opt/myapp/logs/access.log" follow-freq(1)); };

source s_myapp_2 { file("/opt/myapp/logs/error.log" follow-freq(1)); };

source s_myapp_3 { file("/opt/myapp/logs/debug.log" follow-freq(1)); };

5.5. Logging configuration changes

Every time syslog-ng is started, or its configuration is reloaded, it automatically logs the SHA-1 fingerprint ofits configuration file using the internal() message source. That way any modification of the configurationof your syslog-ng clients is visible in the central logs. Note that the log message does not contain the exactchange, nor can the configuration file be retrieved from the fingerprint. Only the fact of the configuration changecan be detected.

NoteModular configuration files that are included in the main configuration file of syslog-ng PE are included when thefingerprint is calculated. However, other external files (for example, scripts used in program sources or destinations) areexcluded, therefore their modifications do not change the fingerprint.

The fingerprint can be examined with the logchksign command-line application, which detects that thefingerprint was indeed generated by a syslog-ng application. Just paste the hashes from the log message afterthe logchksign command like in the following example:

bin/logchksign "cfg-fingerprint='832ef664ff79df8afc66cd955c0c8aaa3c343f31',

cfg-nonce-ndx='0', cfg-signature='785223cfa19ad52b855550be141b00306347b0a9'"

5.6. Modules in syslog-ng PE

The syslog-ng PE application is modular, to increase its flexibility and also to simplify the development ofadditional modules. Most of the functionality of syslog-ng PE is in separate modules. That way it becomes alsopossible to finetune the resource requirements of syslog-ng PE, for example, by loading only the modules thatare actually used in the configuration, or simply omitting modules that are not used but require large amountof memory.

Each module contains one or more plugins, which add some functionality to syslog-ng PE, for example, adestination or a source driver.

■ To display the list of available modules, execute the syslog-ng --version command.

■ To the description of the available modules, execute the syslog-ng --module-registry

command.

79syslog-ng.com

Logging configuration changes

■ To customize which modules are loaded automatically when syslog-ng PE is started, use the--default-modules command-line option of syslog-ng PE.

■ To request loading a module from the syslog-ng PE configuration file, see Section 5.6.1, Loadingmodules (p. 80).

For details on the command-line parameters of syslog-ng PE mentioned in the previous list, see the syslog-ngPE man page at syslog-ng(8) (p. 493).

5.6.1. Loading modules

The syslog-ng Premium Edition application loads every available module during startup, except the snmp()module, and Java-related modules like hdfs() (Java-related modules require the mod-java module). Fordetails on using the snmp() destination driver, see Section 7.12, Sending SNMP traps (p. 267), for details onthe hdfs() driver, see Section 7.4, Storing messages on the Hadoop Distributed File System (HDFS) (p. 198).

To load a module that is not loaded automatically, include the following statement in the syslog-ng PEconfiguration file:

@module <module-name>

Note the following points about the @module statement:

■ The @module statement is a top-level statement, that is, it cannot be nested into any other statement.Usually it is used immediately after the @version statement.

■ Every @module statement loads a single module: loading multiple modules requires a separate@module statement for every module.

■ In the configuration file, the @module statement of a module must be earlier than the module is used.

5.7. Managing complex syslog-ng configurations

The following sections describe some methods that can be useful to simplify the management of large-scalesyslog-ng PE installations. If you are using Puppet to manage your IT infrastructure, you can use it to manageyour syslog-ng PE configurations as well. For details, see Procedure 3.10, Managing syslog-ng PE fromPuppet (p. 58).

5.7.1. Including configuration files

The syslog-ng application supports including external files in its configuration file, so parts of its configurationcan be managed separately. To include the contents of a file in the syslog-ng configuration, use the followingsyntax:

@include "<path to filename>"

80syslog-ng.com

Loading modules

NoteIf you enter only the filename, syslog-ng PE will search for the file in the default directory: /opt/syslog-ng/etc. Ifsyslog-ng PE has been installed to a different directory, use the full path instead.

This imports the entire file into the configuration of syslog-ng PE, at the location of the include statement. The<filename> can be one of the following:

■ A filename, optionally with full path. The filename (not the path) can include UNIX-style wildcardcharacters (*, ?). When using wildcard characters, syslog-ng PE will include every matching file.For details on using wildcard characters, see Section glob (p. 405).

■ A directory. When including a directory, syslog-ng PE will try to include every file from the directory,except files beginning with a ~ (tilde) or a . (dot) character. Including a directory is not recursive.The files are included in alphabetic order, first files beginning with uppercase characters, then filesbeginning with lowercase characters. For example, if the directory contains the a.conf, B. conf,

c.conf, D.conf files, they will be included in the following order: B.conf, D. conf, a.conf,

c.conf.

When including configuration files, consider the following points:

■ If an object is defined twice (for example the original syslog-ng configuration file and the file importedinto this configuration file both define the same option, source, or other object), then the object thatis defined later in the configuration file will be effective. For example, if you set a global option atthe beginning of the configuration file, and later include a file that defines the same option with adifferent value, then the option defined in the imported file will be used.

■ Files can be embedded into each other: the included files can contain include statements as well, upto a maximum depth of 15 levels.

■ You cannot include complete configuration files into each other, only configuration snippets can beincluded. This means that the included file cannot have a @version statement.

■ Include statements can only be used at top level of the configuration file. For example, the followingis correct:

@version: 6.0

@include "example.conf"

But the following is not:

source s_example {

@include "example.conf"

};

WarningThe syslog-ng application will not start if it cannot find a file that is to be included in its configuration. Always double-checkthe filenames, paths, and access rights when including configuration files, and use the --syntax-only command-lineoption to check your configuration.

81syslog-ng.com

Including configuration files

5.7.2. Reusing configuration blocks

To create a reusable configuration snippet and reuse parts of a configuration file, you have to define the block(for example, a source) once, and reference it later. (Such reusable blocks are sometimes called a SourceConfiguration Library, or SCL.) Any syslog-ng object can be a block. Use the following syntax to define ablock:

block type name() {<contents of the block>};

Type must be one of the following: destination, filter, log, parser, rewrite, root, source. Theroot blocks can be used in the "root" context of the configuration file, that is, outside any other statements.

Blocks may be nested into each other, so for example a block can be built from other blocks. Blocks aresomewhat similar to C++ templates.

The type and name combination of each block must be unique, that is, two blocks can have the same name iftheir type is different.

To use a block in your configuration file, you have to do two things:

■ Include the file defining the block in the syslog-ng.conf file — or a file already included intosyslog-ng.conf.

■ Reference the name of the block in your configuration file. This will insert the block into yourconfiguration. For example, to use a block called myblock, include the following line in yourconfiguration:

myblock()

Blocks may have parameters, but even if they do not, the reference must include opening and closingparentheses like in the previous example.

The contents of the block will be inserted into the configuration when syslog-ng PE is started or reloaded.

Example 5.4. Reusing configuration blocksSuppose you are running an application on your hosts that logs into the /opt/var/myapplication.log file. Createa file (for example, myblocks.conf) that stores a source describing this file and how it should be read:

block source myappsource() {

file("/opt/var/myapplication.log" follow-freq(1) default-facility(syslog)); };

Include this file in your main syslog-ng configuration file, reference the block, and use it in a logpath:

@version: 6.0

@include "<correct/path>/myblocks.conf"

source s_myappsource { myappsource(); };

...

log { source(s_myappsource); destination(...); };

To define a block that defines more than one object, use root as the type of the block, and reference the blockfrom the main part of the syslog-ng PE configuration file.

82syslog-ng.com

Reusing configuration blocks

Example 5.5. Defining blocks with multiple elementsThe following example defines a source, a destination, and a log path to connect them.

block root mylogs() {

source s_file { file("/var/log/mylogs.log" follow-freq(1)); };

destination d_local { file("/var/log/messages"); };

log { source(s_file); destination(d_local); };

};

TipSince the block is inserted into the syslog-ng PE configuration when syslog-ng PE is started, the block can be generateddynamically using an external script if needed. This is useful when you are running syslog-ng PE on different hosts andyou want to keep the main configuration identical.

If you want to reuse more than a single configuration object, for example, a logpath and the definitions of its sources anddestinations, use the include feature to reuse the entire snippet. For details, see Section 5.7.1, Including configurationfiles (p. 80).

5.7.2.1. Passing arguments to configuration blocks

Configuration blocks can receive arguments as well. The parameters the block can receive must be specifiedwhen the block is defined, using the following syntax:

block type block_name(argument1(<default-value-of-the-argument>)

argument2(<default-value-of-the-argument>) argument3())

If an argument does not have a default value, use empty parentheses after the name of the argument. To referthe value of the argument in the block, use the name of the argument between backticks (for example,`argument1`).

Example 5.6. Passing arguments to blocksThe following sample defines a file source block, which can receive the name of the file as a parameter. If no parameteris set, it reads messages from the /var/log/messages file.

block source s_logfile (filename("messages")) {

file("/var/log/`filename`" );

};

source s_example {

s_logfile(filename("logfile.log"));

};

If you reference the block with more arguments then specified in its definition, you can use these additionalarguments as a single argument-list within the block. That way, you can use a variable number of optionalarguments in your block. This can be useful when passing arguments to a template, or optional arguments toan underlying driver. To reference this argument-list, insert `__VARARGS__` to the place in the block whereyou want to insert the argument-list. Note that you can use this only once in a block. The following definitionextends the logfile block from the previous example, and passes the optional arguments (follow-freq(1)flags(no-parse)) to the file() source.

block source s_logfile (filename("messages")) {

file("/var/log/`filename`" `__VARARGS__`);

};

83syslog-ng.com

Reusing configuration blocks

source s_example {

s_logfile(filename("logfile.log") follow-freq(1) flags(no-parse));

};

84syslog-ng.com

Reusing configuration blocks

Chapter 6. Collecting log messages — sourcesand source drivers

6.1. How sources work

A source is where syslog-ng receives log messages. Sources consist of one or more drivers, each defining whereand how messages are received.

To define a source, add a source statement to the syslog-ng configuration file using the following syntax:

source <identifier> { source-driver(params); source-driver(params); ... };

Example 6.1. A simple source statementThe following source statement receives messages on the TCP port 1999 of the interface having the 10.1.2.3 IP address.

source s_demo_tcp { network(ip(10.1.2.3) port(1999)); };

Example 6.2. A source statement using two source driversThe following source statement receives messages on the 1999 TCP port and the 1999 UDP port of the interface havingthe 10.1.2.3 IP address.

source s_demo_two_drivers {

network(ip(10.1.2.3) port(1999));

network(ip(10.1.2.3) port(1999) transport("udp")); };

Example 6.3. Setting default priority and facilityIf the message received by the source does not have a proper syslog header, you can use the default-facility() anddefault-priority() options to set the facility and priority of the messages. Note that these values are applied onlyto messages that do not set these parameters in their header.

source headerless_messages { network(default-facility(syslog) default-priority(emerg));

};

Define a source only once. The same source can be used in several log paths. Duplicating sources causessyslog-ng to open the source (TCP/IP port, file, and so on) more than once, which might cause problems. Forexample, include the /dev/log file source only in one source statement, and use this statement in more thanone log path if needed.

WarningSources and destinations are initialized only when they are used in a log statement. For example, syslog-ng PE startslistening on a port or starts polling a file only if the source is used in a log statement. For details on creating log statements,see Chapter 8, Routing messages: log paths, reliability, and filters (p. 311).

85syslog-ng.com

How sources work

To collect log messages on a specific platform, it is important to know how the native syslogd communicateson that platform. The following table summarizes the operation methods of syslogd on some of the testedplatforms:

MethodPlatformA SOCK_DGRAM unix socket named /dev/log. Newerdistributions that use systemd collect log messages intoa journal file.

Linux

A SOCK_DGRAM unix socket named /var/run/log.BSD flavors

An SVR4 style STREAMS device named /dev/log.Solaris (2.5 or below)

In addition to the STREAMS device used in earlierversions, 2.6 uses a new multithreaded IPC method

Solaris (2.6 or above)

called door. By default the door used by syslogd is/etc/.syslog_door.

Special Eventlog containers that can be accessed usingthe proprietary API of Microsoft.

Windows

HP-UX uses a named pipe called /dev/log that ispadded to 2048 bytes, for example source s_hp-ux

{pipe ("/dev/log" pad-size(2048)}.

HP-UX 11 or later

A SOCK_STREAM or SOCK_DGRAM unix socket called/dev/log.

AIX 5.2 and 5.3

Table 6.1. Communication methods used between the applications and syslogd

Each possible communication mechanism has a corresponding source driver in syslog-ng. For example, to opena unix socket with SOCK_DGRAM style communication use the driver unix-dgram. The same socket using theSOCK_STREAM style — as used under Linux — is called unix-stream.

Example 6.4. Source statement on a Linux based operating systemThe following source statement collects the following log messages:

internal(): Messages generated by syslog-ng.■

■ network(transport("udp")): Messages arriving to the 514/UDP port of any interface of the host.

■ unix-dgram("/dev/log");: Messages arriving to the /dev/log socket.

source s_demo {

internal();

network(transport("udp"));

unix-dgram("/dev/log"); };

The following table lists the source drivers available in syslog-ng.

DescriptionNameCollects Windows eventlog messages.eventlog()

Opens the specified file and reads messages.file()

Messages generated internally in syslog-ng.internal()

86syslog-ng.com

How sources work

DescriptionNameReceives messages from remote hosts using theBSD-syslog protocol over IPv4 and IPv6. Supports theTCP, UDP,RLTP™, and TLS network protocols.

network()

Opens the specified named pipe and reads messages.pipe()

Opens the specified application and reads messagesfrom its standard output.

program()

Collects logs from tables of relational databasesql()

Opens the specified STREAMS device on Solaris systemsand reads incoming messages.

sun-stream(), sun-streams()

Listens for incoming messages using the newIETF-standard syslog protocol.

syslog()

Automatically detects which platform syslog-ng PE isrunning on, and collects the native log messages of thatplatform.

system()

Collects messages directly from the journal of platformsthat use systemd.

systemd-journal()

Collects messages from the journal using a socket onplatforms that use systemd.

systemd-syslog()

Opens the specified unix socket in SOCK_DGRAM modeand listens for incoming messages.

unix-dgram()

Opens the specified unix socket in SOCK_STREAMmodeand listens for incoming messages.

unix-stream()

Table 6.2. Source drivers available in syslog-ng

6.2. Collecting messages from Windows eventlog sources

The syslog-ng PE application can collect messages from the standard Windows eventlog containers, as well asfrom custom containers. The syslog-ng PE application forwards the messages from three standard eventlogcontainers (Application, Security, System). To enable or disable these sources, or to add customeventlog containers, complete the following steps:

WarningIf an eventlog container becomes corrupt, syslog-ng PE will stop processing the event source. A log message (Eventlogfile is corrupt) is sent directly to the log server to notify about the error.

WarningHazard of data loss! It is not recommended to setup archiving for the event container. It is possible to lose logs if thereare non-processed events in the event container when the archiving is started. Windows closes and renames the eventcontainer and starts a new one regardless of any reading applications.

To prevent this, enable overwrite events when neededmode in the Windows Event Viewer with the following conditions:

■ The messages are not generated faster than the syslog-ng PE's processing speed.

87syslog-ng.com

Collecting messages from Windows eventlog sources

■ There is enough window between the first and the last events for planned syslog-ng PE stops. Ensure thatnew events will not overwrite the event last read by syslog-ng PE during syslog-ng PE stop.

Example 6.5. Using the eventlog() driver

source s_AGENT_EVENT{

eventlog(channel(Application)

prefix(".SDATA.win@18372.4")

read-old-records(no)

);

eventlog(channel(A*lic?tion)

prefix(".SDATA.win@18372.4")

);

eventlog(channel(*)

prefix(".SDATA.win@18372.4")

);

};

6.2.1. eventlog() source options

The following options can be specified for the eventlog() driver:

default-facility()facility stringType:kernDefault:

Description: This parameter assigns a facility value to the messages received from the eventlog source, if themessage does not specify one.

default-level()priority stringType:

Default:

Description: The alias of the default-priority() parameter.

channel()stringType:

Default:

Description: Set the windows containers, separated with a space. The container name may include wildcardcharacters (for example *).

event-api()EVT_API or EVTX_APIType:Under Windows Vista: EVT_API, Windows Vista and above: EVTX_APIDefault:

Description: To read the messages from eventlog containers, the syslog-ng PE application uses thenativeWindows API tools. The Windows Vista and newer platforms use an XML-based eventlog format. The

88syslog-ng.com

eventlog() source options

API (calledEVTX) that reads the XML-messages from the eventlog container and passes them to syslog-ng PEis inherently slow, severely limiting the performance of syslog-ng PE. The API tools that syslog-ng PE useson the Microsoft Windows XP and 2003 Server platforms is available on the newer platforms as well, and canincrease the speed of reading from eventlog containers, up to 500%. However, using this old API (called EVT)has limitations when used with XML-based eventlog containers.

prefix()stringType:

Default:

Description: To send eventlog message metadata as SDATA when sending the message in IETF-syslog format,enter the prefix in the following format: .SDATA.<name>@<private enterprise number>. For example,.SDATA.mySDATA-field@18372.4. (18372.4 is the private enterprise number of BalaBit IT Security, thedeveloper of syslog-ng PE.)

NoteOn the destination server, you can refer to message fields like: ${.SDATA.<prefix()>.EVENT_SOURCE}

read-old-records()yes|noType:noDefault:

Description: If set to yes, syslog-ng PE will start reading the records from the beginning of the container, ifthe container has not been read yet. If set to no, syslog-ng PE will read only the new records. If the source hasa state in the persist file, this option will have no effect.

6.2.2. Limitations of using the EVT API on Windows Vista or newer

When using the EVT API to read messages from XML-based eventlog containers, note the following limitations.

■ The EVT API suppor ts only conta iners are l i s ted under theHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\ key in theregistry. The three default containers (Security, Application, System) are listed here by default.

■ Derived containers (for example, microsoft-windows-bits-client/analytic) are notsupported.

■ The following macros do not work, that is, their values will be empty: ${EVENT_CATEGORY},${EVENT_MESSAGE_XML}, ${EVENT_MSG_XML}, and ${EVENT_TASK}.

■ The way how the EVT API provided by Microsoft reads the values of the XML-based is not perfect.Therefore, filters that use these macros might not work properly. The following list shows the knownlimitations and errors:

• ${EVENT_LEVEL}: The value of this macro can be incorrect. It will always be a number asexpected, but not necessarily the correct value.

89syslog-ng.com

Limitations of using the EVT API on Windows Vista or newer

• ${EVENT_SOURCE}: It is possible that the value of this macro will be formatted differently. Forexample, Microsoft-Windows-Security-Auditing instead of Microsoft Windows

security auditing.

• ${EVENT_TYPE}: The value of this macro is known to be incorrect in the following scenarios:

• For security audit logs, if ${EVENT_LEVEL} is 4, the value of the ${EVENT_TYPE} macro willbe Audit Success instead of Information. This is known to happen when the "Audit logcleared" event is generated.

• For non-security audit logs, if ${EVENT_LEVEL} is 0, the value of the ${EVENT_TYPE} macrowill be Undefined instead of Information.

• ${EVENT_USERNAME}: The EVT API will always add value of the Username field to this macro.If the Username field of the event is empty, the EVTX API used the TargetUserName or theSubjectUserName instead, but this is not possible with the EVT API. For example, the Usernamefield of events from the security container will be often N/A.

• ${PRI}: The value of this macro is based on the ${EVENT_LEVEL}, therefore, it can be incorrect.

6.3. Collecting internal messages

All messages generated internally by syslog-ng use this special source. To collect warnings, errors and noticesfrom syslog-ng itself, include this source in one of your source statements.

internal()

The syslog-ng application will issue a warning upon startup if none of the defined log paths reference thisdriver.

Example 6.6. Using the internal() driver

source s_local { internal(); };

The syslog-ng PE application sends the following message types from the internal() source:

■ fatal: Priority value: critical (2), Facility value: syslog (5)

■ error: Priority value: error (3), Facility value: syslog (5)

■ warning: Priority value: warning (4), Facility value: syslog (5)

■ notice: Priority value: notice (5), Facility value: syslog (5)

■ info: Priority value: info (6), Facility value: syslog (5)

90syslog-ng.com

Collecting internal messages

6.3.1. internal() source options

The internal() driver has the following options:

host-override()stringType:

Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.7. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

normalize-hostnames()yes | noAccepted values:noDefault:

Description: If enabled (normalize-hostnames(yes)), syslog-ng PE converts the hostnames to lowercase.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

tags()stringType:

Default:

91syslog-ng.com

internal() source options

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

use-fqdn()yes or noType:noDefault:

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specifiedglobally, and per-source as well. The local setting of the source overrides the global option if available.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.4. Collecting messages from text files

Collects log messages from plain-text files, for example from the logfiles of an Apache webserver.

The syslog-ng application notices if a file is renamed or replaced with a new file, so it can correctly follow thefile even if logrotation is used. When syslog-ng is restarted, it records the position of the last sent log messagein the /opt/syslog-ng/var/syslog-ng.persist file, and continues to send messages from this positionafter the restart.

The file driver has a single required parameter specifying the file to open. For the list of available optionalparameters, see Section 6.4.3, file() source options (p. 94).

WarningHazard of data loss! If your log files are on an NFS-mounted network file system, see Section 2.14, NFS file system forlog files (p. 34).

Declaration:

file("filename");

Example 6.8. Using the file() driver

source s_file { file("/var/log/messages"); };

NoteOn Microsoft Windows platforms, enclose paths and filenames between single quotes, forexample, 'C:\temp\logs\mylogs.log'

92syslog-ng.com

Collecting messages from text files

Example 6.9. Tailing filesThe following source checks the access.log file every second for new messages.

source s_tail { file("/var/log/apache/access.log"

follow-freq(1) flags(no-parse)); };

NoteIf the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kernfacility. Use the default-facility() and default-priority() options in the source definition to assign a differentfacility if needed.

Wildcards and file sources.

In syslog-ng PE, the filename (but not the pathname) may include wildcard characters (for example *). Notethat when using wildcards in filenames, always set how often syslog-ng PE should check the file for newmessages using the follow-freq() parameter.

WarningIf you use wildcards in multiple file sources, make sure that the files and folders that match the wildcards do not overlap.That is, every file and folder should belong to only one file source. Monitoring a file from multiple wildcard sources canlead to data loss.

To use wildcards in the file source if your log files are on an NFS file system, set theforce-directory-polling() option to yes to detect newly created files. Note that wildcard file sourcesare available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, and are not yet availablein syslog-ng PE version 7.

When using wildcards, syslog-ng PE monitors every matching file, and can receive new log messages fromany of the files. However, monitoring (polling) many files (that is, more than ten) has a significant overheadand may affect performance. On Linux this overhead is not so significant, because syslog-ng PE uses the inotifyfeature of the kernel.

Also, by default, the operating system notifies syslog-ng PE when an application modifies a logfile. However,in some cases this does not happen, because the file-monitoring API of the operating system does not noticethat the file has changed, for example, when monitoring logfiles of the Windows DHCP service. In such cases,enable the force-directory-polling() option. Note that enabling this option decreases the performanceof syslog-ng PE if you monitor lots of logfiles.

Example 6.10. Using wildcards in the filenameThe following example monitors every file with the .log extension in the /var/application directory for log messages.Note that only syslog-ng PE supports wildcards in the file and pathnames.

source s_file { file("/var/application/*.log" follow-freq(1));};

6.4.1. Notes on reading kernel messages

Note the following points when reading kernel messages on various platforms.

93syslog-ng.com

Notes on reading kernel messages

■ The kernel usually sends log messages to a special file (/dev/kmsg on BSDs, /proc/kmsg onLinux). The file() driver reads log messages from such files. The syslog-ng application canperiodically check the file for new log messages if the follow-freq() option is set.

■ On Linux, the klogd daemon can be used in addition to syslog-ng to read kernel messages andforward them to syslog-ng. klogd used to preprocess kernel messages to resolve symbols and soon, but as this is deprecated by ksymoops there is really no point in running both klogd and syslog-ngin parallel. Also note that running two processes reading /proc/kmsg at the same time might resultin dead-locks.

■ When using syslog-ng to read messages from the /proc/kmsg file, syslog-ng automatically disablesthe follow-freq() parameter to avoid blocking the file.

■ To read the kernel messages on HP-UX platforms, use the following options in the source statement:

file("/dev/klog" program-override("kernel") flags(kernel) follow-freq(0));

6.4.2. File sources and the RFC5424 message format

When reading messages from a file and forwarding them in IETF-syslog (RFC5424) format, syslog-ng PEautomatically adds all file-related information to the file@18372.4 SDATA block. When the source is fileand the transport protocol is syslog or syslog-protocol flags were used in the destination side, the message willcontain the following source file-related information:

■ size: size of the file

■ position: file position the message was read from

■ name: name of the file

Example 6.11. File-related information in message

309 <38>1 2010-10-19T15:50:45.018203+02:00 server1 localprg 1234 - [timeQuality isSynced="0"

tzKnown="0"][file@18372.4 size="184567" pos="1024" name="/var/tmp/msg.txt"] seq: 0000000001,

runid: 1287496244, stamp: 2010-10-19T15:50:45 messagetext

6.4.3. file() source options

The file() driver has the following options:

default-facility()facility stringType:kernDefault:

Description: This parameter assigns a facility value to the messages received from the file source, if the messagedoes not specify one.

94syslog-ng.com

File sources and the RFC5424 message format

default-priority()priority stringType:

Default:

Description: This parameter assigns an emergency level to the messages received from the file source, if themessage does not specify one. For example, default-priority(warning)

encoding()stringType:

Default:

Description: Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslogprotocol. To list the available character sets on a host, execute the iconv -l command. For details on howencoding affects the size of the message, see Section Message size and encoding (p. 25).

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

95syslog-ng.com

file() source options

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

follow-freq()number (seconds)Type:1Default:

Description: Indicates that the source should be checked periodically. This is useful for files which alwaysindicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng willnot attempt to use poll() on the file, but checks whether the file changed every time the follow-freq()

interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

force-directory-polling()yes or noType:noDefault:

Description: Specifies whether syslog-ng should force the polling of the logfiles that match the wildcardedfilenames specified in the file() driver.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

96syslog-ng.com

file() source options

NoteEnabling this option decreases performance if you monitor lots of logfiles.

To use wildcards in the file source if your log files are on an NFS file system, set theforce-directory-polling() option to yes to detect newly created files. Note that wildcard file sourcesare available only in syslog-ng PE version 6.0.3 and newer versions of the 6.x branch, and are not yet availablein syslog-ng PE version 7.

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()numberType:1000Default:

Description: The size of the initial window, this value is used during flow control. Make sure thatlog-iw-size() is larger than the value of log-fetch-limit().

When using wildcards in the filenames, syslog-ng PE attempts to read log-fetch-limit() number ofmessages from each file. For optimal performance, make sure that log-iw-size() is greater thanlog-fetch-limit()*(the-number-of-matching-files).

Example 6.12. Initial window size of file sourcesIf log-fetch-limit() is 10, and your wildcard file source has 200 files, then log-iw-size() should be at least 2000.

97syslog-ng.com

file() source options

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

log-prefix() (DEPRECATED)stringType:

Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string toany log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE:This option is deprecated. Use program-override() instead.

multi-line-garbage()regular expressionType:empty stringDefault:

Description: Use the multi-line-garbage() option when processing multi-line messages that containunneeded parts between the messages. Specify a string or regular expression that matches the beginning of theunneeded message parts. If the multi-line-garbage() option is set, syslog-ng PE ignores the lines betweenthe line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See alsothe multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but nomatching line is received between two lines that match multi-line-prefix(), syslog-ng PE will continueto process the incoming lines as a single message until a line matching multi-line-garbage() is received.

WarningIf the multi-line-garbage() option is set, syslog-ng PE discards lines between the line matching themulti-line-garbage() and the next line matching multi-line-prefix().

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

multi-line-prefix()regular expressionType:empty stringDefault:

98syslog-ng.com

file() source options

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messagesthat contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matchesthe beginning of the log messages. Use as simple regular expressions as possible, because complex regularexpressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix()

option is set, syslog-ng PE ignores newline characters from the source until a line matches the regular expressionagain, and treats the lines between the matching lines as a single message. See also the multi-line-garbage()option.

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

TipTo make multi-line messages more readable when written to a file, use a template in the destination andinstead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). Thisexpression inserts a tab after every newline character (except when a tab is already present), indenting everyline of the message after the first. For example:

■

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );

};

For details on using templates, see Section 14.1.2, Templates and macros (p. 379).

■ To actually convert the lines of multi-line messages to single line (by replacing the newline characters withwhitespaces), use the flags(no-multi-line) option in the source.

Example 6.13. Processing Tomcat logsThe log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages startwith the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

SEVERE: Catalina.start:

LifecycleException: service.getName(): "Catalina"; Protocol handler start failed:

java.net.BindException: Address already in use<null>:8080

at org.apache.catalina.connector.Connector.start(Connector.java:1138)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:583)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

INFO: Server startup in 1206 ms

2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

99syslog-ng.com

file() source options

To process these messages, specify a regular expression matching the timestamp of the messages in themulti-line-prefix() option. Such an expression is the following:

source s_file{ file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0)

multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));

};

Note that the flags(no-parse) is needed to avoid syslog-ng PE trying to interpret the date in the message.

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

read-old-records()yes|noType:yesDefault:

Description: If set to yes, syslog-ng PE will start reading the records from the beginning of the file, if the filehas not been read yet. If set to no, syslog-ng PE will read only the new records. If the source has a state in thepersist file, this option will have no effect.

recursiveyes or noType:noDefault:

Description: When enabled, syslog-ng PE monitors every subdirectory of the directory set in the path of thefile parameter, and reads log messages from files with the set filename. The recursive option can be usedtogether with wildcards in the filename.

100syslog-ng.com

file() source options

Example 6.14. Monitoring multiple directoriesThe following example reads files having the .log extension from the /var/application/ directory and itssubdirectories. Note that only syslog-ng PE supports recursive directory handling and wildcards in the file and pathnames.

source s_file_subdirectories { file("/var/application/*.log"

recursive(yes)

follow-freq(1)

log-fetch-limit(100)

);};

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.5. Collecting messages using the RFC3164 protocol (network() driver)

The network() source driver can receive syslog messages conforming to RFC3164 from the network using theTCP, TLS, and UDP networking protocols.

You can use the RLTP™ protocol as well. For details about the RLTP™ protocol, see Chapter 12, ReliableLog Transfer Protocol™ (p. 364).

■ UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messagesbetween hosts. It may lose messages, and no attempt is made to retransmit lost messages. TheBSD-syslog protocol traditionally uses UDP.Use UDP only if you have no other choice.

101syslog-ng.com

Collecting messages using the RFC3164 protocol (network() driver)

For details on minimizing message loss when using UDP, see the following tutorial:Collecting logmessages from UDP sources.

■ TCP provides connection-oriented service: the client and the server establish a connection, eachmessage is acknowledged, and lost packets are resent. TCP can detect lost connections, and messagesare lost, only if the TCP connection breaks. When a TCP connection is broken, messages that theclient has sent but were not yet received on the server are lost.

■ The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) over TCP.For details, see Section 10.2, Encrypting log messages with TLS (p. 351).

When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server,you can use different protocols and options. Every combination has its advantages and disadvantages. The mostimportant thing is to use matching protocols and options, so the server handles the incoming log messagesproperly. For details, see Section 2.13, Things to consider when forwarding messages between syslog-ng PEhosts (p. 31).

Declaration:

network([options]);

By default, the network() driver binds to 0.0.0.0, meaning that it listens on every available IPV4 interfaceon the TCP/601 port. To limit accepted connections to only one interface, use the localip() parameter. Tolisten on IPv6 addresses, use the ip-protocol(6) option.

Example 6.15. Using the network() driverUsing only the default settings: listen on every available IPV4 interface on the TCP/601 port.

source s_network {

network();

};

UDP source listening on 192.168.1.1 (the default port for UDP is 514):

source s_network {

network(

ip("192.168.1.1")

transport("udp")

);

};

TCP source listening on the IPv6 localhost, port 2222:

source s_network6 {

network(

ip("::1")

transport("tcp")

port(2222)

ip-protocol(6)

);

};

A TCP source listening on a TLS-encrypted channel.

source s_network {

network(

transport("tcp")

port(2222)

tls(peer-verify("required-trusted")

key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt")

);

102syslog-ng.com

Collecting messages using the RFC3164 protocol (network() driver)

);

};

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslogmessages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses boththe IETF-syslog message format and the protocol. For details, see Section 6.10, Collecting messages using the IETFsyslog protocol (syslog() driver) (p. 140).

source s_tcp_syslog {

network(

ip("127.0.0.1")

flags(syslog-protocol)

);

};

For details on the options of the network() source, see Section 6.5.1, network() source options (p. 103).

6.5.1. network() source options

The network() driver has the following options.

encoding()stringType:

Default:

Description: Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslogprotocol. To list the available character sets on a host, execute the iconv -l command. For details on howencoding affects the size of the message, see Section Message size and encoding (p. 25).

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

103syslog-ng.com

network() source options

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

■ threaded: The threaded flag enables multithreading for the source. For details on multithreading,see Chapter 18, Multithreading and scaling in syslog-ng PE (p. 449).

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

104syslog-ng.com

network() source options

NoteThe syslog source uses multiple threads only if the source uses the tls or tcp transport protocols.

host-override()stringType:

Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

ip() or localip()stringType:0.0.0.0Default:

Description: The IP address to bind to. By default, syslog-ng PE listens on every available interface. Note thatthis is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, syslog-ng PE automatically joins the necessarymulticast group. TCP does not support multicasting.

ip-protocol()number (IP version)Type:4Default:

Description: Determines the internet protocol version of the given driver (network() or syslog()). Thepossible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is 4.

ip-tos()number (type of service)Type:0Default:

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()number (hops)Type:0Default:

Description: Specifies the Time-To-Live value of outgoing packets.

105syslog-ng.com

network() source options

keep-alive()yes or noType:yesDefault:

Description: Specifies whether connections to sources should be closed when syslog-ng is forced to reload itsconfiguration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of thesyslog-ng connections, client-side (destination) connections are always reopened after receiving a HUP signalunless the keep-alive option is enabled for the destination.

keep-hostname()yes or noType:noDefault:

Description: Enable or disable hostname rewriting.

■ If enabled (keep-hostname(yes)), syslog-ng PE assumes that the incoming log message was sentby the host specified in the HOST field of the message.

■ If disabled (keep-hostname(no)), syslog-ng PE rewrites the HOST field of the message, either tothe IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns()

parameter is set to yes and the IP address can be resolved to a hostname) of the host sending themessage to syslog-ng PE. For details on using name resolution in syslog-ng PE, see Section 20.3,Using name resolution in syslog-ng (p. 461).

NoteIf the log message does not contain a hostname in its HOST field, syslog-ng PE automatically adds a hostname to themessage.

For messages received from the network, this hostname is the address of the host that sent the message (thismeans the address of the last hop if the message was transferred via a relay).

■

■ For messages received from the local host, syslog-ng PE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the globaloption if available.

NoteWhen relaying messages, enable this option on the syslog-ng PE server and also on every relay, otherwise syslog-ng PEwill treat incoming messages as if they were sent by the last relay.

keep-timestamp()yes or noType:yesDefault:

106syslog-ng.com

network() source options

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.16. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

max-connections()number (simultaneous connections)Type:10Default:

107syslog-ng.com

network() source options

Description: Specifies the maximum number of simultaneous connections.

multi-line-garbage()regular expressionType:empty stringDefault:

Description: Use the multi-line-garbage() option when processing multi-line messages that containunneeded parts between the messages. Specify a string or regular expression that matches the beginning of theunneeded message parts. If the multi-line-garbage() option is set, syslog-ng PE ignores the lines betweenthe line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See alsothe multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but nomatching line is received between two lines that match multi-line-prefix(), syslog-ng PE will continueto process the incoming lines as a single message until a line matching multi-line-garbage() is received.

WarningIf the multi-line-garbage() option is set, syslog-ng PE discards lines between the line matching themulti-line-garbage() and the next line matching multi-line-prefix().

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

multi-line-prefix()regular expressionType:empty stringDefault:

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messagesthat contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matchesthe beginning of the log messages. Use as simple regular expressions as possible, because complex regularexpressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix()

option is set, syslog-ng PE ignores newline characters from the source until a line matches the regular expressionagain, and treats the lines between the matching lines as a single message. See also the multi-line-garbage()option.

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

108syslog-ng.com

network() source options

TipTo make multi-line messages more readable when written to a file, use a template in the destination andinstead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). Thisexpression inserts a tab after every newline character (except when a tab is already present), indenting everyline of the message after the first. For example:

■

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );

};

For details on using templates, see Section 14.1.2, Templates and macros (p. 379).

■ To actually convert the lines of multi-line messages to single line (by replacing the newline characters withwhitespaces), use the flags(no-multi-line) option in the source.

Example 6.17. Processing Tomcat logsThe log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages startwith the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

SEVERE: Catalina.start:

LifecycleException: service.getName(): "Catalina"; Protocol handler start failed:

java.net.BindException: Address already in use<null>:8080

at org.apache.catalina.connector.Connector.start(Connector.java:1138)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:583)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

INFO: Server startup in 1206 ms

2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in themulti-line-prefix() option. Such an expression is the following:

source s_file{ file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0)

multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));

};

Note that the flags(no-parse) is needed to avoid syslog-ng PE trying to interpret the date in the message.

WarningIf you receive messages using the UDP protocol, do not use multi-line processing. If every line of a multi-line messageis received in the same UDP packet, everything is fine, but if a multi-line message is fragmented into multiple UDPpackets, the order they are received (thus the way how they are processed) cannot be guaranteed, and causes problems.

109syslog-ng.com

network() source options

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

port() or localport()number (port number)Type:In case of TCP transport: 601

In case of UDP transport: 514

Default:

Description: The port number to bind to.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

so-broadcast()yes or noType:noDefault:

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messagesto a broadcast address. For details, see the socket(7) manual page.

so-keepalive()yes or noType:noDefault:

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

110syslog-ng.com

network() source options

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

so-sndbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

transport()rltp, udp, tcp, or tlsType:tcpDefault:

111syslog-ng.com

network() source options

Description: Specifies the protocol used to receive messages from the source.

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

tls()tls optionsType:n/aDefault:

Description: This option sets various options related to TLS encryption, for example, key/certificate files andtrusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4,TLS options (p. 356).

use-dns()yes, no, persist_onlyType:yesDefault:

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locallyfrom file (for example from /etc/hosts). The syslog-ng PE application blocks on DNS queries, so enablingDNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint withfirewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can bespecified globally, and per-source as well. The local setting of the source overrides the global option if available.

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

use-fqdn()yes or noType:noDefault:

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specifiedglobally, and per-source as well. The local setting of the source overrides the global option if available.

112syslog-ng.com

network() source options

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.6. Collecting messages from named pipes

The pipe driver opens a named pipe with the specified name and listens for messages. It is used as the nativemessage delivery protocol on HP-UX.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. For the list ofavailable optional parameters, see Section 6.6.1, pipe() source options (p. 114).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

pipe(filename);

NoteAs of syslog-ng Open Source Edition 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipeusing the mkfifo(1) command.

Pipe is very similar to the file() driver, but there are a few differences, for example pipe() opens its argumentin read-write mode, therefore it is not recommended to be used on special files like /proc/kmsg.

WarningIt is not recommended to use pipe() on anything else than real pipes.■

■ By default, syslog-ng PE uses the flags(no-hostname) option for pipes, meaning that syslog-ng PEassumes that the log messages received from the pipe do not contain the hostname field. If your messagesdo contain the hostname field, use flags(expect-hostname). For details, see Section flags() (p. 114).

113syslog-ng.com

Collecting messages from named pipes

Example 6.18. Using the pipe() driver

source s_pipe { pipe("/dev/pipe" pad-size(2048)); };

6.6.1. pipe() source options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

The pipe driver has the following options:

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

114syslog-ng.com

pipe() source options

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

follow-freq()number (seconds)Type:1Default:

Description: Indicates that the source should be checked periodically. This is useful for files which alwaysindicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng willnot attempt to use poll() on the file, but checks whether the file changed every time the follow-freq()

interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

115syslog-ng.com

pipe() source options

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.19. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

log-prefix() (DEPRECATED)stringType:

Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string toany log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE:This option is deprecated. Use program-override() instead.

116syslog-ng.com

pipe() source options

optional()yes or noType:

Default:

Description: Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attemptsto initialize the source will be made until the configuration is reloaded. This option currently applies to thepipe(), unix-dgram, and unix-stream drivers.

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

pipe()filename with pathType:

Default:

Description: The filename of the pipe to read messages from.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

117syslog-ng.com

pipe() source options

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.7. Receiving messages from external applications

The program driver starts an external application and reads messages from the standard output (stdout) of theapplication. It is mainly useful to receive log messages from daemons that accept incoming messages andconvert them to log messages.

The program driver has a single required parameter, specifying the name of the application to start.

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

program(filename);

Example 6.20. Using the program() driver

source s_program { program("/etc/init.d/mydaemon"); };

NoteThe program is restarted automatically if it exits.

118syslog-ng.com

Receiving messages from external applications

6.7.1. program() source options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

The program() driver has the following options:

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

119syslog-ng.com

program() source options

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

120syslog-ng.com

program() source options

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.21. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

log-prefix() (DEPRECATED)stringType:

Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string toany log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE:This option is deprecated. Use program-override() instead.

optional()yes or noType:

Default:

Description: Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attemptsto initialize the source will be made until the configuration is reloaded. This option currently applies to thepipe(), unix-dgram, and unix-stream drivers.

pad-size()number (bytes)Type:0Default:

121syslog-ng.com

program() source options

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

program()filename with pathType:

Default:

Description: The name of the application to start and read messages from.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

122syslog-ng.com

program() source options

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.8. Collecting messages from tables or relational database

The sql() driver collects messages from an SQL database. Currently the Microsoft SQL (MSSQL), MySQL,Oracle and PostgreSQL databases are supported.

The sql() driver has the following required parameters: database(), host(), table(), type(),uid-column(), datetime-column() or date-column() and time-column().

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

sql(options);

Note that the sql() source driver has the following restrictions and limitations:

■ The sql() source driver does not monitor rotated tables. Therefore, every source can follow onlyone table.

■ Timestamps columns with timezone are not supported. The syslog-ng PE application will retrievethe timestamp from these columns, but without the timezone information.

■ The sql() source driver ignores the log-msg-size() option, that is, messages read from the SQLdatabase can be longer than the maximal message length set in the log-msg-size() option.

■ There is an ID column that is the monotonically increasing unique ID of the monitored table. It isnot possible to use more than one ID column as complex ID.

Example 6.22. Using a MySQL sourceWith the following configuration, syslog-ng PE reads the records from the test_log table ordered by the id column.

The test_log table has the following structure:

CREATE TABLE “test_log” (

“id” BIGINT NOT NULL AUTO_INCREMENT,

“timestamp” TIMESTAMP NOT NULL,

“host” VARCHAR(64) NOT NULL,

“program” VARCHAR(64) NOT NULL,

“log” VARCHAR(1024) NOT NULL

);

The matching syslog-ng PE configuration that reads this table is the following:

123syslog-ng.com

Collecting messages from tables or relational database

source s_mysql { sql(

type("mysql")

host("10.140.32.29")

username("user")

password("secret")

database("test_db")

table("test_log")

datetime-column("timestamp")

uid-column("id")

message-template("${.sql.id} ${.sql.log}")

program-template("${.sql.program}")

host-template("${.sql.host}")

fast-follow-mode(yes)

);};

6.8.1. Supported SQL sources by platform

NoteIf a particular database is not supported on a platform you are using it, use the sql() source on a different syslog-ng PEhost to retrieve the records remotely.

OraclePgSQLMySQLMSSQLPlatform \ Database-✔✔✔AIX

-✔✔✔FreeBSD 8

-✔✔✔FreeBSD 9

-✔✔✔FreeBSD 10

--✔✔HP-UX 11v2_onIA64

✔✔✔✔L i n u x(linux_glibc236):CentOS 5

✔✔✔✔L i n u x(linux_glibc236):CentOS 6

✔✔✔✔L i n u x(linux_glibc236):CentOS 7

✔✔✔✔L i n u x(linux_glibc236):Debian 7

✔✔✔✔L i n u x(linux_glibc236):openSUSE 11

124syslog-ng.com

Supported SQL sources by platform

OraclePgSQLMySQLMSSQLPlatform \ Database✔✔✔✔L i n u x

(linux_glibc236):Oracle Linux 5

✔✔✔✔L i n u x(linux_glibc236):Oracle Linux 6

✔✔✔✔L i n u x(linux_glibc236):Oracle Linux 7

✔✔✔✔L i n u x(linux_glibc236):SLES 11

✔✔✔✔L i n u x(linux_glibc236):Red Hat EL 5

✔✔✔✔L i n u x(linux_glibc236):Red Hat EL 6

✔✔✔✔L i n u x(linux_glibc236):Red Hat EL 7

✔✔✔✔L i n u x(linux_glibc236):Ubuntu 12.04

✔✔✔✔L i n u x(linux_glibc236):Ubuntu 14.04

✔✔✔✔Solaris 10 on SPARCand SPARC64

✔✔✔✔Solaris 11 on SPARCand SPARC64

✔-✔✔Solaris 10 on x86_64

✔-✔✔Solaris 11 on x86_64Table 6.3. Supported SQL sources by platform

6.8.2. sql() source options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

125syslog-ng.com

sql() source options

The sql() driver has the following options.

archive-query()stringType:

Default:

Description: The SQL-like statement which is executed after syslog-ng PE has queried a batch of records (asset in the log-fetch-limit() option). This statement can be used for example to archive or delete the recordsprocessed by syslog-ng PE. Note that the user account that syslog-ng PE uses to access the database requiresthe appropriate privileges to execute the statement. If executing the statement fails, syslog-ng PE will log theerror message returned by the database, and continue processing the other records.

For details on customizing queries, see Section 6.8.3, Customizing SQL queries (p. 134).

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

Example 6.23. A sample archive queryThe following statement deletes the records already retreived from the database if the table is read from the beginning.

archive-query("DELETE FROM test_logs WHERE id <= $last_read_uid")

columns()string listType:emptyDefault:

Description: The list of the name of the columns that will be queried. The default value is empty, meaning thatall of the columns will be queried.

Example 6.24. SQL source option columnscolumns("id","date","message")

connect-query()stringType:

Default:

Description: The SQL-like statement which is executed after syslog-ng PE has successfully connected to thedatabase.

For details on customizing queries, see Section 6.8.3, Customizing SQL queries (p. 134).

126syslog-ng.com

sql() source options

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

Example 6.25. A sample connect query

connect-query("SET COLLATION_CONNECTION='utf8_general_ci'")

database()stringType:logsDefault:

Description: Name of the database that stores the logs. Macros cannot be used in database name. Also, whenusing an Oracle database, you cannot use the same database() settings in more than one destination.

date-column(col_name, [format])date, stringType:

Default:

Description: The column containing the date of the logrecord. The format value has to be in strptime format.For details, see the strptime manual page (man strptime).

NoteIf the type of the column is string, this is a required parameter.

datetime-column(col_name, [format])stringType:

Default:

The following column types are supported:

■ MySQL: timestamp, datetime, int

■ PostgreSQL: timestamp, int

■ Oracle: timestamp, int

■ MSSQL: datetime, int

Description: The column containing the timestamp. If the type is int, it is considered to contain a UNIXtimestamp. The format value is required if the type is string, and has to be in strptime format. For details, seethe strptime manual page (man strptime).

127syslog-ng.com

sql() source options

Example 6.26. SQL source option datetime-column(col_name, [format])datetime("timestampcol", "%Y-%m-%d")

default-facility()facility stringType:local0Default:

Description: This parameter assigns a facility value to the messages received from the sql source.

default-priority()priority stringType:infoDefault:

Description: This parameter assigns an emergency level to the messages received from the sql source.

fast-follow-mode()yes|noType:yesDefault:

Description: If set to yes, syslog-ng PE reads the database table as fast as possible, until it reaches the lastrecord. After this, it will execute only one query in follow-freq() time. If it is set to no, syslog-ng PEexecutes only one query in follow-freq() time.

fetch-query()stringType:Default value is generated in running time of syslog-ngDefault:

Description: The SQL-like statement used to collect the records from the database.

NoteIf this parameter is defined, syslog-ng PE does not check or validate it whether it is correct. Ensure that the customizedstatements are correct.

For details on customizing queries, see Section 6.8.3, Customizing SQL queries (p. 134).

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

128syslog-ng.com

sql() source options

Example 6.27. A sample fetch query

fetch-query("SELECT * FROM $table WHERE id > $last_read_uid AND test_logs.log LIKE '%ERROR%'

ORDER BY $uid")

The default fetch queries are the following:

■ MSSQL:

SELECT TOP $fetch_limit $columns FROM $table WHERE $uid > '$last_read_uid'

ORDER BY $uid

■ MySQL:

SELECT $columns FROM $table WHERE $uid > '$last_read_uid' ORDER BY $uid

LIMIT 0,$fetch_limit

■ Oracle:

SELECT $columns FROM (SELECT $table.*, Row_Number() OVER (ORDER BY $uid)

FetchRow FROM $table WHERE $uid > '$last_read_uid' ) WHERE FetchRow BETWEEN

0 AND $fetch_limit

■ PostgreSQL:

SELECT $columns FROM $table WHERE $uid > '$last_read_uid' ORDER BY $uid

LIMIT $fetch_limit

follow-freq()number (seconds)Type:10Default:

Indicates that the source should be checked periodically. This is useful for SQL sources which always indicatereadability, even though no new records were appended. If this value is higher than zero, syslog-ng will notattempt to use poll() on the SQL source, but checks whether the SQL source changed every time thefollow-freq() interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used aswell.

host()hostname or IP addressType:n/aDefault:

Description: Hostname of the database server. Note that Oracle destinations do not use this parameter, butretrieve the hostname from the /etc/tnsnames.ora file.

129syslog-ng.com

sql() source options

NoteIf you specify host="localhost", syslog-ng will use a socket to connect to the local database server. Usehost="127.0.0.1" to force TCP communication between syslog-ng and the local database server.

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for exampleMYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

host-template()stringType:Empty stringDefault:

Description: The template for defining the HOST part of the message. If the host-template() option is notspecified, the value of the host() option will be used in the HOST part of the message.

NoteThis option requires the option keep-hostname() to be enabled: keep-hostname(yes).

log-fetch-limit()Type:

100Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.28. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

max-uid-query()stringType:"SELECT max($uid) FROM $table"Default:

130syslog-ng.com

sql() source options

Description: Used for retrieving the ID of the last row (that is, the last row of the last fetch) from the databasesource.

NoteIf this parameter is defined, syslog-ng PE does not check or validate whether it is correct. Ensure that the customizedstatements are correct.

This option is mandatory if read-old-records(no) and fetch-query() is defined.

NoteWhen fetch-query() is defined, the default value will be:

"SELECT MAX($uid) FROM ( <fetch-query()> )"

Following a restart or reload of syslog-ng PE, if no new data has arrived since the last fetch, the custom query defined infetch-query() will fetch everything starting from the very beginning. To avoid this and retrieve only the ID of thelast row, replace the above value with something similar:

"SELECT max($uid) FROM $table"

This option cannot be used if read-old-records(yes).

message-template()stringType:

Default:

Description: The alias of the template() parameter.

password()stringType:n/aDefault:

Description: Password of the database user.

port()number (port number)Type:1433 TCP for MSSQL, 3306 TCP for MySQL, 1521 for Oracle, and 5432 TCP for PostgreSQLDefault:

Description: The port number to connect to.

prefix()stringType:".sql"Default:

Description: This prefix will be added to the name of the macros created from the database columns.

131syslog-ng.com

sql() source options

Example 6.29. SQL source option prefix()If a database column is called column1, and the prefix option is set as prefix("customprefix."), the macro for thecolumn will be called customprefix.column1.

program-template()stringType:Empty stringDefault:

Description: The template for defining the PROGRAM part of the message. If not specified, the PROGRAM

message part will be empty.

read-old-records()yes|noType:noDefault:

Description: If set to yes, syslog-ng PE will start reading the records from the beginning of the table, if thetable has not been read yet. If set to no, syslog-ng PE will read only the new records.

table()stringType:

Default:

Description: The name of the monitored table. Only a single literal name is accepted, macros cannot be usedin the name of the table. Monitoring rotated tables is not supported.

table-init-query()stringType:

Default:

Description: The SQL-like statement which is executed before fetching the first batch of records.

For details on customizing queries, see Section 6.8.3, Customizing SQL queries (p. 134).

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

tags()stringType:

Default:

132syslog-ng.com

sql() source options

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

template()stringType:

Default:

Description: The template of the message (${MSG}) to be generated. If not specified, the following templatewill be used: "$(format-welf --key <prefix>*)")where <prefix> is the value of the prefix() option.This template converts the retrieved records into the WebTrends Enhanced Log file Format (WELF).

■ For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

■ For details on the format-welf() template function, see Section format-welf (p. 394).

NoteThe format-welf function does not keep the order of columns between queries.

Example 6.30. SQL source option template()Using the default template for a table that has two columns (id and message) The result of the first query is the following:'.sql.id=12 .sql.message="test message"', whereas the result of the second query can be:'.sql.message="test message" .sql.id=12 '

time-column(col_name, [format])time, stringType:

Default:

Description: The column containing the time of the logrecord. The format value has to be in strptime format.

NoteIf the type of the column is string, this is a required parameter.

time-reopen()number (seconds)Accepted values:60Default:

Description: The time to wait in seconds before a dead connection is reestablished.

133syslog-ng.com

sql() source options

time-zone()timezone in +/-HH:MM formatType:

Default:

Description: The default timezone, if set. If this option is not set, the default timezone is the local timezone.

type()mssql, mysql, oracle, or pgsqlType:

Default:

Description: Specifies the type of the database, that is, the DBI database driver to use. Use the mssql optionto send logs to an MSSQL database. For details, see the examples of the databases on the following sections.

uid-column()stringType:

Default:

Description: The monotonically increasing unique ID of the monitored table (for example auto_increment).This column must be a type where the greater (>) operation is interpreted.

NoteThe value of the first record of this column should not be 0: syslog-ng PE will skip this value.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

username()stringType:n/aDefault:

Description: Name of the database user.

6.8.3. Customizing SQL queries

Every query executed by the SQL source can be customized. These customized queries are similar to SQLstatements, but in can also refer to syslog-ng PE-specific variables with the prefix $. For example, $table isthe name of the table.

134syslog-ng.com

Customizing SQL queries

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

The available variables are the following:

NoteThe variables in the statement are not macro references.

■ $uid: The name of the uid-column().

■ $table: The name of the table() option.

■ $last_read_uid: The uid value of the last read record.

■ $columns: Reach the columns() option of the SQL source (the default is *). If columns is definedin the configuration (for example columns("id" "message")), then the value of this variablewill be a comma separated list (for example "id,message").

■ $fetch_limit: Reach the value of the log-fetch-limit() option.

Example 6.31. SQL source fetch-queryThe following queries records that are older than the last record:

SELECT * FROM $table WHERE $table.$uid > $last_read_uid ORDER BY $table.$uid

6.9. Collecting messages on Sun Solaris

Solaris uses its STREAMS framework to send messages to the syslogd process. Solaris 2.5.1 and above usesan IPC called door in addition to STREAMS, to confirm the delivery of a message. The syslog-ng applicationsupports the IPC mechanism via the door() option (see below).

NoteThe sun-streams() driver must be enabled when the syslog-ng application is compiled (see ./configure --help).

The sun-streams() driver has a single required argument specifying the STREAMS device to open, and thedoor() option. For the list of available optional parameters, see Section 6.9.1, sun-streams() sourceoptions (p. 136).

135syslog-ng.com

Collecting messages on Sun Solaris

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

sun-streams(<name_of_the_streams_device> door(<filename_of_the_door>));

Example 6.32. Using the sun-streams() driver

source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door")); };

6.9.1. sun-streams() source options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

The sun-streams() driver has the following options.

door()stringType:noneDefault:

Description: Specifies the filename of a door to open, needed on Solaris above 2.5.1.

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

136syslog-ng.com

sun-streams() source options

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

137syslog-ng.com

sun-streams() source options

follow-freq()number (seconds)Type:1Default:

Description: Indicates that the source should be checked periodically. This is useful for files which alwaysindicate readability, even though no new lines were appended. If this value is higher than zero, syslog-ng willnot attempt to use poll() on the file, but checks whether the file changed every time the follow-freq()

interval (in seconds) has elapsed. Floating-point numbers (for example 1.5) can be used as well.

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.33. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

138syslog-ng.com

sun-streams() source options

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

log-prefix() (DEPRECATED)stringType:

Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string toany log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE:This option is deprecated. Use program-override() instead.

optional()yes or noType:

Default:

Description: Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attemptsto initialize the source will be made until the configuration is reloaded. This option currently applies to thepipe(), unix-dgram, and unix-stream drivers.

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

139syslog-ng.com

sun-streams() source options

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.10. Collecting messages using the IETF syslog protocol (syslog() driver)

The syslog() driver can receive messages from the network using the standard IETF-syslog protocol (asdescribed in RFC5424-26). UDP, TCP, and TLS-encrypted TCP can all be used to transport the messages.

NoteThe syslog() driver can also receive BSD-syslog-formatted messages (described in RFC 3164, see Section 2.10.1,BSD-syslog or legacy-syslog messages (p. 19)) if they are sent using the IETF-syslog protocol.

In syslog-ng PE versions 3.1 and earlier, the syslog() driver could handle only messages in the IETF-syslog (RFC5424-26) format.

For the list of available optional parameters, see Section 6.10.1, syslog() source options (p. 141).

Declaration:

syslog(ip() port() transport() options());

140syslog-ng.com

Collecting messages using the IETF syslog protocol (syslog() driver)

Example 6.34. Using the syslog() driverTCP source listening on the localhost on port 1999.

source s_syslog { syslog(ip(127.0.0.1) port(1999) transport("tcp")); };

UDP source with defaults.

source s_udp { syslog( transport("udp")); };

Encrypted source where the client is also authenticated. For details on the encryption settings, see Section 10.4, TLSoptions (p. 356).

source s_syslog_tls{ syslog(

ip(10.100.20.40)

transport("tls")

tls(

peer-verify(required-trusted)

ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')

key-file('/opt/syslog-ng/etc/syslog-ng/keys/server_privatekey.pem')

cert-file('/opt/syslog-ng/etc/syslog-ng/keys/server_certificate.pem')

)

);};

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

6.10.1. syslog() source options

The syslog() driver has the following options.

encoding()stringType:

Default:

Description: Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslogprotocol. To list the available character sets on a host, execute the iconv -l command. For details on howencoding affects the size of the message, see Section Message size and encoding (p. 25).

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

141syslog-ng.com

syslog() source options

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

142syslog-ng.com

syslog() source options

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

■ threaded: The threaded flag enables multithreading for the source. For details on multithreading,see Chapter 18, Multithreading and scaling in syslog-ng PE (p. 449).

NoteThe syslog source uses multiple threads only if the source uses the tls or tcp transport protocols.

host-override()stringType:

Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

ip() or localip()stringType:0.0.0.0Default:

Description: The IP address to bind to. By default, syslog-ng PE listens on every available interface. Note thatthis is not the address where messages are accepted from.

If you specify a multicast bind address and use the udp transport, syslog-ng PE automatically joins the necessarymulticast group. TCP does not support multicasting.

ip-protocol()number (IP version)Type:4Default:

Description: Determines the internet protocol version of the given driver (network() or syslog()). Thepossible values are 4 and 6, corresponding to IPv4 and IPv6. The default value is 4.

ip-tos()number (type of service)Type:0Default:

Description: Specifies the Type-of-Service value of outgoing packets.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

143syslog-ng.com

syslog() source options

ip-ttl()number (hops)Type:0Default:

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()yes or noType:yesDefault:

Description: Specifies whether connections to sources should be closed when syslog-ng is forced to reload itsconfiguration (upon the receipt of a SIGHUP signal). Note that this applies to the server (source) side of thesyslog-ng connections, client-side (destination) connections are always reopened after receiving a HUP signalunless the keep-alive option is enabled for the destination.

keep-hostname()yes or noType:noDefault:

Description: Enable or disable hostname rewriting.

■ If enabled (keep-hostname(yes)), syslog-ng PE assumes that the incoming log message was sentby the host specified in the HOST field of the message.

■ If disabled (keep-hostname(no)), syslog-ng PE rewrites the HOST field of the message, either tothe IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns()

parameter is set to yes and the IP address can be resolved to a hostname) of the host sending themessage to syslog-ng PE. For details on using name resolution in syslog-ng PE, see Section 20.3,Using name resolution in syslog-ng (p. 461).

NoteIf the log message does not contain a hostname in its HOST field, syslog-ng PE automatically adds a hostname to themessage.

For messages received from the network, this hostname is the address of the host that sent the message (thismeans the address of the last hop if the message was transferred via a relay).

■

■ For messages received from the local host, syslog-ng PE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the globaloption if available.

NoteWhen relaying messages, enable this option on the syslog-ng PE server and also on every relay, otherwise syslog-ng PEwill treat incoming messages as if they were sent by the last relay.

144syslog-ng.com

syslog() source options

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

Example 6.35. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

145syslog-ng.com

syslog() source options

max-connections()number (simultaneous connections)Type:10Default:

Description: Specifies the maximum number of simultaneous connections.

multi-line-garbage()regular expressionType:empty stringDefault:

Description: Use the multi-line-garbage() option when processing multi-line messages that containunneeded parts between the messages. Specify a string or regular expression that matches the beginning of theunneeded message parts. If the multi-line-garbage() option is set, syslog-ng PE ignores the lines betweenthe line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See alsothe multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but nomatching line is received between two lines that match multi-line-prefix(), syslog-ng PE will continueto process the incoming lines as a single message until a line matching multi-line-garbage() is received.

WarningIf the multi-line-garbage() option is set, syslog-ng PE discards lines between the line matching themulti-line-garbage() and the next line matching multi-line-prefix().

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

multi-line-prefix()regular expressionType:empty stringDefault:

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messagesthat contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matchesthe beginning of the log messages. Use as simple regular expressions as possible, because complex regularexpressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix()

option is set, syslog-ng PE ignores newline characters from the source until a line matches the regular expressionagain, and treats the lines between the matching lines as a single message. See also the multi-line-garbage()option.

146syslog-ng.com

syslog() source options

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

TipTo make multi-line messages more readable when written to a file, use a template in the destination andinstead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). Thisexpression inserts a tab after every newline character (except when a tab is already present), indenting everyline of the message after the first. For example:

■

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );

};

For details on using templates, see Section 14.1.2, Templates and macros (p. 379).

■ To actually convert the lines of multi-line messages to single line (by replacing the newline characters withwhitespaces), use the flags(no-multi-line) option in the source.

Example 6.36. Processing Tomcat logsThe log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages startwith the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

SEVERE: Catalina.start:

LifecycleException: service.getName(): "Catalina"; Protocol handler start failed:

java.net.BindException: Address already in use<null>:8080

at org.apache.catalina.connector.Connector.start(Connector.java:1138)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:583)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

INFO: Server startup in 1206 ms

2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in themulti-line-prefix() option. Such an expression is the following:

source s_file{ file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0)

multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));

};

Note that the flags(no-parse) is needed to avoid syslog-ng PE trying to interpret the date in the message.

147syslog-ng.com

syslog() source options

WarningIf you receive messages using the UDP protocol, do not use multi-line processing. If every line of a multi-line messageis received in the same UDP packet, everything is fine, but if a multi-line message is fragmented into multiple UDPpackets, the order they are received (thus the way how they are processed) cannot be guaranteed, and causes problems.

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

Padding was set, and couldn't read enough bytes

port() or localport()number (port number)Type:In case of TCP transport: 601

In case of UDP transport: 514

Default:

Description: The port number to bind to.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

so-broadcast()yes or noType:noDefault:

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messagesto a broadcast address. For details, see the socket(7) manual page.

so-keepalive()yes or noType:noDefault:

148syslog-ng.com

syslog() source options

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

so-sndbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

tags()stringType:

Default:

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

149syslog-ng.com

syslog() source options

transport()rltp, udp, tcp, or tlsType:tcpDefault:

Description: Specifies the protocol used to receive messages from the source.

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

tls()tls optionsType:n/aDefault:

Description: This option sets various options related to TLS encryption, for example, key/certificate files andtrusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4,TLS options (p. 356).

use-dns()yes, no, persist_onlyType:yesDefault:

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locallyfrom file (for example from /etc/hosts). The syslog-ng PE application blocks on DNS queries, so enablingDNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint withfirewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can bespecified globally, and per-source as well. The local setting of the source overrides the global option if available.

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

use-fqdn()yes or noType:noDefault:

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specifiedglobally, and per-source as well. The local setting of the source overrides the global option if available.

150syslog-ng.com

syslog() source options

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

6.11. Collecting the system-specific log messages of a platform

Starting with version 4 F1, syslog-ng PE can automatically collect the system-specific log messages of the hoston a number of platforms using the system() driver. If the system() driver is included in the syslog-ng PEconfiguration file, syslog-ng PE automatically adds the following sources to the syslog-ng PE configuration.

Notesyslog-ng PE versions 4.1-5.0 used an external script to generate the system() source, but this was problematic in certainsituations, for example, when the host used a strict AppArmor profile. Therefore, the system() source is now generatedinternally in syslog-ng PE.

The system() driver is also used in the default configuration file of syslog-ng PE. For details on the defaultconfiguration file, see Example 4.1, The default configuration file of syslog-ng PE (p. 69).

WarningIf syslog-ng PE does not recognize the platform it is installed on, it does not add any sources.

Message sourcePlatformunix-dgram("/dev/log");AIX and Tru64

unix-dgram("/var/run/log");FreeBSD

unix-dgram("/var/run/logpriv"

perm(0600));

file("/dev/klog" follow-freq(0)

program-override("kernel")

flags(no-parse));

pipe("/dev/log" pad-size(2048));HP-UX

unix-dgram("/dev/log");Linux

151syslog-ng.com

Collecting the system-specific log messages of a platform

Message sourcePlatformfile("/proc/kmsg"

program-override("kernel")

flags(kernel));

sun-streams("/dev/log");Solaris 8

sun-streams("/dev/log"

door("/etc/.syslog_door"));

Solaris 9

sun-streams("/dev/log"

door("/var/run/syslog_door"));

Solaris 10

Table 6.4. Sources automatically added by syslog-ng Premium Edition

6.12. Collecting messages from the systemd-journal system log storage

The systemd-journal() source is used on various Linux distributions, such as RHEL (from RHEL7) andCentOS. The systemd-journal() source driver can read the structured name-value format of the journaldsystem service, making it easier to reach the custom fields in the message.

The systemd-journal() source driver is designed to read only local messages through the systemd-journalAPI. It is not possible to set the location of the journal files, or the directories.

NoteThe log-msg-size() option is not applicable for this source. Use the max-field-size() option instead.

NoteThis source will not handle the following cases:

corrupted journal file■

■ incorrect journal configuration

■ any other journald-related bugs

NoteIf you are using RHEL-7, the default source in the configuration is systemd-journal() instead ofunix-dgram("/dev/log") and file("/proc/kmsg"). If you are using unix-dgram("/dev/log") orunix-stream("/dev/log") in your configuration as a source, syslog-ng PE will revert to using systemd-journal()instead.

WarningOnly one systemd-journal() source can be configured in the configuration file. If there are more than onesystemd-journal() sources configured, syslog-ng PE will not start.

Declaration:

152syslog-ng.com

Collecting messages from the systemd-journal system log storage

systemd-journal(options);

Example 6.37. Sending all fields through syslog protocol using the systemd-journal() driverTo send all fields through the syslog protocol, enter the prefix in the following format: ".SDATA.<name>".

@version: 6.0

source s_journald {

systemd-journal(prefix(".SDATA.journald."));

};

destination d_network {

syslog("server.host");

};

log {

source(s_journald);

destination(d_network);

};

Example 6.38. Filtering for a specific field using the systemd-journal() driver

@version: 6.0

source s_journald {

systemd-journal(prefix(".SDATA.journald."));

};

filter f_uid {"${.SDATA.journald._UID}" eq "1000"};

destination d_network {

syslog("server.host");

};

log {

source(s_journald);

filter(f_uid);

destination(d_network);

};

Example 6.39. Sending all fields in value-pairs using the systemd-journal() driver

@version: 6.0

source s_local {

systemd-journal(prefix("journald."));

};

destination d_network {

network("server.host" template("$(format_json --scope rfc5424 --key journald.*)\n"));

};

log {

source(s_local);

destination(d_network);

};

The journal contains credential information about the process that sent the log message. The syslog-ng PEapplication makes this information available in the following macros:

153syslog-ng.com

Collecting messages from the systemd-journal system log storage

syslog-ng predefined macroJournald field$MESSAGEMESSAGE

$HOST_HOSTNAME

$PID_PID

$PROGRAM_COMM if does not exist SYSLOG_IDENTIFIER

$FACILITY_NUMSYSLOG_FACILITY

$LEVEL_NUMPRIORITY

6.12.1. systemd-journal() source options

The systemd-journal() driver has the following options:

default-facility()facility stringType:local0Default:

Description: The default facility value if the SYSLOG_FACILITY entry does not exist.

default-level()stringType:noticeDefault:

Description: The default level value if the PRIORITY entry does not exist.

host-override()stringType:

Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

keep-hostname()yes or noType:noDefault:

Description: Enable or disable hostname rewriting.

■ If enabled (keep-hostname(yes)), syslog-ng PE will retain the hostname information read fromthe systemd journal messages.

■ If disabled (keep-hostname(no)), syslog-ng PE will use the hostname that has been set up for theoperating system instance that syslog-ng is running on. To query or set this value, use thehostnamectl command.

154syslog-ng.com

systemd-journal() source options

This option can be specified globally, and per-source as well. The local setting of the source overrides the globaloption if available.

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

max-field-size()number (characters)Type:65536Default:

Description: The maximum length of a field's value.

prefix()stringType:

Default:

Description: If this option is set, every non-built-in mapped names get a prefix (for example:".SDATA.journald.").

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

use-fqdn()yes or noType:noDefault:

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specifiedglobally, and per-source as well. The local setting of the source overrides the global option if available.

155syslog-ng.com

systemd-journal() source options

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

6.13. Collecting systemd messages using a socket

On platforms running systemd, the systemd-syslog() driver reads the log messages of systemd using the/run/systemd/journal/syslog socket. Note the following points about this driver:

■ If possible, use the more reliable systemd-journal() driver instead.

■ The socket activation of systemd is buggy, causing some log messages to get lost during systemstartup.

■ If syslog-ng PE is running in a jail or a Linux Container (LXC), it will not read from the /dev/kmsgor /proc/kmsg files.

Declaration:

systemd-syslog();

Example 6.40. Using the systemd-syslog() driver

@version: 6.0

source s_systemdd {

systemd-syslog();

};

destination d_network {

syslog("server.host");

};

log {

source(s_systemdd);

destination(d_network);

};

6.14. Collecting messages from remote hosts using the BSD syslog protocol

NoteThe tcp(), tcp6(), udp(), and udp6() drivers are obsolete. Use the network() source and the network() destinationinstead. For details, see Section 6.5, Collecting messages using the RFC3164 protocol (network() driver) (p. 101) andSection 7.8, Sending messages to a remote log server using the RFC3164 protocol (network() driver) (p. 234), respectively.

The tcp(), tcp6(), udp(), udp6() drivers can receive syslog messages conforming to RFC3164 from thenetwork using the TCP and UDP networking protocols. The tcp6() and udp6() drivers use the IPv6 networkprotocol, while tcp() and udp() use IPv4.

To convert your existing tcp(), tcp6(), udp(), udp6() source drivers to use the network() driver, seeProcedure 6.14.1.1, Change an old source driver to the network() driver (p. 157).

156syslog-ng.com

Collecting systemd messages using a socket

6.14.1. tcp(), tcp6(), udp() and udp6() source options — OBSOLETE

NoteThe tcp(), tcp6(), udp(), and udp6() drivers are obsolete. Use the network() source and the network() destinationinstead. For details, see Section 6.5, Collecting messages using the RFC3164 protocol (network() driver) (p. 101) andSection 7.8, Sending messages to a remote log server using the RFC3164 protocol (network() driver) (p. 234), respectively.

To convert your existing tcp(), tcp6(), udp(), udp6() source drivers to use the network() driver, seeProcedure 6.14.1.1, Change an old source driver to the network() driver (p. 157).

6.14.1.1. Procedure – Change an old source driver to the network() driver

To replace your existing tcp(), tcp6(), udp(), udp6() sources with a network() source, complete thefollowing steps.

Step 1. Replace the driver with network. For example, replace udp( with network(

Step 2. Set the transport protocol.

■ If you used TLS-encryption, add the transport("tls") option, then continue with thenext step.

■ If you used the tcp or tcp6 driver, add the transport("tcp") option.

■ If you used the udp or udp driver, add the transport("udp") option.

Step 3. If you use IPv6 (that is, the udp6 or tcp6 driver), add the ip-protocol("6") option.

Step 4. If you did not specify the port used in the old driver, check Section 6.5.1, network() sourceoptions (p. 103) and verify that your clients send the messages to the default port of the transport protocolyou use. Otherwise, set the appropriate port number in your source using the port() option.

Step 5. All other options are identical. Test your configuration with the syslog-ng --syntax-only

command.The following configuration shows a simple tcp source.

source s_old_tcp {

tcp(

ip(127.0.0.1) port(1999)

tls(

peer-verify("required-trusted")

key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")

cert-file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt')

)

);

};

When replaced with the network() driver, it looks like this.

source s_new_network_tcp {

network(

transport("tls")

ip(127.0.0.1) port(1999)

157syslog-ng.com

tcp(), tcp6(), udp() and udp6() source options — OBSOLETE

tls(

peer-verify("required-trusted")

key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")

cert-file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt')

)

);

};

6.15. Collecting messages from UNIX domain sockets

The unix-stream() and unix-dgram() drivers open an AF_UNIX socket and start listening on it for messages.The unix-stream() driver is primarily used on Linux and uses SOCK_STREAM semantics (connection oriented,no messages are lost), while unix-dgram() is used on BSDs and uses SOCK_DGRAM semantics: this may resultin lost local messages if the system is overloaded.

To avoid denial of service attacks when using connection-oriented protocols, the number of simultaneouslyaccepted connections should be limited. This can be achieved using the max-connections() parameter. Thedefault value of this parameter is quite strict, you might have to increase it on a busy system.

Both unix-stream and unix-dgram have a single required argument that specifies the filename of the socket tocreate. For the list of available optional parameters, see Section 6.15.1, unix-stream() and unix-dgram() sourceoptions (p. 159)

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

unix-stream(filename [options]);

unix-dgram(filename [options]);

Notesyslogd on Linux originally used SOCK_STREAM sockets, but some distributions switched to SOCK_DGRAM around 1999to fix a possible DoS problem. On Linux you can choose to use whichever driver you like as syslog clients automaticallydetect the socket type being used.

Example 6.41. Using the unix-stream() and unix-dgram() drivers

source s_stream { unix-stream("/dev/log" max-connections(10)); };

source s_dgram { unix-dgram("/var/run/log"); };

158syslog-ng.com

Collecting messages from UNIX domain sockets

6.15.1. unix-stream() and unix-dgram() source options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

These two drivers behave similarly: they open an AF_UNIX socket and start listening on it for messages. Thefollowing options can be specified for these drivers:

create-dirs()yes or noType:noDefault:

Description: Enable creating non-existing directories when creating the socket files.

encoding()stringType:

Default:

Description: Specifies the characterset (encoding, for example UTF-8) of messages using the legacy BSD-syslogprotocol. To list the available character sets on a host, execute the iconv -l command. For details on howencoding affects the size of the message, see Section Message size and encoding (p. 25).

flags()assume-utf8, empty-lines, expect-hostname, kernel, no-multi-line, no-parse, dont-store-legacy-msghdr,syslog-protocol, validate-utf8

Type:

empty setDefault:

Description: Specifies the log parsing options of the source.

■ assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, butdoes not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incomingmessage, use the validate-utf8 flag.

■ dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the logmessage. This is useful of the original format of a non-syslog-compliant message must be retained(syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msgin the following message: Jan 22 10:06:11 host program:msg). If you do not want to storethe original header of the message, enable the dont-store-legacy-msghdr flag.

■ empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ngPE removes empty lines automatically.

159syslog-ng.com

unix-stream() and unix-dgram() source options

■ expect-hostname: If the expect-hostname flag is enabled, syslog-ng PE will assume that the logmessage contains a hostname and parse the message accordingly. This is the default behavior forTCP sources. Note that pipe sources use the no-hostname flag by default.

■ kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if notspecified otherwise.

■ no-hostname: Enable the no-hostname flag if the log message does not include the hostname ofthe sender host. That way syslog-ng PE assumes that the first part of the message header is${PROGRAM} instead of ${HOST}. For example:

source s_dell { network(port(2000) flags(no-hostname)); };

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line. Note that this happens only if the underlying transport method actuallysupports multi-line messages. Currently the rltp, syslog(), network(), unix-dgram() driverssupport multi-line messages.

■ no-parse: By default, syslog-ng PE parses incoming messages as syslog messages. The no-parseflag completely disables syslog message parsing and processes the complete line as the message partof a syslog message. The syslog-ng PE application will generate a new syslog header (timestamp,host, and so on) automatically and put the entire incoming message into the MSG part of the syslogmessage. This flag is useful for parsing messages not complying to the syslog format.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled,and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In thiscase, syslog-ng PE generates a new syslog header (timestamp, host, and so on) automatically. Notethat since flags(no-parse) disables message parsing, it interferes with other flags, for example,disables flags(no-multi-line).

■ syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to beformatted according to the new IETF syslog protocol standard (RFC5424), but without the frameheader. Note that this flag is not needed for the syslog driver, which handles only messages thathave a frame header.

■ validate-utf8: The validate-utf8 flag enables encoding-verification for messages formattedaccording to the new IETF syslog standard (for details, see Section 2.10.2, IETF-syslogmessages (p. 21)). If theBOMcharacter is missing, but the message is otherwise UTF-8 compliant,syslog-ng automatically adds the BOM character to the message.

group()stringType:rootDefault:

Description: Set the gid of the socket.

The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.

160syslog-ng.com

unix-stream() and unix-dgram() source options

host-override()stringType:

Default:

Description: Replaces the ${HOST} part of the message with the parameter string.

keep-alive()yes or noType:yesDefault:

Description: Selects whether to keep connections open when syslog-ng is restarted, cannot be used withunix-dgram().

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

log-fetch-limit()number (messages)Type:10Default:

Description:The maximum number of messages fetched from a source during a single poll loop. The destinationqueues might fill up before flow-control could stop reading if log-fetch-limit() is too high.

log-iw-size()number (messages)Type:1000Default:

Description:The size of the initial window, this value is used during flow control. If the max-connections()option is set, the log-iw-size() will be divided by the number of connections, otherwise log-iw-size()is divided by 10 (the default value of the max-connections() option). The resulting number is the initialwindow size of each connection. For optimal performance when receiving messages from syslog-ng PE clients,make sure that the window size is larger than the flush-lines() option set in the destination of your clients.

161syslog-ng.com

unix-stream() and unix-dgram() source options

Example 6.42. Initial window size of a connectionIf log-iw-size(1000) and max-connections(10), then each connection will have an initial window size of 100.

log-msg-size()number (bytes)Type:Use the global log-msg-size() option, which defaults to 65535.Default:

Description: Specifies the maximum length of incoming log messages. Uses the value of the global option ifnot specified. For details on how encoding affects the size of the message, see Section Message size andencoding (p. 25).

log-prefix() (DEPRECATED)stringType:

Default:

Description: A string added to the beginning of every log message. It can be used to add an arbitrary string toany log source, though it is most commonly used for adding kernel: to the kernel messages on Linux. NOTE:This option is deprecated. Use program-override() instead.

max-connections()number (simultaneous connections)Type:256Default:

Description: Limits the number of simultaneously open connections. Cannot be used with unix-dgram().

multi-line-garbage()regular expressionType:empty stringDefault:

Description: Use the multi-line-garbage() option when processing multi-line messages that containunneeded parts between the messages. Specify a string or regular expression that matches the beginning of theunneeded message parts. If the multi-line-garbage() option is set, syslog-ng PE ignores the lines betweenthe line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See alsothe multi-line-prefix() option.

When receiving multi-line messages from a source when the multi-line-garbage() option is set, but nomatching line is received between two lines that match multi-line-prefix(), syslog-ng PE will continueto process the incoming lines as a single message until a line matching multi-line-garbage() is received.

WarningIf the multi-line-garbage() option is set, syslog-ng PE discards lines between the line matching themulti-line-garbage() and the next line matching multi-line-prefix().

162syslog-ng.com

unix-stream() and unix-dgram() source options

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

This option is not available for the unix-dgram driver.

multi-line-prefix()regular expressionType:empty stringDefault:

Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messagesthat contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matchesthe beginning of the log messages. Use as simple regular expressions as possible, because complex regularexpressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix()

option is set, syslog-ng PE ignores newline characters from the source until a line matches the regular expressionagain, and treats the lines between the matching lines as a single message. See also the multi-line-garbage()option.

NoteStarting with syslog-ng PE version 3.2.1, a message is considered complete if no new lines arrive to the message for 10seconds, even if no line matching the multi-line-garbage() option is received.

TipTo make multi-line messages more readable when written to a file, use a template in the destination andinstead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). Thisexpression inserts a tab after every newline character (except when a tab is already present), indenting everyline of the message after the first. For example:

■

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );

};

For details on using templates, see Section 14.1.2, Templates and macros (p. 379).

■ To actually convert the lines of multi-line messages to single line (by replacing the newline characters withwhitespaces), use the flags(no-multi-line) option in the source.

Example 6.43. Processing Tomcat logsThe log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages startwith the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

SEVERE: Catalina.start:

LifecycleException: service.getName(): "Catalina"; Protocol handler start failed:

java.net.BindException: Address already in use<null>:8080

at org.apache.catalina.connector.Connector.start(Connector.java:1138)

at org.apache.catalina.core.StandardService.start(StandardService.java:531)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)

at org.apache.catalina.startup.Catalina.start(Catalina.java:583)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

163syslog-ng.com

unix-stream() and unix-dgram() source options

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)

2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start

INFO: Server startup in 1206 ms

2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

To process these messages, specify a regular expression matching the timestamp of the messages in themulti-line-prefix() option. Such an expression is the following:

source s_file{ file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0)

multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));

};

Note that the flags(no-parse) is needed to avoid syslog-ng PE trying to interpret the date in the message.

This option is not available for the unix-dgram driver.

optional()yes or noType:

Default:

Description: Instruct syslog-ng to ignore the error if a specific source cannot be initialized. No other attemptsto initialize the source will be made until the configuration is reloaded. This option currently applies to thepipe(), unix-dgram, and unix-stream drivers.

owner()stringType:rootDefault:

Description: Set the uid of the socket.

pad-size()number (bytes)Type:0Default:

Description: Specifies input padding. Some operating systems (such as HP-UX) pad all messages to blockboundary. This option can be used to specify the block size. (HP-UX uses 2048 bytes). The syslog-ng PEapplication will pad reads from the associated device to the number of bytes set in pad-size(). Mostly usedon HP-UX where /dev/log is a named pipe and every write is padded to 2048 bytes. If pad-size() wasgiven and the incoming message does not fit into pad-size(), syslog-ng will not read anymore from this pipeand displays the following error message:

164syslog-ng.com

unix-stream() and unix-dgram() source options

Padding was set, and couldn't read enough bytes

perm()number (octal notation)Type:0666Default:

Description: Set the permission mask. For octal numbers prefix the number with '0', for example: use 0755for rwxr-xr-x.

program-override()stringType:

Default:

Description:Replaces the ${PROGRAM} part of the message with the parameter string. For example, to markevery message coming from the kernel, include the program-override("kernel") option in the sourcecontaining /proc/kmsg.

so-keepalive()yes or noType:noDefault:

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on the receiver host (thatis, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms, for example, on Red HatEnterprise Linux 5, even low message load (~200 messages per second) can result in message loss, unless the so-rcvbuf()option of the source is increased. In such cases, you will need to increase the net.core.rmem_max parameter of thehost (for example, to 1024000), but do not modify net.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate of incomingmessages per second. For example, to receive 2000 messages per second, set the so-rcvbuf() at least to 2 097 152

bytes.

tags()stringType:

Default:

165syslog-ng.com

unix-stream() and unix-dgram() source options

Description:Label the messages received from the source with custom tags. Tags must be unique, and enclosedbetween double quotes. When adding multiple tags, separate them with comma, for example tags("dmz",

"router"). This option is available only in syslog-ng 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:

Default:

Description: The default timezone for messages read from the source. Applies only if no timezone is specifiedwithin the message itself.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

use-syslogng-pid()yes or noType:noDefault:

Description: If the value of this option is yes, then the PID value of the message will be overridden with thePID of the running syslog-ng process.

166syslog-ng.com

unix-stream() and unix-dgram() source options

Chapter 7. Sending and storing log messages —destinations and destination drivers

A destination is where a log message is sent if the filtering rules match. Similarly to sources, destinations consistof one or more drivers, each defining where and how messages are sent.

TipIf no drivers are defined for a destination, all messages sent to the destination are discarded. This is equivalent to omittingthe destination from the log statement.

To define a destination, add a destination statement to the syslog-ng configuration file using the followingsyntax:

destination <identifier> {

destination-driver(params); destination-driver(params); ... };

Example 7.1. A simple destination statementThe following destination statement sends messages to the TCP port 1999 of the 10.1.2.3 host.

destination d_demo_tcp { network("10.1.2.3" port(1999)); };

If name resolution is configured, you can use the hostname of the target server as well.

destination d_tcp { network("target_host" port(1999)); };

WarningDo not define the same drivers with the same parameters more than once, because it will cause problems.For example, do not open the same file in multiple destinations.

■

■ Do not use the same destination in different log paths, because it can cause problems with most destinationtypes. Instead, use filters and log paths to avoid such situations.

■ Sources and destinations are initialized only when they are used in a log statement. For example, syslog-ngPE starts listening on a port or starts polling a file only if the source is used in a log statement. For detailson creating log statements, see Chapter 8, Routing messages: log paths, reliability, and filters (p. 311).

■ Hazard of data loss! If your log files are on an NFS-mounted network file system, see Section 2.14, NFSfile system for log files (p. 34).

The following table lists the destination drivers available in syslog-ng PE.

DescriptionNameSends messages to an Elasticsearch server. Theelasticsearch2 driver supports Elasticsearch version2 and newer.

elasticsearch and elasticsearch2

Writes messages to the specified file.file()

167syslog-ng.com

DescriptionNameSends messages into a file on a Hadoop DistributedFile System (HDFS) node.

hdfs

Publishes log messages to the Apache Kafka messagebus, where subscribers can access them.

kafka

Writes messages to the specified binary logstore file.logstore()

Sends messages to a MongoDB database.mongodb()

Sends messages to a remote host using the BSD-syslogprotocol over IPv4 and IPv6. Supports the TCP,UDP,RLTP™, and TLS network protocols.

network()

Writes messages to the specified named pipe.pipe()

Forks and launches the specified program, and sendsmessages to its standard input.

program()

Sends messages send mail to trusted recipients, througha controlled channel using the SMTP protocol.

smtp()

Sends messages to the specified remote host using theSNMP v2c or v3 protocol.

snmp()

Sends messages into an SQL database. In addition tothe standard syslog-ng packages, the sql() destination

sql()

requires database-specific packages to be installed.Refer to the section appropriate for your platform inChapter 3, Installing syslog-ng (p. 36).

Sends messages to the specified remote host using theIETF-syslog protocol. The IETF standard supports

syslog()

message transport using the UDP, TCP, and TLSnetworking protocols.

Sends messages to the specified unix socket inSOCK_DGRAM style (BSD).

unix-dgram()

Sends messages to the specified unix socket inSOCK_STREAM style (Linux).

unix-stream()

Sends messages to the terminal of the specified user,if the user is logged in.

usertty()

Table 7.1. Destination drivers available in syslog-ng

7.1. Sending messages directly to Elasticsearch version 1.x

Starting with version 5.6 of syslog-ng PE can directly send log messages to Elasticsearch, allowing you tosearch and analyze your data in real time, and visualize it with Kibana.

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

168syslog-ng.com

Sending messages directly to Elasticsearch version 1.x

Note the following limitations when using the syslog-ng PE elasticsearch destination:

■ This destination is only supported on the Linux platforms that use the linux glibc2.11 installer,including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (TrustyTahr).

■ Since syslog-ng PE uses the official Java Elasticsearch libraries, the elasticsearch destinationhas significant memory usage.

■ The log messages of the underlying client libraries are available in the internal() source ofsyslog-ng PE.

Declaration:

@module mod-java

@include "scl.conf"

elasticsearch(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

cluster("syslog-ng")

);

Example 7.2. Sending log data to Elasticsearch version 1.xThe following example defines an elasticsearch destination that sends messages in transport mode to an Elasticsearchserver version 1.x running on the localhost, using only the required parameters.

@module mod-java

@include "scl.conf"

destination d_elastic {

elasticsearch(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

);

};

The following example sends 10000 messages in a batch, in node mode, and includes a custom unique ID for each message.

@module mod-java

@include "scl.conf"

options {

threaded(yes);

use_uniqid(yes);

};

source s_syslog {

syslog();

};

destination d_elastic {

elasticsearch(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

cluster("syslog-ng")

client_mode("node")

custom_id("${UNIQID}")

flush-limit("10000")

);

};

log {

169syslog-ng.com

Sending messages directly to Elasticsearch version 1.x

source(s_syslog);

destination(d_elastic);

flags(flow-control);

};

■ To install the software required for the elasticsearch destination, see Procedure 7.1.1,Prerequisites (p. 170).

■ For details on how the elasticsearch destination works, see Section 7.1.2, How syslog-ng PEinteracts with Elasticsearch (p. 171).

■ For the list of options, see Section 7.1.4, Elasticsearch destination options (p. 172).

7.1.1. Procedure – Prerequisites

To send messages from syslog-ng PE to Elasticsearch, complete the following steps.

Steps:

Step 1. If you want to use the Java-based modules of syslog-ng PE (for example, the Elasticsearch, HDFS, orKafka destinations), you must compile syslog-ng PE with Java support.

■ Download and install the Java Runtime Environment (JRE), 1.7 (or newer). The Java-basedmodules of syslog-ng PE are tested and supported when using the Oracle implementationof Java. Other implementations are untested and unsupported, they may or may not workas expected.

■ Install gradle version 2.2.1 or newer.

■ Set LD_LIBRARY_PATH to include the libjvm.so file, forexample:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATHNote that many platforms have a simplified links for Java libraries. Use the simplified pathif available. If you use a startup script to start syslog-ng PE set LD_LIBRARY_PATH in thescript as well.

■ If you are behind an HTTP proxy, create a gradle.properties under themodules/java-modules/ directory. Set the proxy parameters in the file. For details, seeThe Gradle User Guide.

Step 2. Download the Elasticsearch libraries version 1.5 or newer from the 1.x line fromhttps://www.elastic.co/downloads/elasticsearch.One Identity tests the destination using Elasticsearchversion 1.5. To use Elasticsearch 2.x or newer, use the elasticsearch2() destination (see Section7.2, Sending messages directly to Elasticsearch version 2.0 or higher (p. 179)).

Step 3. Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files intoa single directory (for example, /opt/elasticsearch/lib/) where syslog-ng PE can access them.You must specify this directory in the syslog-ng PE configuration file. The files are located in the libdirectory and its subdirectories of the Elasticsearch release package.

170syslog-ng.com

Sending messages directly to Elasticsearch version 1.x

7.1.2. How syslog-ng PE interacts with Elasticsearch

The syslog-ng PE application sends the log messages to the official Elasticsearch client library, which forwardsthe data to the Elasticsearch nodes. The way how syslog-ng PE interacts with Elasticsearch is described in thefollowing steps.

■ After syslog-ng PE is started and the first message arrives to the elasticsearch destination, theelasticsearch destination tries to connect to the Elasticsearch server or cluster. If the connectionfails, syslog-ng PE will repeatedly attempt to connect again after the period set in time-reopen()expires.

■ If the connection is established, syslog-ng PE sends JSON-formatted messages to Elasticsearch.

• If flush_limit is set to 1: syslog-ng PE sends the message reliably: it sends a message toElasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeatssending the message, as set in the retries() parameter. If sending the message fails forretries() times, syslog-ng PE drops the message.

This method ensures reliable message transfer, but is slow.

• If flush_limit is higher than 1: syslog-ng PE sends messages in a batch, and receives theresponse asynchronously. In case of a problem, syslog-ng PE cannot resend the messages.

This method is relatively fast (depending on the size of flush_limit), but the transfer is notreliable. In transport mode, several messages can be lost before syslog-ng PE recognizes the error.Node mode is more reliable in this sense, because the message loss rate is significantly lower.

• If concurrent-requests is higher than 1, syslog-ng PE can send multiple batchessimultaneously, increasing performance (and also the number of messages that can be lost in caseof an error). For details, see Section concurrent_requests() (p. 173).

7.1.3. Client modes

The syslog-ng PE application can interact with Elasticsearch in transport mode or node mode.

■ Transport mode. The syslog-ng PE application uses the transport client API of Elasticsearch,and uses the server(), port(), and cluster() options from the syslog-ng PE configuration file.

■ Node mode. The syslog-ng PE application acts as an Elasticsearch node (client no-data), usingthe node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearchconfiguration file specified in the resource() option.

NoteIn Node mode, it is required to define the home of the elasticsearch installation with the path.home

paramter in the .yml file. For example: path.home: /usr/share/elasticsearch.

171syslog-ng.com

How syslog-ng PE interacts with Elasticsearch

■ Shield mode. This mode is available only from syslog-ng PE version 5.6. In this mode, syslog-ngPE uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster()options from the syslog-ng PE configuration file, but with Shield (X-Pack security) support. Formore details about Shield, see: https://www.elastic.co/products/x-pack/security.

7.1.4. Elasticsearch destination options

The elasticsearch destination can directly send log messages to Elasticsearch, allowing you to search andanalyze your data in real time, and visualize it withKibana. The elasticsearch destination has the followingoptions.

Required options:

The following options are required: index(), type(). In node mode, either the cluster() or the resource()option is required as well. Note that to use elasticsearch, you must add the following lines to the beginningof your syslog-ng PE configuration:

@module mod-java

@include "scl.conf"

client_lib_dir()stringType:N/ADefault:

Description: Include the path to the directory where you copied the required libraries (see Procedure 7.1.1,Prerequisites (p. 170)), for example, client_lib_dir(/user/share/elasticsearch-2.2.0/lib).

client_mode()transport | node | shieldType:nodeDefault:

Description: Specifies the client mode used to connect to the Elasticsearch server, for example,option("client-mode", "transport").

■ Transport mode. The syslog-ng PE application uses the transport client API of Elasticsearch,and uses the server(), port(), and cluster() options from the syslog-ng PE configuration file.

■ Node mode. The syslog-ng PE application acts as an Elasticsearch node (client no-data), usingthe node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearchconfiguration file specified in the resource() option.

NoteIn Node mode, it is required to define the home of the elasticsearch installation with the path.home

paramter in the .yml file. For example: path.home: /usr/share/elasticsearch.

172syslog-ng.com

Elasticsearch destination options

■ Shield mode. This mode is available only from syslog-ng PE version 5.6. In this mode, syslog-ngPE uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster()options from the syslog-ng PE configuration file, but with Shield (X-Pack security) support. Formore details about Shield, see: https://www.elastic.co/products/x-pack/security.

NoteIn Node mode, it is required to define the home of the elasticsearch installation with the path.home paramter in the .ymlfile. For example: path.home: /usr/share/elasticsearch.

■ To use this mode, add the Shield .jar file (shield-x.x.x.jar) to the same directory where yourElasticsearch .jar files are located. You can download the Shield distribution and extract the .jar filemanually, or you can get it from the Elasticsearch Maven repository.It inherits the Transport mode options, but the Shield-related options must be configured in the .ymlfile (see the resource() option of syslog-ng PE). For more details about the possible options, see:https://www.elastic.co/guide/en/shield/current/reference.html#ref-ssl-tls-settings.

Example 7.3. Example for the .yml file

shield.user: es_admin:********

shield.transport.ssl: true

shield.ssl.keystore.path: /usr/share/elasticsearch/node.jks

shield.ssl.keystore.password: mypassword

cluster()stringType:N/ADefault:

Description: Specifies the name or the Elasticsearch cluster, for example,cluster("my-elasticsearch-cluster"). Optionally, you can specify the name of the cluster in theElasticsearch resource file. For details, see Section resource() (p. 178).

concurrent_requests()numberType:0Default:

Description: The number of concurrent (simultaneous) requests that syslog-ng PE sends to the Elasticsearchserver. Set this option to 1 or higher to increase performance. When using the concurrent_requests()

option, make sure that the flush-limit() option is higher than one, otherwise it will not have any noticeableeffect. For details, see Section flush_limit() (p. 176).

WarningHazard of data loss! Using the concurrent-requests() option increases the number of messages lost in case theElasticsearch server becomes unaccessible.

173syslog-ng.com

Elasticsearch destination options

custom_id()template or template functionType:N/ADefault:

Description: Use this option to specify a custom ID for the records inserted into Elasticsearch. If this optionis not set, the Elasticsearch server automatically generates and ID for the message. For example:custom_id(${UNIQID}) (Note that to use the ${UNIQID} macro, the use-uniqid() global option mustbe enabled. For details, see Section use-uniqid() (p. 349).)

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

174syslog-ng.com

Elasticsearch destination options

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.4. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

175syslog-ng.com

Elasticsearch destination options

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

flush_limit()numberType:5000Default:

Description: The number of messages that syslog-ng PE sends to the Elasticsearch server in a single batch.

■ If flush_limit is set to 1: syslog-ng PE sends the message reliably: it sends a message toElasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeatssending the message, as set in the retries() parameter. If sending the message fails for retries()times, syslog-ng PE drops the message.

This method ensures reliable message transfer, but is slow.

■ If flush_limit is higher than 1: syslog-ng PE sends messages in a batch, and receives the responseasynchronously. In case of a problem, syslog-ng PE cannot resend the messages.

This method is relatively fast (depending on the size of flush_limit), but the transfer is not reliable.In transport mode, several messages can be lost before syslog-ng PE recognizes the error. Nodemode is more reliable in this sense, because the message loss rate is significantly lower.

■ If concurrent-requests is higher than 1, syslog-ng PE can send multiple batches simultaneously,increasing performance (and also the number of messages that can be lost in case of an error). Fordetails, see Section concurrent_requests() (p. 173).

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions can

176syslog-ng.com

Elasticsearch destination options

always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

index()stringType:N/ADefault:

Description: Name of the Elasticsearch index to store the log messages. You can use macros and templates aswell. For example, index("syslog-ng_${YEAR}.${MONTH}.${DAY}").

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:Use the global setting (which defaults to drop-message)Default:

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

port()numberType:9300Default:

177syslog-ng.com

Elasticsearch destination options

Description: The port number of the Elasticsearch server. This option is used only in transport mode:client-mode("transport")

retries()number (of attempts)Type:3Default:

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ngPE could not send a message, it will try again until the number of attempts reaches retries, then drops themessage.

resource()stringType:N/ADefault:

Description: The list of Elasticsearch resources to load, separated by semicolons. For example,resource("/home/user/elasticsearch/elasticsearch.yml;/home/user/elasticsearch/elasticsearch2.yml").

server()list of hostnamesType:127.0.0.1Default:

Description: Specifies the hostname or IP address of the Elasticsearch server. When specifying an IP address,IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. When specifying multipleaddresses, use space to separate the addresses, for example, server("127.0.0.1

remote-server-hostname1 remote-server-hostname2")

This option is used only in transport mode: client-mode("transport")

template()template or template functionType:$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})Default:

Description: The message as sent to the Elasticsearch server. Typically, you will want to use the command-linenotation of the format-json template function.

To add a @timestamp field to the message, for example, to use with Kibana, include the@timestamp=${ISODATE} expression in the template. For example: template($(format-json --scope

rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))

For details on formatting messages in JSON format, see Section format-json (p. 393).

throttle()number (messages per second)Type:0Default:

178syslog-ng.com

Elasticsearch destination options

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

type()stringType:N/ADefault:

Description: The type of the index. For example, type("test").

7.2. Sending messages directly to Elasticsearch version 2.0 or higher

Starting with version 5.6 of syslog-ng PE can directly send log messages to Elasticsearch, allowing you tosearch and analyze your data in real time, and visualize it with Kibana.

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

Note the following limitations when using the syslog-ng PE elasticsearch2 destination:

■ This destination is only supported on the Linux platforms that use the linux glibc2.11 installer,including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (TrustyTahr).

179syslog-ng.com

Sending messages directly to Elasticsearch version 2.0 or higher

■ Since syslog-ng PE uses the official Java Elasticsearch libraries, the elasticsearch2 destinationhas significant memory usage.

■ The log messages of the underlying client libraries are available in the internal() source ofsyslog-ng PE.

Declaration:

@module mod-java

@include "scl.conf"

elasticsearch2(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

cluster("syslog-ng")

);

Example 7.5. Sending log data to Elasticsearch version 2.x and aboveThe following example defines an elasticsearch2 destination that sends messages in transport mode to an Elasticsearchserver running on the localhost, using only the required parameters.

@module mod-java

@include "scl.conf"

destination d_elastic {

elasticsearch2(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

);

};

The following example sends 10000 messages in a batch, in node mode, and includes a custom unique ID for each message.

@module mod-java

@include "scl.conf"

options {

threaded(yes);

use_uniqid(yes);

};

source s_syslog {

syslog();

};

destination d_elastic {

elasticsearch2(

index("syslog-ng_${YEAR}.${MONTH}.${DAY}")

type("test")

cluster("syslog-ng")

client_mode("node")

custom_id("${UNIQID}")

flush-limit("10000")

);

};

log {

source(s_syslog);

destination(d_elastic);

flags(flow-control);

};

180syslog-ng.com

Sending messages directly to Elasticsearch version 2.0 or higher

■ To install the software required for the elasticsearch2 destination, see Procedure 7.2.1,Prerequisites (p. 181).

■ For details on how the elasticsearch2 destination works, see Section 7.2.2, How syslog-ng PEinteracts with Elasticsearch (p. 181).

■ For the list of options, see Section 7.2.4, Elasticsearch destination options (p. 182).

7.2.1. Procedure – Prerequisites

To send messages from syslog-ng PE to Elasticsearch, complete the following steps.

Steps:

Step 1. Download and install the Java Runtime Environment (JRE), 2.x (or newer). The syslog-ng PEelasticsearch2 destination is tested and supported when using the Oracle implementation of Java.Other implementations are untested and unsupported, they may or may not work as expected.

Step 2. Download the Elasticsearch libraries (version 2.x or newer from the 2.x line) fromhttps://www.elastic.co/downloads/elasticsearch.One Identity tests the destination using Elasticsearchversion 2.4.To use Elasticsearch 1.x, use the elasticsearch() destination (see Section 7.1, Sendingmessages directly to Elasticsearch version 1.x (p. 168)).

Step 3. Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files intoa single directory (for example, /opt/elasticsearch/lib/) where syslog-ng PE can access them.You must specify this directory in the syslog-ng PE configuration file. The files are located in the libdirectory and its subdirectories of the Elasticsearch release package.

7.2.2. How syslog-ng PE interacts with Elasticsearch

The syslog-ng PE application sends the log messages to the official Elasticsearch client library, which forwardsthe data to the Elasticsearch nodes. The way how syslog-ng PE interacts with Elasticsearch is described in thefollowing steps.

■ After syslog-ng PE is started and the first message arrives to the elasticsearch2 destination, theelasticsearch2 destination tries to connect to the Elasticsearch server or cluster. If the connectionfails, syslog-ng PE will repeatedly attempt to connect again after the period set in time-reopen()expires.

■ If the connection is established, syslog-ng PE sends JSON-formatted messages to Elasticsearch.

• If flush_limit is set to 1: syslog-ng PE sends the message reliably: it sends a message toElasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeatssending the message, as set in the retries() parameter. If sending the message fails forretries() times, syslog-ng PE drops the message.

This method ensures reliable message transfer, but is slow.

181syslog-ng.com

How syslog-ng PE interacts with Elasticsearch

• If flush_limit is higher than 1: syslog-ng PE sends messages in a batch, and receives theresponse asynchronously. In case of a problem, syslog-ng PE cannot resend the messages.

This method is relatively fast (depending on the size of flush_limit), but the transfer is notreliable. In transport mode, several messages can be lost before syslog-ng PE recognizes the error.Node mode is more reliable in this sense, because the message loss rate is significantly lower.

• If concurrent-requests is higher than 1, syslog-ng PE can send multiple batchessimultaneously, increasing performance (and also the number of messages that can be lost in caseof an error). For details, see Section concurrent_requests() (p. 173).

7.2.3. Client modes

The syslog-ng PE application can interact with Elasticsearch in transport mode or node mode.

■ Transport mode. The syslog-ng PE application uses the transport client API of Elasticsearch,and uses the server(), port(), and cluster() options from the syslog-ng PE configuration file.

■ Node mode. The syslog-ng PE application acts as an Elasticsearch node (client no-data), usingthe node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearchconfiguration file specified in the resource() option.

NoteIn Node mode, it is required to define the home of the elasticsearch installation with the path.home

paramter in the .yml file. For example: path.home: /usr/share/elasticsearch.

■ Shield mode. This mode is available only from syslog-ng PE version 5.6. In this mode, syslog-ngPE uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster()options from the syslog-ng PE configuration file, but with Shield (X-Pack security) support. Formore details about Shield, see: https://www.elastic.co/products/x-pack/security.

7.2.4. Elasticsearch destination options

The elasticsearch2 destination can directly send log messages to Elasticsearch, allowing you to searchand analyze your data in real time, and visualize it with Kibana. The elasticsearch2 destination has thefollowing options.

Required options:

The following options are required: index(), type(). In node mode, either the cluster() or the resource()option is required as well. Note that to use elasticsearch2, you must add the following lines to the beginningof your syslog-ng PE configuration:

@module mod-java

@include "scl.conf"

182syslog-ng.com

Client modes

client_lib_dir()stringType:N/ADefault:

Description: Include the path to the directory where you copied the required libraries (see Procedure 7.2.1,Prerequisites (p. 181)), for example, client_lib_dir(/user/share/elasticsearch-2.2.0/lib).

client_mode()transport | node | shieldType:nodeDefault:

Description: Specifies the client mode used to connect to the Elasticsearch server, for example,client_mode("node").

■ Transport mode. The syslog-ng PE application uses the transport client API of Elasticsearch,and uses the server(), port(), and cluster() options from the syslog-ng PE configuration file.

■ Node mode. The syslog-ng PE application acts as an Elasticsearch node (client no-data), usingthe node client API of Elasticsearch. Further options for the node can be describe in an Elasticsearchconfiguration file specified in the resource() option.

NoteIn Node mode, it is required to define the home of the elasticsearch installation with the path.home

paramter in the .yml file. For example: path.home: /usr/share/elasticsearch.

■ Shield mode. This mode is available only from syslog-ng PE version 5.6. In this mode, syslog-ngPE uses the transport client API of Elasticsearch, and uses the server(), port(), and cluster()options from the syslog-ng PE configuration file, but with Shield (X-Pack security) support. Formore details about Shield, see: https://www.elastic.co/products/x-pack/security.

■ To use this mode, add the Shield .jar file (shield-x.x.x.jar) to the same directory where yourElasticsearch .jar files are located. You can download the Shield distribution and extract the .jar filemanually, or you can get it from the Elasticsearch Maven repository.It inherits the Transport mode options, but the Shield-related options must be configured in the .ymlfile (see the resource() option of syslog-ng PE). For more details about the possible options, see:https://www.elastic.co/guide/en/shield/current/reference.html#ref-ssl-tls-settings.

Example 7.6. Example for the .yml file

shield.user: es_admin:********

shield.transport.ssl: true

shield.ssl.keystore.path: /usr/share/elasticsearch/node.jks

shield.ssl.keystore.password: mypassword

183syslog-ng.com

Elasticsearch destination options

cluster()stringType:N/ADefault:

Description: Specifies the name or the Elasticsearch cluster, for example,cluster("my-elasticsearch-cluster"). Optionally, you can specify the name of the cluster in theElasticsearch resource file. For details, see Section resource() (p. 189).

concurrent_requests()numberType:0Default:

Description: The number of concurrent (simultaneous) requests that syslog-ng PE sends to the Elasticsearchserver. Set this option to 1 or higher to increase performance. When using the concurrent_requests()

option, make sure that the flush-limit() option is higher than one, otherwise it will not have any noticeableeffect. For details, see Section flush_limit() (p. 187).

WarningHazard of data loss! Using the concurrent-requests() option increases the number of messages lost in case theElasticsearch server becomes unaccessible.

custom_id()template or template functionType:N/ADefault:

Description: Use this option to specify a custom ID for the records inserted into Elasticsearch. If this optionis not set, the Elasticsearch server automatically generates and ID for the message. For example:custom_id(${UNIQID}) (Note that to use the ${UNIQID} macro, the use-uniqid() global option mustbe enabled. For details, see Section use-uniqid() (p. 349).)

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

184syslog-ng.com

Elasticsearch destination options

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

185syslog-ng.com

Elasticsearch destination options

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.7. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

186syslog-ng.com

Elasticsearch destination options

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

flush_limit()numberType:5000Default:

Description: The number of messages that syslog-ng PE sends to the Elasticsearch server in a single batch.

■ If flush_limit is set to 1: syslog-ng PE sends the message reliably: it sends a message toElasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeatssending the message, as set in the retries() parameter. If sending the message fails for retries()times, syslog-ng PE drops the message.

This method ensures reliable message transfer, but is slow.

■ If flush_limit is higher than 1: syslog-ng PE sends messages in a batch, and receives the responseasynchronously. In case of a problem, syslog-ng PE cannot resend the messages.

This method is relatively fast (depending on the size of flush_limit), but the transfer is not reliable.In transport mode, several messages can be lost before syslog-ng PE recognizes the error. Nodemode is more reliable in this sense, because the message loss rate is significantly lower.

■ If concurrent-requests is higher than 1, syslog-ng PE can send multiple batches simultaneously,increasing performance (and also the number of messages that can be lost in case of an error). Fordetails, see Section concurrent_requests() (p. 173).

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

index()stringType:N/ADefault:

187syslog-ng.com

Elasticsearch destination options

Description: Name of the Elasticsearch index to store the log messages. You can use macros and templates aswell. For example, index("syslog-ng_${YEAR}.${MONTH}.${DAY}").

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:Use the global setting (which defaults to drop-message)Default:

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

port()numberType:9300Default:

Description: The port number of the Elasticsearch server. This option is used only in transport mode:client_mode("transport")

retries()number (of attempts)Type:3Default:

188syslog-ng.com

Elasticsearch destination options

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ngPE could not send a message, it will try again until the number of attempts reaches retries, then drops themessage.

resource()stringType:N/ADefault:

Description: The list of Elasticsearch resources to load, separated by semicolons. For example,resource("/home/user/elasticsearch/elasticsearch.yml;/home/user/elasticsearch/elasticsearch2.yml").

server()list of hostnamesType:127.0.0.1Default:

Description: Specifies the hostname or IP address of the Elasticsearch server. When specifying an IP address,IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. When specifying multipleaddresses, use space to separate the addresses, for example, server("127.0.0.1

remote-server-hostname1 remote-server-hostname2")

This option is used only in transport mode: client_mode("transport")

template()template or template functionType:$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})Default:

Description: The message as sent to the Elasticsearch server. Typically, you will want to use the command-linenotation of the format-json template function.

To add a @timestamp field to the message, for example, to use with Kibana, include the@timestamp=${ISODATE} expression in the template. For example: template($(format-json --scope

rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))

For details on formatting messages in JSON format, see Section format-json (p. 393).

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

189syslog-ng.com

Elasticsearch destination options

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

type()stringType:N/ADefault:

Description: The type of the index. For example, type("test").

7.3. Storing messages in plain-text files

The file driver is one of the most important destination drivers in syslog-ng. It allows to output messages to thespecified text file, or to a set of files.

The destination filename may include macros which get expanded when the message is written, thus a simplefile() driver may create several files: for example, syslog-ng PE can store the messages of client hosts in aseparate file for each host. For more information on available macros see Section 14.1.5, Macros of syslog-ngPE (p. 383).

If the expanded filename refers to a directory which does not exist, it will be created depending on thecreate-dirs() setting (both global and a per destination option).

The file() has a single required parameter that specifies the filename that stores the log messages. For thelist of available optional parameters, see Section 7.3.1, file() destination options (p. 191).

Declaration:

file(filename options());

190syslog-ng.com

Storing messages in plain-text files

Example 7.8. Using the file() driver

destination d_file { file("/var/log/messages"); };

NoteOn Microsoft Windows platforms, enclose paths and filenames between single quotes, forexample, 'C:\temp\logs\mylogs.log'

Example 7.9. Using the file() driver with macros in the file name and a template for the message

destination d_file {

file("/var/log/${YEAR}.${MONTH}.${DAY}/messages"

template("${HOUR}:${MIN}:${SEC} ${TZ} ${HOST} [${LEVEL}] ${MSG}\n")

template-escape(no));

};

NoteWhen using this destination, update the configuration of your log rotation program to rotate these files. Otherwise, thelog files can become very large.

Also, after rotating the log files, reload syslog-ng PE using the syslog-ng-ctl reload command, or use anothermethod to send a SIGHUP to syslog-ng PE.

NoteThe syslog-ng PE application uses UNIX-style line-breaks. This means that on Windows, the end of the line is markedwith LF instead of CRLF.

WarningSince the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no newmessages are written to a file within 60 seconds (controlled by the time-reap() global option), it is closed, and its stateis freed.

Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its neededmemory is more than the amount available on the syslog-ng server.

The most suspicious macro is ${PROGRAM}, where the number of possible variations is rather high. Do not use the${PROGRAM} macro in insecure environments.

7.3.1. file() destination options

The file() driver outputs messages to the specified text file, or to a set of files. The file() destination hasthe following options:

WarningWhen creating several thousands separate log files, syslog-ng might not be able to open the required number of files. Thismight happen for example when using the ${HOST} macro in the filename while receiving messages from a large numberof hosts. To overcome this problem, adjust the --fd-limit command-line parameter of syslog-ng or the global ulimitparameter of your host. For setting the --fd-limit command-line parameter of syslog-ng see the syslog-ng(8) (p. 493)manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

191syslog-ng.com

file() destination options

create-dirs()yes or noType:noDefault:

Description: Enable creating non-existing directories.

dir-group()stringType:Use the global settingsDefault:

Description: The group of the directories created by syslog-ng. To preserve the original properties of an existingdirectory, use the option without specifying an attribute: dir-group().

dir-owner()stringType:Use the global settingsDefault:

Description: The owner of the directories created by syslog-ng. To preserve the original properties of an existingdirectory, use the option without specifying an attribute: dir-owner().

dir-perm()number (octal notation)Type:Use the global settingsDefault:

Description: The permission mask of directories created by syslog-ng. Log directories are only created if a fileafter macro expansion refers to a non-existing directory, and directory creation is enabled (see also thecreate-dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute:dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), thedefault permission of the directories is masked with the umask of the parent process (typically 0022).

flags()no-multi-line, syslog-protocol, threadedType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message header

192syslog-ng.com

file() destination options

is formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

■ threaded: The threaded flag enables multithreading for the destination. For details on multithreading,see Chapter 18, Multithreading and scaling in syslog-ng PE (p. 449).

NoteThe file destination uses multiple threads only if the destination filename contains macros.

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

193syslog-ng.com

file() destination options

fsync()yes or noType:noDefault:

Description: Forces an fsync() call on the destination fd after each write. Note: enabling this option mayseriously degrade performance.

group()stringType:Use the global settingsDefault:

Description: Set the group of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: group().

local-time-zone()name of the timezone, or the timezone offsetType:The local timezone.Default:

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

194syslog-ng.com

file() destination options

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

195syslog-ng.com

file() destination options

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

overwrite-if-older()number (seconds)Type:0Default:

Description: If set to a value higher than 0, syslog-ng PE checks when the file was last modified before startingto write into the file. If the file is older than the specified amount of time (in seconds), then syslog-ng removesthe existing file and opens a new file with the same name. In combination with for example the ${WEEKDAY}macro, this can be used for simple log rotation, in case not all history has to be kept. (Note that in this weeklylog rotation example if its Monday 00:01, then the file from last Monday is not seven days old, because it wasprobably last modified shortly before 23:59 last Monday, so it is actually not even six days old. So in this case,set the overwrite-if-older() parameter to a-bit-less-than-six-days, for example, to 518000 seconds.

owner()stringType:Use the global settingsDefault:

Description: Set the owner of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: owner().

pad-size()number (bytes)Type:0Default:

Description: If set, syslog-ng PE will pad output messages to the specified size (in bytes). Some operatingsystems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the blocksize. (HP-UX uses 2048 bytes).

WarningHazard of data loss! If the size of the incoming message is larger than the previously set pad-size() value, syslog-ng willtruncate the message to the specified size. Therefore, all message content above that size will be lost.

perm()number (octal notation)Type:Use the global settingsDefault:

196syslog-ng.com

file() destination options

Description: The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the numberwith 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing file, use the option without specifying an attribute: perm().

suppress()secondsType:0 (disabled)Default:

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

template-escape()yes or noType:noDefault:

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

197syslog-ng.com

file() destination options

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.4. Storing messages on the Hadoop Distributed File System (HDFS)

Starting with version 5.3, syslog-ng PE can send plain-text log files to the Hadoop Distributed File System(HDFS), allowing you to store your log data on a distributed, scalable file system. This is especially useful ifyou have huge amount of log messages that would be difficult to store otherwise, or if you want to process yourmessages using Hadoop tools (for example, Apache Pig).

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

Note the following limitations when using the syslog-ng PE hdfs destination:

■ This destination is only supported on the Linux platforms that use the linux glibc2.11 installer,including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (TrustyTahr).

■ Since syslog-ng PE uses the official Java HDFS client, the hdfs destination has significant memoryusage (about 400MB).

■ The syslog-ng PE application always creates a new file if the previous has been closed. Appendingdata to existing files is not supported.

■ Macros are not supported in the file path and the filename. You can use only simple file paths foryour log files, for example, /usr/hadoop/logfile.txt.

■ You cannot set when log messages are flushed. Hadoop performs this action automatically, dependingon its configured block size, and the amount of data received. There is no way for the syslog-ng PEapplication to influence when the messages are actually written to disk. This means that syslog-ngPE cannot guarantee that a message sent to HDFS is actually written to disk. When using flow-controlor RLTP™, syslog-ng PE acknowledges a message as written to disk when it passes the message tothe HDFS client. This method is as reliable as your HDFS environment.

■ The log messages of the underlying client libraries are available in the internal() source ofsyslog-ng PE.

NoteThe hdfs destination has been tested with Hortonworks Data Platform.

198syslog-ng.com

Storing messages on the Hadoop Distributed File System (HDFS)

Declaration:

@module mod-java

@include "scl.conf"

hdfs(

client_lib_dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-hadoop-libraries>")

hdfs_uri("hdfs://NameNode:8020")

hdfs_file("<path-to-logfile>")

);

Example 7.10. Storing logfiles on HDFSThe following example defines an hdfs destination using only the required parameters.

@module mod-java

@include "scl.conf"

destination d_hdfs {

hdfs(

client_lib_dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/hadoop/libs")

hdfs_uri("hdfs://10.140.32.80:8020")

hdfs_file("/user/log/logfile.txt")

);

};

■ To install the software required for the hdfs destination, see Procedure 7.4.1, Prerequisites (p. 199).

■ For details on how the hdfs destination works, see Procedure 7.4.2, How syslog-ng PE interactswith HDFS (p. 200).

■ For details on using MapR-FS, see Procedure 7.4.3, Storing messages with MapR-FS (p. 200).

■ For the list of options, see Section 7.4.4, HDSF destination options (p. 201).

7.4.1. Procedure – Prerequisites

To send messages from syslog-ng PE to HDFS, complete the following steps.

Steps:

Step 1. If you want to use the Java-based modules of syslog-ng PE (for example, the Elasticsearch, HDFS, orKafka destinations), download and install the Java Runtime Environment (JRE), 1.7 (or newer).The Java-based modules of syslog-ng PE are tested and supported when using the Oracle implementationof Java. Other implementations are untested and unsupported, they may or may not work as expected.

Step 2. Download the Hadoop Distributed File System (HDFS) libraries (version 2.x) fromhttp://hadoop.apache.org/releases.html.

Step 3. Extract the HDFS libraries into a temporary directory, then collect the various .jar files into a singledirectory (for example, /opt/hadoop/lib/) where syslog-ng PE can access them. You must specifythis directory in the syslog-ng PE configuration file. The files are located in the various lib directoriesunder the share/ directory of the Hadoop release package. (For example, in Hadoop 2.7, required

199syslog-ng.com

Storing messages on the Hadoop Distributed File System (HDFS)

files are common/hadoop-common-2.7.0.jar, common/libs/*.jar,hdfs/hadoop-hdfs-2.7.0.jar, hdfs/lib/*, but this may change between Hadoop releases, soit is easier to copy every .jar file into a single directory.

7.4.2. Procedure – How syslog-ng PE interacts with HDFS

The syslog-ng PE application sends the log messages to the official HDFS client library, which forwards thedata to the HDFS nodes. The way how syslog-ng PE interacts with HDFS is described in the following steps.

Step 1. After syslog-ng PE is started and the first message arrives to the hdfs destination, the hdfs destinationtries to connect to the HDFS NameNode. If the connection fails, syslog-ng PE will repeatedly attemptto connect again after the period set in time-reopen() expires.

Step 2. syslog-ng PE checks if the path to the logfile exists. If a directory does not exist syslog-ng PEautomatically creates it. syslog-ng PE creates the destination file (using the filename set in the syslog-ngPE configuration file, with a UUID suffix to make it unique, for example,/usr/hadoop/logfile.txt.3dc1c59e-ab3b-4b71-9e81-93db477ed9d9) and writes themessage into the file. After the file is created, syslog-ng PE will write all incoming messages into thehdfs destination.

NoteYou cannot set when log messages are flushed. Hadoop performs this action automatically, depending on itsconfigured block size, and the amount of data received. There is no way for the syslog-ng PE application toinfluence when the messages are actually written to disk. This means that syslog-ng PE cannot guarantee thata message sent to HDFS is actually written to disk. When using flow-control or RLTP™, syslog-ng PEacknowledges a message as written to disk when it passes the message to the HDFS client. This method isas reliable as your HDFS environment.

Step 3. If the HDFS client returns an error, syslog-ng PE attempts to close the file, then opens a new file andrepeats sending the message (trying to connect to HDFS and send the message), as set in the retries()parameter. If sending the message fails for retries() times, syslog-ng PE drops the message.

Step 4. The syslog-ng PE application closes the destination file in the following cases:

■ syslog-ng PE is reloaded

■ syslog-ng PE is restarted

■ The HDFS client returns an error.

Step 5. If the file is closed and you have set an archive directory, syslog-ng PE moves the file to this directory.If syslog-ng PE cannot move the file for some reason (for example, syslog-ng PE cannot connect tothe HDFS NameNode), the file remains at its original location, syslog-ng PE will not try to move itagain.

7.4.3. Procedure – Storing messages with MapR-FS

The syslog-ng PE application is also compatible with MapR File System (MapR-FS), starting from version 5.4,syslog-ng Premium Edition is MapR certified. MapR-FS provides better performance, reliability, efficiency,maintainability, and ease of use compared to the default Hadoop Distributed Files System (HDFS). To useMapR-FS with syslog-ng PE, complete the following steps:

200syslog-ng.com

Storing messages on the Hadoop Distributed File System (HDFS)

Step 1. Install MapR libraries. Instead of the official Apache HDFS libraries, MapR uses different libraries.The supported version is MapR 4.x.

Step a. Download the libraries from the Maven Repository and Artifacts for MapR or get itfrom an already existing MapR installation.

Step b. Install MapR. If you do not know how to install MapR, follow the instructions on theMapR website.

Step 2. In a default MapR installation, the required libraries are installed in the following path: /opt/mapr/lib.

Enter the path where MapR was installed in the class-path option of the hdfs destination, forexample:

class_path("/opt/mapr/lib/")

If the libraries were downloaded from the Maven Repository, the following additional libraries willbe requiered. Note that the version numbers in the filenames can be different in the various Hadoopreleases:commons-collections-3.2.1.jar, commons-logging-1.1.3.jar,hadoop-auth-2.5.1.jar, log4j-1.2.15.jar, slf4j-api-1.7.5.jar,commons-configuration-1.6.jar, guava-13.0.1.jar, hadoop-common-2.5.1.jar,maprfs-4.0.2-mapr.jar, slf4j-log4j12-1.7.5.jar, commons-lang-2.5.jar,hadoop-0.20.2-dev-core.jar, json-20080701.jar, protobuf-java-2.5.0.jar,zookeeper-3.4.5-mapr-1406.jar.

Step 3. Configure the hdfs destination in syslog-ng PE.

Example 7.11. Storing logfiles with MapR-FSThe following example defines an hdfs destination for MapR-FS using only the required parameters.

@module mod-java

@include "scl.conf"

destination d_mapr {

hdfs(

client_lib_dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:/opt/mapr/lib/")

hdfs_uri(maprfs://10.140.32.80")

hdfs_file("/user/log/logfile.txt")

);

};

7.4.4. HDSF destination options

The hdfs destination stores the log messages in files on the Hadoop Distributed File System (HDFS). Thehdfs destination has the following options.

The following options are required: hdfs_file(), hdfs_uri(). Note that to use hdfs, you must add thefollowing lines to the beginning of your syslog-ng PE configuration:

@module mod-java

@include "scl.conf"

201syslog-ng.com

HDSF destination options

client_lib_dir()stringType:N/ADefault:

Description: Include the path to the directory where you copied the required libraries (see Procedure 7.4.1,Prerequisites (p. 199)), for example, client_lib_dir(/user/share/hadoop/lib).

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

202syslog-ng.com

HDSF destination options

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.12. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

203syslog-ng.com

HDSF destination options

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

hdfs_archive_dir()stringType:N/ADefault:

Description: The path where syslog-ng PE will move the closed log files. If syslog-ng PE cannot move thefile for some reason (for example, syslog-ng PE cannot connect to the HDFS NameNode), the file remains atits original location. For example, hdfs_archive_dir("/usr/hdfs/archive/").

hdfs_file()stringType:N/ADefault:

Description:The path and name of the log file. For example, hdfs_file("/usr/hdfs/mylogfile.txt").syslog-ng PE checks if the path to the logfile exists. If a directory does not exist syslog-ng PE automaticallycreates it.

Macros are not supported in the file path and the filename. You can use only simple file paths for your log files,for example, /usr/hadoop/logfile.txt.

204syslog-ng.com

HDSF destination options

hdfs_max_filename_length()numberType:255Default:

Description: The maximum length of the filename. This filename (including the UUID that syslog-ng PEappends to it) cannot be longer than what the file system permits. If the filename is longer than the value ofhdfs_max_filename_length, syslog-ng PE will automatically truncate the filename. For example,hdfs_max_filename_length("255").

hdfs_resources()stringType:N/ADefault:

Description: The list of Hadoop resources to load, separated by semicolons. For example,hdfs_resources("/home/user/hadoop/core-site.xml;/home/user/hadoop/hdfs-site.xml").

hdfs_uri()stringType:N/ADefault:

Description:The URI of the HDFS NameNode is in hdfs://IPaddress:port or hdfs://hostname:portformat. When using MapR-FS, the URI of the MapR-FS NameNode is in maprfs://IPaddress ormaprfs://hostname format, for example: maprfs://10.140.32.80. The IP address of the node can beIPv4 or IPv6. For example, hdfs_uri("hdfs://10.140.32.80:8020"). The IPv6 address must be enclosedi n s q u a r e b r a c k e t s ( [ ] ) a s s p e c i f i e d b y R F C 2 7 3 2 , f o r e x a m p l e ,hdfs_uri("hdfs://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:8020").

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:Use the global setting (which defaults to drop-message)Default:

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

205syslog-ng.com

HDSF destination options

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

retries()number (of attempts)Type:3Default:

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ngPE could not send a message, it will try again until the number of attempts reaches retries, then drops themessage.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

206syslog-ng.com

HDSF destination options

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.5. Publishing messages to Apache Kafka

Starting with version 5.4, syslog-ng PE can directly publish log messages to the Apache Kafka message bus,where subscribers can access them.

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

■ This destination is only supported on the Linux platforms that use the linux glibc2.11 installer,including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (TrustyTahr).

■ Since syslog-ng PE uses the official Java Kafka producer, the kafka destination has significantmemory usage.

■ The log messages of the underlying client libraries are available in the internal() source ofsyslog-ng PE.

Declaration:

@module mod-java

@include "scl.conf"

kafka(

client_lib_dir(/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>)

kafka_bootstrap_servers("1.2.3.4:9092,192.168.0.2:9092")

topic("${HOST}")

207syslog-ng.com

Publishing messages to Apache Kafka

);

Example 7.13. Sending log data to Apache KafkaThe following example defines a kafka destination, using only the required parameters.

@module mod-java

@include "scl.conf"

destination d_kafka {

kafka(

client_lib_dir(/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib)

kafka_bootstrap_servers("1.2.3.4:9092,192.168.0.2:9092")

topic("${HOST}")

);

};

■ To install the software required for the kafka destination, see Procedure 7.5.1, Prerequisites (p. 208).

■ For details on how the kafka destination works, see Section 7.5.2, How syslog-ng PE interacts withApache Kafka (p. 209).

■ For the list of options, see Section 7.5.3, Kafka destination options (p. 209).

7.5.1. Procedure – Prerequisites

To publish messages from syslog-ng PE to Apache Kafka, complete the following steps.

Steps:

Step 1. If you want to use the Java-based modules of syslog-ng PE (for example, the Elasticsearch, HDFS, orKafka destinations), download and install the Java Runtime Environment (JRE), 1.7 (or newer).The Java-based modules of syslog-ng PE are tested and supported when using the Oracle implementationof Java. Other implementations are untested and unsupported, they may or may not work as expected.

Step 2. Download the latest stable binary release of the Apache Kafka libraries (version 0.8.2 or newer) fromhttp://kafka.apache.org/downloads.html.

Step 3. Extract the Apache Kafka libraries into a single directory. If needed, collect the various .jar filesinto a single directory (for example, /opt/kafka/lib/) where syslog-ng PE can access them. Youmust specify this directory in the syslog-ng PE configuration file.

Step 4. Check if the following files in the Kafka libraries have the same version number:slf4j-api-<version-number>.jar, slf4j-log4j12-<version-number>.jar. If the versionnumber of these files is different, complete the following steps:

Step a. Delete one of the files (for example, slf4j-log4j12-<version-number>.jar).

Step b. Download a version that matches the version number of the other file (for example,1.7.6) from the official SLF4J distribution.

Step c. Copy the downloaded file into the directory of your Kafka library files (for example,/opt/kafka/lib/).

208syslog-ng.com

Publishing messages to Apache Kafka

7.5.2. How syslog-ng PE interacts with Apache Kafka

When stopping the syslog-ng PE application, syslog-ng PE will not stop until all Java threads are finished,including the threads started by the Kafka Producer. There is no way (except for the kill -9 command) tostop syslog-ng PE before the Kafka Producer stops. To change this behavior set the properties of the KafkaProducer in its properties file, and reference the file in the properties-file option.

The syslog-ng PE kafka destination tries to reconnect to the brokers in a tight loop. This can look as spinning,because of a lot of similar debug messages. To decrease the amount of such messages, set a bigger timeoutusing the following properties:

retry.backoff.ms=1000

reconnect.backoff.ms=1000

For details on using property files, see Section properties_file() (p. 213). For details on the properties that youcan set in the property file, see the Apache Kafka documentation.

7.5.3. Kafka destination options

The kafka destination of syslog-ng PE can directly publish log messages to the Apache Kafka message bus,where subscribers can access them. The kafka destination has the following options.

Required options:

The following options are required: kafka-bootstrap-servers(), topic(). Note that to use kafka, youmust add the following lines to the beginning of your syslog-ng PE configuration:

@module mod-java

@include "scl.conf"

client_lib_dir()stringType:N/ADefault:

Description: Include the path to the directory where you copied the required libraries (see Procedure 7.5.1,Prerequisites (p. 208)), for example, client_lib_dir(/user/share/kafka/lib).

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

209syslog-ng.com

How syslog-ng PE interacts with Apache Kafka

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

210syslog-ng.com

Kafka destination options

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.14. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

211syslog-ng.com

Kafka destination options

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

kafka_bootstrap_servers()list of hostnamesType:

Default:

Description: Specifies the hostname or IP address of the Kafka server. When specifying an IP address, IPv4(for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well. Use a colon (:) after theaddress to specify the port number of the server. When specifying multiple addresses, use a comma to separatet h e a d d r e s s e s , f o r e x a m p l e ,kafka-bootstrap-servers("127.0.0.1:2525,remote-server-hostname:6464")

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:Use the global setting (which defaults to drop-message)Default:

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

212syslog-ng.com

Kafka destination options

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

key()templateType:N/ADefault:

Description: The key of the partition under which the message is published. You can use templates to changethe topic dynamically based on the source or the content of the message, for example, key("${PROGRAM}").

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

properties_file()string (absolute path)Type:N/ADefault:

Description: The absolute path and filename of the Kafka properties file to load. For example,properties_file("/opt/syslog-ng/etc/kafka_dest.properties"). The syslog-ng PE applicationreads this file and passes the properties to the Kafka Producer. If a property is defined both in the syslog-ng PEconfiguration file (syslog-ng.conf) and in the properties file, then syslog-ng PE uses the definition fromthe syslog-ng PE configuration file.

The syslog-ng PE kafka destination supports all properties of the official Kafka producer. For details, see theApache Kafka documentation.

The kafka-bootstrap-servers option is translated to the bootstrap.servers property.

For example, the following properties file defines the acknowledgement method and compression:

acks=all

compression.type=snappy

retries()number (of attempts)Type:3Default:

213syslog-ng.com

Kafka destination options

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ngPE could not send a message, it will try again until the number of attempts reaches retries, then drops themessage.

template()template or template functionType:$ISODATE $HOST $MSGHDR$MSG\nDefault:

Description: The message as published to Apache Kafka. You can use templates and template functions (forexample, format-json()) to format the message, for example, template("$(format-json --scope

rfc5424 --exclude DATE --key ISODATE)").

For details on formatting messages in JSON format, see Section format-json (p. 393).

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

topic()templateType:N/ADefault:

Description: The Kafka topic under which the message is published. You can use templates to change the topicdynamically based on the source or the content of the message, for example, topic("${HOST}").

sync_send()true | falseType:falseDefault:

Description: When sync_send is set to true, syslog-ng PE sends the message reliably: it sends a messageto the Kafka server, then waits for a reply. In case of failure, syslog-ng PE repeats sending the message, as setin the retries() parameter. If sending the message fails for retries() times, syslog-ng PE drops themessage.

This method ensures reliable message transfer, but is very slow.

When sync_send is set to false, syslog-ng PE sends messages asynchronously, and receives the responseasynchronously. In case of a problem, syslog-ng PE cannot resend the messages.

This method is fast, but the transfer is not reliable. Several thousands of messages can be lost before syslog-ngPE recognizes the error.

214syslog-ng.com

Kafka destination options

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.6. Storing messages in encrypted files

The syslog-ng PE application can store log messages securely in encrypted, compressed and timestamped binaryfiles. Timestamps can be requested from an external Timestamping Authority (TSA).

Logstore files consist of individual chunks, every chunk can be encrypted, compressed, and timestampedseparately. Chunks contain compressed log messages and header information needed for retrieving messagesfrom the logstore file.

The syslog-ng PE application generates an SHA-1 hash for every chunk to verify the integrity of the chunk.The hashes of the chunks are chained together to prevent injecting chunks into the logstore file. The syslog-ngPE application can encrypt the logstore using various algorithms, using the aes128 encryption algorithm inCBC mode and the hmac-sha1 hashing (HMAC) algorithm as default. For other algorithms, see Sectioncipher() (p. 219) and Section digest() (p. 220).

The destination filename may include macros which get expanded when the message is written, thus a simplelogstore() driver may create several files. For more information on available macros see Section 14.1.5,Macros of syslog-ng PE (p. 383).

If the expanded filename refers to a directory which does not exist, it will be created depending on thecreate-dirs() setting (both global and a per destination option).

The logstore() has a single required parameter that specifies the filename that stores the log messages. Forthe list of available optional parameters, see Section 7.6.3, logstore() destination options (p. 219).

215syslog-ng.com

Storing messages in encrypted files

WarningHazard of data loss! If your log files are on an NFS-mounted network file system, see Section 2.14, NFS file system forlog files (p. 34).

Declaration:

logstore(filename options());

Example 7.15. Using the logstore() driverA simple example saving and compressing log messages.

destination d_logstore { logstore("/var/log/messages.lgs" compress(5) ); };

NoteOn Microsoft Windows platforms, enclose paths and filenames between single quotes, forexample, 'C:\temp\logs\mylogs.log'

A more detailed example that encrypts messages, modifies the parameters for closing chunks, and sets file privileges.

destination d_logstore { logstore("/var/log/messages-logstore.lgs"

encrypt-certificate("/opt/syslog-ng/etc/syslog-ng/keys/10-100-20-40/public-certificate-of-the-server.pem")

owner("balabit")

group("balabit")

perm(0777)

); };

The URL to the Timestamping Authority and if needed, the OID of the timestamping policy can be set as global options,or also per logstore destination. The following example specifies the URL and the OID as global options:

options {

timestamp-url("http://10.50.50.50:8080/");

timestamp-policy("0.4.0.2023.1.1");

};

NoteWhen using the logstore() destination, update the configuration of your log rotation program to rotate these files.Otherwise, the log files can become very large.

WarningSince the state of each created file must be tracked by syslog-ng, it consumes some memory for each file. If no newmessages are written to a file within 60 seconds (controlled by the time-reap() global option), it is closed, and its stateis freed.

Exploiting this, a DoS attack can be mounted against the system. If the number of possible destination files and its neededmemory is more than the amount available on the syslog-ng server.

The most suspicious macro is ${PROGRAM}, where the number of possible variations is rather high. Do not use the${PROGRAM} macro in insecure environments.

216syslog-ng.com

Storing messages in encrypted files

7.6.1. Displaying the contents of logstore files

To display the contents of a logstore file, use the lgstool (formerly called logcat) command supplied withsyslog-ng, for example lgstool cat /var/log/messages.lgs. Log messages available in the journal fileof the logstore (but not yet written to the logstore file itself) are displayed as well.

To display the contents of encrypted log files, specify the private key of the certificate used to encrypt the file,for example lgstool cat -k private.key /var/log/messages.lgs. The contents of the file are sentto the standard output, so it is possible to use grep and other tools to find particular log messages, for examplelgstool cat /var/log/messages.lgs |grep 192.168.1.1. For further details, see lgstool(1) (p. 472).

TipThe lgstool utility is available for Microsoft Windows operating systems at the syslog-ng Downloads page.

WarningFor files that are in use by syslog-ng, the last chunk that is open cannot be read.

7.6.2. Journal files

Starting with syslog-ng Premium Edition 3.2, syslog-ng PE processes log messages into a journal file beforewriting them to the logstore file. That way logstore files are consistent even if syslog-ng PE crashes unexpectedly,avoiding losing messages. Note that this does not protect against losing messages if the operating system crashes.

A journal file is automatically created for every logstore file that syslog-ng PE opens. A journal file consistsof journal blocks which store the log messages. When a journal block fills up with messages, syslog-ng PEwrites the entire block into the logstore file and starts to reuse the journal block (one journal block becomesone chunk in the logstore file). If the messages cannot be written to the logstore file (for example, because thedisk becomes unaccessible, or file operations are slow), messages are put to the next journal block (syslog-ngPE uses four blocks by default). When all journal blocks become full, syslog-ng PE will stop processing incomingtraffic. syslog-ng PE starts accepting messages to the logstore file again when the first journal block issuccessfully written to the logstore file. If syslog-ng PE receives a HUP or STOP signal, or no new messagearrives into the logstore for the period set in the time-reap() parameter, it writes every journal block to thelogstore.

When syslog-ng PE is restarted, it automatically processes the journal files to the logstore files, unless a particularlogstore file is not part of configuration of syslog-ng PE. Such orphaned journal files can be processed withthe lgstool recover command. For details on processing orphaned journal files, see the section called “Therecover command” (p. 474).

WarningIf a particular logstore destination receives messages at a constant but very low message rate (for example,a 100-byte message every 30 seconds), messages do not get written to the logstore file for a long time,because the journal block does not get full, and messages are more frequent than the time-reap() time.This becomes a problem when using logrotate to rotate the logstore files, because log messages will not be

■

217syslog-ng.com

Displaying the contents of logstore files

in the files they are expected. To avoid this situation, either use time-based macros in the filenames of thelogstore files, or send a HUP signal to syslog-ng PE right before rotating the logstore files.

■ When every block of a journal becomes full and syslog-ng PE stops processing incoming traffic, it will notread new messages at all until a block is successfully written to the related logstore file. This is in contrastwith flow-control, where only messages from the source related to the particular destination are not processed.

■ The messages in the journal file are in plain-text format: they are neither encrypted nor compressed. Thejournal file has the same permission as the logstore, by default, root privileges are required to access them.Make sure you consider this if you change the permissions of the journal file (owner, group, perm) in thesyslog-ng PE configuration file.

NoteJournal files are located in the same folder as the logstore file. The name of the journal file is the same as the logstorefile with .jor suffix added. For example, the journal file for messages.lgs is messages.lgs.jor.

The syslog-ng PE application uses a separate journal file for every logstore file. Every journal file is processedby a separate thread. The journal files are mapped into the memory. The journal of an individual logstore fileuses up to journal-block-size()*journal-block-count()memory address, which is 4MB by default.However, if you have several logstore files open in parallel (for example, you are collecting log messages from500 hosts and storing them in separate files for every host, and the hosts are continuously sending messages)the memory requirements for journaling rise quickly (to ~2GB for the 500 hosts). To limit the memory use ofjournals, adjust the logstore-journal-shmem-threshold() global option (by default, it is 512 MB).

If the memory required for the journal files exceeds the logstore-journal-shmem-threshold() limit,syslog-ng PE will store only a single journal block of every journal file in the memory, and — if more blocksare needed for a journal — store the additional blocks on the hard disk. Opening new logstore files meansallocating memory for one new journal block for every new file. In extreme situations involving large traffic,this can lead to syslog-ng PE consuming the entire memory of the system. Adjust the journal-block-size()and your file-naming conventions as needed to avoid such situations. For details on logstore journals, see Section7.6.2, Journal files (p. 217).

WarningIf you have a large amount of open logstore files in parrallel (for example, you are using the ${HOST} or ${PROGRAM}macros in your filenames) consider lowering the journal-block-size() to avoid syslog-ng PE consuming the entirememory of the system.

Example 7.16. Calculating memory usage of logstore journalsIf you are using the default settings (4 journal blocks for every logstore journal, one block is 1MB,logstore-journal-shmem-threshold() is 512MB), this means that syslog-ng PE will allocate 4MB memory forevery open logstore file, up to 512MB if you have 128 open logstore files. Opening a new logstore file would require 4more megabytes of memory for journaling, bringing the total required memory to 516MB, which is above thelogstore-journal-shmem-threshold(). In this case, syslog-ng PE switches to storing only a single journal blockin the memory, lowering the memory requirements of journaling to 129MB. However, opening more and more logstorefiles will require more and more memory, and this is not limited, except when syslog-ng PE reaches the maximum numberof files that can be open (as set in the --fd-limit command-line option).

218syslog-ng.com

Journal files

7.6.3. logstore() destination options

The logstore driver stores log messages in binary files that can be encrypted, compressed, checked for integrity,and timestamped by an external Timestamping Authority (TSA). Otherwise, it is very similar to the file()destination.

WarningWhen creating several thousands separate log files, syslog-ng might not be able to open the required number of files. Thismight happen for example when using the ${HOST} macro in the filename while receiving messages from a large numberof hosts. To overcome this problem, adjust the --fd-limit command-line parameter of syslog-ng or the global ulimitparameter of your host. For setting the --fd-limit command-line parameter of syslog-ng see the syslog-ng(8) (p. 493)manual page. For setting the ulimit parameter of the host, see the documentation of your operating system.

NoteWhen using this destination, update the configuration of your log rotation program to rotate these files. Otherwise, thelog files can become very large.

Also, after rotating the log files, reload syslog-ng PE using the syslog-ng-ctl reload command, or use anothermethod to send a SIGHUP to syslog-ng PE.

The logstore() has a single required parameter that specifies the filename that stores the log messages.

Declaration:

logstore(filename options());

The logstore() destination has the following options:

cipher()stringType:aes-128-cbcDefault:

Description: Set the cipher method used to encrypt the logstore. The following cipher methods are available:aes-128-cbc, aes-128-cfb, aes-128-cfb1, aes-128-cfb8, aes-128-ecb, aes-128-ofb ,aes-192-cbc, aes-192-cfb, aes-192-cfb1, aes-192-cfb8, aes-192-ecb, aes-192-ofb ,aes-256-cbc, aes-256-cfb, aes-256-cfb1, aes-256-cfb8, aes-256-ecb, aes-256-ofb , aes128, aes192 , aes256, bf , bf-cbc , bf-cfb, bf-ecb , bf-ofb , blowfish, cast , cast-cbc , cast5-cbc, cast5-cfb, cast5-ecb, cast5-ofb , des, des-cbc, des-cfb , des-cfb1 , des-cfb8 , des-ecb ,des-ede, des-ede-cbc, des-ede-cfb , des-ede-ofb, des-ede3 , des-ede3-cbc, des-ede3-cfb,des-ede3-ofb, des-ofb , des3 , desx , desx-cbc, rc2, rc2-40-cbc , rc2-64-cbc, rc2-cbc, rc2-cfb,rc2-ecb , rc2-ofb, rc4, and rc4-40. By default, syslog-ng PE uses the aes-128-cbc method.

Note that the size of the digest hash must be equal to or larger than the key size of the cipher method. Forexample, to use the aes-256-cbc cipher method, the digest method must be at least SHA-256. This option isavailable in syslog-ng PE 3.1 and later.

219syslog-ng.com

logstore() destination options

chunk-size()number (kilobytes)Type:128Default:

Description: This option is obsolete. Use the journal-block-size() option instead.

Size of a logstore chunk in kilobytes. Note that this size refers to the compressed size of the chunk. Also, thegzip library used for compressing the messages has a 32k long buffer, so messages may not appear in the actuallogfile until this buffer is not filled. Logstore chunks are closed when they reach the specified size, or whenthe time limit set in chunk-time() expires.

chunk-time()number (seconds)Type:5Default:

Description: This option is obsolete.

Time limit in seconds: syslog-ng PE closes the chunk if no new messages arrive until the time limit expires.Logstore chunks are closed when the time limit expires, or when they reach the size specified in thechunk-size() parameter. If the time limit set in the time-reap() parameter expires, the entire file is closed.

compress()number (between 0-9)Type:3Default:

Description: Compression level. 0 means uncompressed files, while 1-9 is the compression level used by gzip(9 means the highest but slowest compression, 3 is usually a good compromise).

create-dirs()yes or noType:noDefault:

Description: Enable creating non-existing directories.

digest()stringType:SHA1Default:

Description: Set the digest method to use. The following digest methods are available: MD4, MD5, SHA0 (SHA),SHA1, RIPEMD160, SHA224, SHA256, SHA384, and SHA512. By default, syslog-ng PE uses the SHA1 method.

Note that the size of the digest hash must be equal to or larger than the key size of the cipher method. Forexample, to use the aes-256-cbc cipher method, the digest method must be at least SHA256. This option isavailable in syslog-ng PE 3.1 and later.

220syslog-ng.com

logstore() destination options

dir-group()stringType:Use the global settingsDefault:

Description: The group of the directories created by syslog-ng. To preserve the original properties of an existingdirectory, use the option without specifying an attribute: dir-group().

dir-owner()stringType:Use the global settingsDefault:

Description: The owner of the directories created by syslog-ng. To preserve the original properties of an existingdirectory, use the option without specifying an attribute: dir-owner().

dir-perm()number (octal notation)Type:Use the global settingsDefault:

Description: The permission mask of directories created by syslog-ng. Log directories are only created if a fileafter macro expansion refers to a non-existing directory, and directory creation is enabled (see also thecreate-dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute:dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), thedefault permission of the directories is masked with the umask of the parent process (typically 0022).

encrypt-certificate()filenameType:noneDefault:

Description:Name of a file, that contains an X.509 certificate (and the public key) in PEM format. The syslog-ngPE application uses this certificate to encrypt the logstore files which can be decrypted using the private keyof the certificate.

flags()serializedType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

■ The serialized flag instructs the driver to store the log messages in a serialized format. Whenusing the lgstool utility to display messages from the logstore, the messages can be reformattedwith a template only if the serialized flag has been enabled on the logstore.

221syslog-ng.com

logstore() destination options

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

group()stringType:Use the global settingsDefault:

Description: Set the group of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: group().

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

journal-block-count()number (1-255)Type:4Default:

Description: The number of blocks in the journal file. If set to 0, syslog-ng will set it to the default value (4).The maximal value is 255. If journal-block-count() is set higher than 255, syslog-ng will use the maximumvalue.

NoteBy default, journal files are mapped into the memory of the host. To influence the amount of memory addresses used byjournal files, see the logstore-journal-shmem-threshold() global option.

Example 7.17. Setting journal block number and sizeThe following example sets the size of a journal block to 512KB and increases the number of blocks to 5.

destination d_logstore {

logstore("/var/log/messages-logstore.lgs"

encrypt-certificate

("/opt/syslog-ng/etc/syslog-ng/keys/public-server-certificate.pem")

journal-block-size(524288)

journal-block-count(5));

};

222syslog-ng.com

logstore() destination options

journal-block-size()number (bytes)Type:1048576Default:

Description: The size of blocks (in bytes) in the journal file. The size of the block must be a multiple of thepage size: if not, syslog-ng PE automatically increases it to the next multiple of the page size. The maximumsize of a journal block is 32MB, the minimum size is 256KB. If the value specified as journal-block-size()is lower than minimum size or higher than the maximum size, syslog-ng PE will use the minimum or maximumsize, respectively.

NoteAt least one journal block for every logstore file open is mapped into the memory. For details on logstorejournals, see Section 7.6.2, Journal files (p. 217).

■

■ The size of the journal block is not equal with the size of logstore chunks, because the records in the logstorefile can be encrypted or compressed.

Example 7.18. Setting journal block number and sizeThe following example sets the size of a journal block to 512KB and increases the number of blocks to 5.

destination d_logstore {

logstore("/var/log/messages-logstore.lgs"

encrypt-certificate

("/opt/syslog-ng/etc/syslog-ng/keys/public-server-certificate.pem")

journal-block-size(524288)

journal-block-count(5));

};

owner()stringType:Use the global settingsDefault:

Description: Set the owner of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: owner().

perm()number (octal notation)Type:Use the global settingsDefault:

Description: The permission mask of the file if it is created by syslog-ng. For octal numbers prefix the numberwith 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing file, use the option without specifying an attribute: perm().

template()stringType:A format conforming to the default logfile format.Default:

223syslog-ng.com

logstore() destination options

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

timestamp-freq()number (seconds)Type:Use global setting.Default:

Description: The minimum time (in seconds) that should expire between two timestamping requests. Whensyslog-ng closes a chunk, it checks how much time has expired since the last timestamping request: if it ishigher than the value set in the timestamp-freq() parameter, it requests a new timestamp from the authorityset in the timestamp-url() parameter.

By default, timestamping is disabled: the timestamp-freq() global option is set to 0. To enable timestamping,set it to a positive value.

timestamp-policy()stringType:

Default:

Description: If the Timestamping Server has timestamping policies configured, specify the OID of the policyto use with this parameter. syslog-ng PE will include this ID in the timestamping requests sent to the TSA. Thisoption is available in syslog-ng PE 3.1 and later.

timestamp-url()stringType:Use global setting.Default:

Description: The URL of the Timestamping Authority used to request timestamps to sign logstore chunks.Note that syslog-ng PE currently supports only Timestamping Authorities that conform to RFC3161 InternetX.509 Public Key Infrastructure Time-Stamp Protocol, other protocols likeMicrosoft Authenticode Timestampingare not supported.

224syslog-ng.com

logstore() destination options

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.7. Storing messages in a MongoDB database

The mongodb() driver sends messages to aMongoDB database. MongoDB is a schema-free, document-orienteddatabase. For the list of available optional parameters, see Section 7.7.2, mongodb() destination options (p. 227).

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

mongodb(parameters);

The mongodb() driver does not support creating indexes, as that can be a very complex operation in MongoDB.If needed, the administrator of the MongoDB database must ensure that indexes are created on the collections.

The mongodb() driver does not add the _id field to the message: the MongoDB server will do that automatically,if none is present. If you want to override this field from syslog-ng PE, use the key() parameter of thevalue-pairs() option.

The syslog-ng PE mongodb() driver is compatible with MongoDB server version 1.4 and newer.

225syslog-ng.com

Storing messages in a MongoDB database

NoteBy default, syslog-ng PE handles every message field as a string. For details on how to send selected fields as other typesof data (for example, handle the PID as a number), see Section 2.12.1, Specifying data types in value-pairs (p. 26).

Example 7.19. Using the mongodb() driverThe following example creates a mongodb() destination using only default values.

destination d_mongodb {

mongodb();

};

The following example displays the default values, and is equivalent with the previous example.

destination d_mongodb {

mongodb(

servers("localhost:27017")

database("syslog")

collection("messages")

value-pairs(

scope("selected-macros" "nv-pairs" "sdata")

)

);

};

7.7.1. Procedure – How syslog-ng PE connects the MongoDB server

When syslog-ng PE connects the MongoDB server during startup, it completes the following steps.

Step 1. The syslog-ng PE application connects the first address listed in the servers() option.

Step 2. If the server is accessible and it is a master MongoDB server, syslog-ng PE authenticateson the server (if needed), then starts sending the log messages to the server.

■

■ If the server is not accessible, or it is not a master server in a MongoDB replicaset and itdoes not send the address of the master server, syslog-ng PE connects the next address listedin the servers() option.

■ If the server is not a master server in a MongoDB replicaset, but it sends the address of themaster server, syslog-ng PE connects the received address.

Step 3. When syslog-ng PE connects the master MongoDB server, it retrieves the list of replicas (from thereplSet option of the server), and appends this list to the servers() option.

Warning■ This means that syslog-ng PE can send log messages to addresses that are not listed in its

configuration.

■ Make sure to include the address of your master server in your syslog-ng PE configuration file,otherwise you risk losing log messages if all the addresses listed in the syslog-ng PE configurationare offline.

■ Addresses retrieved from the MongoDB servers are not stored, and can be lost when syslog-ngPE is restarted. The retrieved addresses are not lost if the server() option of the destinationwas not changed in the configuration file since the last restart.

■ The failover mechanism used in the mongodb() driver is different from the client-side failoverused in other drivers.

226syslog-ng.com

Storing messages in a MongoDB database

Step 4. The syslog-ng PE application attempts to connect another server if the servers() list contains atleast two addresses, and one of the following events happens:

■ The safe-mode() option is set to no, and the MongoDB server becomes unreachable.

■ The safe-mode() option is set to yes, and syslog-ng PE cannot insert a log message intothe database because of an error.

In such case, syslog-ng PE starts to connect the addresses in from the servers() list (starting fromthe first address) to find the new master server, authenticates on the new server (if needed), thencontinues to send the log messages to the new master server.

During this failover step, one message can be lost if the safe-mode() option is disabled.

Step 5. If the original master becomes accessible again, syslog-ng PE will automatically connect to the originalmaster.

7.7.2. mongodb() destination options

The mongodb() driver sends messages to a MongoDB database. MongoDB is a schema-free, document-orienteddatabase.

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

The mongodb() destination has the following options:

collection()stringType:messagesDefault:

Description: The name of the MongoDB collection where the log messages are stored (collections are similarto SQL tables). Note that the name of the collection must not start with a dollar sign ($), and that it may containdot (.) characters.

WarningHazard of data loss! The syslog-ng PE application does not verify that the specified collection name does not containinvalid characters. If you specify a collection with an invalid name, the log messages sent to the MongoDB database willbe irrevocably lost without any warning.

227syslog-ng.com

mongodb() destination options

database()stringType:syslogDefault:

Description: The name of the MongoDB database where the log messages are stored. Note that the name ofthe database must not start with a dollar sign ($) and it cannot contain dot (.) characters.

WarningHazard of data loss! The syslog-ng PE application does not verify that the specified database name does not containinvalid characters. If you specify a database with an invalid name, the log messages sent to the MongoDB database willbe irrevocably lost without any warning.

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

228syslog-ng.com

mongodb() destination options

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

229syslog-ng.com

mongodb() destination options

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.20. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

230syslog-ng.com

mongodb() destination options

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

local-time-zone()name of the timezone, or the timezone offsetType:The local timezone.Default:

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:Use the global setting (which defaults to drop-message)Default:

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

231syslog-ng.com

mongodb() destination options

password()stringType:n/aDefault:

Description: Password of the database user.

path()stringType:emptyDefault:

Description: If the path() option is set, syslog-ng PE will connect to the database using the specified UNIXdomain socket. Note that you cannot set the path() and the servers() options at the same time.

retries()number (of attempts)Type:3Default:

Description: The number of times syslog-ng PE attempts to send a message to this destination. If syslog-ngPE could not send a message, it will try again until the number of attempts reaches retries, then drops themessage.

For MongoDB operations, syslog-ng PE uses a one-minute timeout: if an operation times out, syslog-ng PEassumes the operation has failed.

safe-mode()yes or noType:yesDefault:

Description: If safe-mode() is enabled, syslog-ng PE performs an extra check after each insert to verify thatthe insert succeeded. The insert is successful only if this second check is successful.

Note that disabling this option increases the performance of the driver, but can result in message loss. Usingsafe-mode(yes) is technically equivalent of using the RLTP™ protocol between syslog-ng PE and theMongoDB server. If you use the reliable(yes) option of disk-buffer() in your destinations, make surethat the safe-mode() option of the mongodb() destination is set to yes.

servers()list of hostname:port pairsType:127.0.0.1:27017Default:

Description: Specifies the hostname or IP address and the port number of the database server. When specifyingan IP address, IPv4 (for example, 192.168.0.1) or IPv6 (for example, [::1]) can be used as well.

232syslog-ng.com

mongodb() destination options

To send the messages to a MongoDB replicaset, specify the addresses of the database servers as acomma-separated list, for example: servers(192.168.1.1:27017,192.168.3.3:27017)

For details on how syslog-ng PE connects the MongoDB server, see Procedure 7.7.1, How syslog-ng PEconnects the MongoDB server (p. 226).

To connect to the server using a UNIX domain socket, use path option. Note that you cannot set the path()and the servers() options at the same time.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

username()stringType:n/aDefault:

Description: Name of the database user. Note that the mongodb() driver currently does not supportTLS-encrypted authentication.

value-pairs()parameter list of the value-pairs() optionType:scope("selected-macros" "nv-pairs")Default:

Description: The value-pairs() option creates structured name-value pairs from the data and metadata ofthe log message. For details on using value-pairs(), see Section 2.12, Structuring macros, metadata, andother value-pairs (p. 25).

NoteEmpty keys are not logged.

233syslog-ng.com

mongodb() destination options

7.8. Sending messages to a remote log server using the RFC3164 protocol (network()driver)

The network() destination driver can send syslog messages conforming to RFC3164 from the network usingthe TCP, TLS, and UDP networking protocols.

You can use the RLTP™ protocol as well. For details about the RLTP™ protocol, see Chapter 12, ReliableLog Transfer Protocol™ (p. 364).

■ UDP is a simple datagram oriented protocol, which provides "best effort service" to transfer messagesbetween hosts. It may lose messages, and no attempt is made to retransmit lost messages. TheBSD-syslog protocol traditionally uses UDP.Use UDP only if you have no other choice.

For details on minimizing message loss when using UDP, see the following tutorial:Collecting logmessages from UDP sources.

■ TCP provides connection-oriented service: the client and the server establish a connection, eachmessage is acknowledged, and lost packets are resent. TCP can detect lost connections, and messagesare lost, only if the TCP connection breaks. When a TCP connection is broken, messages that theclient has sent but were not yet received on the server are lost.

■ The syslog-ng application supports TLS (Transport Layer Security, also known as SSL) over TCP.For details, see Section 10.2, Encrypting log messages with TLS (p. 351).

When you send your log messages from a syslog-ng PE client through the network to a syslog-ng PE server,you can use different protocols and options. Every combination has its advantages and disadvantages. The mostimportant thing is to use matching protocols and options, so the server handles the incoming log messagesproperly. For details, see Section 2.13, Things to consider when forwarding messages between syslog-ng PEhosts (p. 31).

Declaration:

network("<destination-address>" [options]);

The network() destination has a single required parameter that specifies the destination host address wheremessages should be sent. If name resolution is configured, you can use the hostname of the target server. Bydefault, syslog-ng PE sends messages using the TCP protocol to port 601.

Example 7.21. Using the network() driverTCP destination that sends messages to 10.1.2.3, port 1999:

destination d_tcp { network("10.1.2.3" port(1999)); };

If name resolution is configured, you can use the hostname of the target server as well.

destination d_tcp { network("target_host" port(1999)); };

TCP destination that sends messages to the ::1 IPv6 address, port 2222.

destination d_tcp6 {

network(

"::1"

port(2222)

transport(tcp)

ip-protocol(6)

234syslog-ng.com

Sending messages to a remote log server using the RFC3164 protocol (network() driver)

);

};

To send messages using the IETF-syslog message format without using the IETF-syslog protocol, enable thesyslog-protocol flag. (For details on how to use the IETF-syslog protocol, see Section 7.14.1, syslog() destinationoptions (p. 289).)

destination d_tcp { network("10.1.2.3" port(1999) flags(syslog-protocol) ); };

7.8.1. network() destination options

The network() driver sends messages to a remote host (for example a syslog-ng server or relay) on the localintranet or internet using the RFC3164 syslog protocol (for details about the protocol, see Section 2.10.1,BSD-syslog or legacy-syslog messages (p. 19)). The network() driver supports sending messages using theUDP, TCP, RLTP ™or the encrypted TLS networking protocols.

These destinations have the following options:

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

235syslog-ng.com

network() destination options

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

236syslog-ng.com

network() destination options

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.22. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

failover-servers()list of IP addresses and fully-qualified domain namesType:0Default:

237syslog-ng.com

network() destination options

Description:Available only in syslog-ng Premium Edition version 3.2 and later. Specifies a secondary destinationserver where log messages are sent if the primary server becomes unaccessible. To list several failover servers,separate the address of the servers with comma. The time syslog-ng PE waits for the a server before switchingto the next failover server is set in the time-reopen() option. For details about how client-side failover works,see Section 8.4, Client-side failover (p. 324).

WarningThe failover servers must be accessible on the same port as the primary server.

NoteThis option is not available for the connection-less UDP protocol, because in this case the client does not detect that thedestination becomes unaccessible.

Example 7.23. Specifying failover servers for syslog() destinationsThe following example specifies two failover servers for a simple syslog() destination.

destination d_syslog_tcp{

syslog("10.100.20.40"

transport("tcp")

port(6514)

failover-servers("10.2.3.4", "myfailoverserver")

);};

The following example specifies a failover server for a network() destination that uses TLS encryption.

destination d_syslog_tls{

network("10.100.20.40"

transport("tls")

port(6514)

failover-servers("10.2.3.4", "myfailoverserver")

tls(peer-verify(required-trusted)

ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')

key-file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')

cert-file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem'))

);};

flags()no_multi_line, syslog-protocolType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message headeris formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

238syslog-ng.com

network() destination options

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

ip-tos()number (type of service)Type:0Default:

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()number (hops)Type:0Default:

Description: Specifies the Time-To-Live value of outgoing packets.

239syslog-ng.com

network() destination options

keep-alive()yes or noType:yesDefault:

Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Notethat this applies to the client (destination) side of the syslog-ng connections, server-side (source) connectionsare always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

localip()stringType:0.0.0.0Default:

Description: The IP address to bind to before connecting to target.

localport()number (port number)Type:0Default:

Description: The port number to bind to. Messages are sent from this port.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

240syslog-ng.com

network() destination options

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

241syslog-ng.com

network() destination options

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

port() or localport()number (port number)Type:UDP — 514, TCP — 601, TLS — 6514.Default:

Description: The port number to connect to.

so-broadcast()yes or noType:noDefault:

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messagesto a broadcast address. For details, see the socket(7) manual page.

so-keepalive()yes or noType:noDefault:

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

so-sndbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

242syslog-ng.com

network() destination options

spoof-interface()string (the name of the interface)Type:emptyDefault:

Description: Specifies the name of the interface that is used to send the spoofed messages. Using this optionis required only on Microsoft Windows platforms. The name of the interface is the internal device identifierused by WinPcap, which has absolutely no relation to any other possible device identifiers used by Windows.To determine the identifier of the interface, download and install the WinPcap application fromhttps://www.winpcap.org/, then execute the iflist.exe file that is installed with syslog-ng PE. A sample ofthe iflist.exe output is shown below. The first line is the identifier of the interface, but note that you needuse an ext ra backs lash charac ter. For example , i f the ident i f ie r i s\Device\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}, you must usespoof-interface("\\Device\\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}" in the configurationfile of syslog-ng PE.

\Device\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}

Description: VMware Accelerated AMD PCNet Adapter (Microsoft's Packet

Scheduler)

Loopback: no

Address Family: #2

Address Family Name: AF_INET

Address: 192.168.250.1

Netmask: 255.255.255.0

Broadcast Address: 255.255.255.255

Example 7.24. Spoofing the source address on Microsoft WindowsThe following UDP destination spoofs the address of the original client. On Microsoft Windows platforms, thespoof-interface() option must be also set.

destination d_udp {

network("10.30.0.32" port(5555)

transport("udp")

spoof-source(yes)

spoof-interface("\\Device\\NPF_{578D4231-71C1-470F-AFFE-2CB1E5BC35A8}")

);

};

WarningTo use spoofing, you must also use the spoof-source(yes) option.

spoof-source()yes or noType:noDefault:

Description: Enables source address spoofing. This means that the host running syslog-ng generates UDPpackets with the source IP address matching the original sender of the message. This is useful when you wantto perform some kind of preprocessing via syslog-ng, then forward messages to your central log management

243syslog-ng.com

network() destination options

solution with the source address of the original sender. This option only works for UDP destinations, thoughthe original message can be received by TCP as well.

WarningTo use spoofing on Microsoft Windows platforms, you must also set the spoof-interface() option as well.

When using the spoof-source option, syslog-ng PE automatically truncates long messages to 1024 bytes, regardlessof the settings of log-msg-size().

suppress()secondsType:0 (disabled)Default:

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

NoteIf a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the$MESSAGE part of the log), the structure of the header is fixed.

template-escape()yes or noType:noDefault:

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

244syslog-ng.com

network() destination options

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

tls()tls optionsType:n/aDefault:

Description: This option sets various options related to TLS encryption, for example, key/certificate files andtrusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4,TLS options (p. 356).

transport()rltp, udp, tcp, or tlsType:tcpDefault:

Description: Specifies the protocol used to send messages to the destination server.

If you use the udp transport, syslog-ng PE automatically sends multicast packets if a multicast destinationaddress is specified. The tcp transport does not support multicasting.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

245syslog-ng.com

network() destination options

7.9. Sending messages to named pipes

The pipe() driver sends messages to a named pipe like /dev/xconsole.

The pipe driver has a single required parameter, specifying the filename of the pipe to open. The filename caninclude macros. For the list of available optional parameters, see Section 7.9.1, pipe() destination options (p. 246).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

pipe(filename);

WarningStarting with syslog-ng PE 3.0.2, pipes are created automatically. In earlier versions, you had to create the pipe using themkfifo(1) command.

Example 7.25. Using the pipe() driver

destination d_pipe { pipe("/dev/xconsole"); };

7.9.1. pipe() destination options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

This driver sends messages to a named pipe like /dev/xconsole.

The pipe() destination has the following options:

flags()no_multi_line, syslog-protocolType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

246syslog-ng.com

Sending messages to named pipes

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message headeris formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

group()stringType:Use the global settingsDefault:

247syslog-ng.com

pipe() destination options

Description: Set the group of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: group().

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

248syslog-ng.com

pipe() destination options

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

owner()stringType:Use the global settingsDefault:

Description: Set the owner of the created file to the one specified. To preserve the original properties of anexisting file, use the option without specifying an attribute: owner().

pad-size()number (bytes)Type:0Default:

Description: If set, syslog-ng PE will pad output messages to the specified size (in bytes). Some operatingsystems (such as HP-UX) pad all messages to block boundary. This option can be used to specify the blocksize. (HP-UX uses 2048 bytes).

WarningHazard of data loss! If the size of the incoming message is larger than the previously set pad-size() value, syslog-ng willtruncate the message to the specified size. Therefore, all message content above that size will be lost.

249syslog-ng.com

pipe() destination options

perm()number (octal notation)Type:0600Default:

Description:The permission mask of the pipe. For octal numbers prefix the number with '0', for example: use0755 for rwxr-xr-x.

suppress()secondsType:0 (disabled)Default:

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

template-escape()yes or noType:noDefault:

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

250syslog-ng.com

pipe() destination options

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.10. Sending messages to external applications

The program() driver starts an external application or script and sends the log messages to its standard input(stdin). Usually, every message is a single line (ending with a newline character), which your script canprocess. Make sure that your script runs in a loop and keeps reading the standard input — it should not exit.(If your script exits, syslog-ng PE tries to restart it.)

The program() driver has a single required parameter, specifying a program name to start. The program isexecuted with the help of the current shell, so the command may include both file patterns and I/O redirections.For the list of available optional parameters, see Section 7.10.1, program() destination options (p. 252).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

program(command_to_run);

NoteThe syslog-ng PE application must be able to start and restart the external program, and have the necessarypermissions to do so. For example, if your host is running AppArmor, you might have to modify yourAppArmor configuration to enable syslog-ng PE to execute external applications.

■

■ The syslog-ng PE application executes program destinations through the standard system shell. If the systemshell is not bash and you experience problems with the program destination, try changing the /bin/sh linkto /bin/bash.

251syslog-ng.com

Sending messages to external applications

■ If the external program exits, the syslog-ng PE application automatically restarts it. However it is notrecommended to launch programs for single messages, because if the message rate is high, launching severalinstances of an application might overload the system, resulting in Denial of Service.

■ When the syslog-ng PE application stops, it will automatically stop the external program. To avoid restartingthe application when syslog-ng PE is only reloaded, enable the keep-alive() option in the programdestination.

■ Certain external applications buffer the log messages, which might cause unexpected latency and otherproblems. For example, if you send the log messages to an external Perl script, Perl uses a line buffer forterminal output and block buffer otherwise. You might want to disable buffering in the external application.

Example 7.26. Using the program() destination driverThe message format does not include the priority and facility values by default. To add these values, specify a templatefor the program destination, as shown in the following example. Make sure to end your template with a newline character(\n).

destination d_prog { program("/bin/script" template("<${PRI}>${DATE} ${HOST} ${MSG}\n")

); };

The following shell script writes the incoming messages into the /tmp/testlog file.

#!/bin/bash

while read line ; do

echo $line >> /tmp/testlog

done

7.10.1. program() destination options

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

This driver starts an external application or script and sends the log messages to its standard input (stdin).

The program() destination has the following options:

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

252syslog-ng.com

program() destination options

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

253syslog-ng.com

program() destination options

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.27. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

254syslog-ng.com

program() destination options

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

flags()no_multi_line, syslog-protocolType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message headeris formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

255syslog-ng.com

program() destination options

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

keep-alive()yes or noType:noDefault:

Description: Specifies whether the external program should be closed when syslog-ng PE is reloaded.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

256syslog-ng.com

program() destination options

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

suppress()secondsType:0 (disabled)Default:

257syslog-ng.com

program() destination options

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

Make sure to end your template with a newline character (\n).

template-escape()yes or noType:noDefault:

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

258syslog-ng.com

program() destination options

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.11. Generating SMTP messages (e-mail) from logs

The destination is aimed at a fully controlled local, or near-local, trusted SMTP server. The goal is to send mailto trusted recipients, through a controlled channel. It hands mails over to an SMTP server, and that is all it does,therefore the resulting solution is as reliable as sending an e-mail in general. For example, syslog-ng PE doesnot verify whether the recipient exists.

The smtp() driver sends e-mail messages triggered by log messages. The smtp() driver uses SMTP, withoutneeding external applications. You can customize the main fields of the e-mail, add extra headers, send thee-mail to multiple recipients, and so on.

The subject(), body(), and header() fields may include macros which get expanded in the e-mail. Formore information on available macros see Section 14.1.5, Macros of syslog-ng PE (p. 383).

The smtp() driver has the following required parameters: host(), port(), from(), to(), subject(), andbody(). For the list of available optional parameters, see Section 7.11.1, smtp() destination options (p. 260).

NoteWindows platforms are not supported. Mark mode is not supported at all, if SMTP receives a mark message, it will dropthe message.

The smtp() destination driver is available only in 5 F2 and later.

Declaration:

smtp(from() to() subject() body() options());

Example 7.28. Using the smtp() driverThe following example defines an smtp() destination using only the required parameters.

destination d_smtp {

smtp(

host("localhost")

port(25)

from("syslog-ng alert service" "noreply@example.com")

to("Admin #1" "admin1@example.com")

subject("[ALERT] Important log message of $LEVEL condition received from

$HOST/$PROGRAM!")

body("Hi!\nThe syslog-ng alerting service detected the following important log

message:\n $MSG\n-- \nsyslog-ng\n")

259syslog-ng.com

Generating SMTP messages (e-mail) from logs

);

};

The following example sets some optional parameters as well.

destination d_smtp {

smtp(

host("localhost")

port(25)

from("syslog-ng alert service" "noreply@example.com")

to("Admin #1" "admin1@example.com")

to("Admin #2" "admin2@example.com")

cc("Admin BOSS" "admin.boss@example.com")

bcc("Blind CC" "blindcc@example.com")

subject("[ALERT] Important log message of $LEVEL condition received from

$HOST/$PROGRAM!")

body("Hi!\nThe syslog-ng alerting service detected the following important log

message:\n $MSG\n-- \nsyslog-ng\n")

header("X-Program", "$PROGRAM")

);

};

Example 7.29. Simple e-mail alerting with the smtp() driverThe following example sends an e-mail alert if the eth0 network interface of the host is down.

filter f_linkdown {

match("eth0: link down" value("MESSAGE"));

};

destination d_alert {

smtp(

host("localhost") port(25)

from("syslog-ng alert service" "syslog@localhost")

reply-to("Admins" "root@localhost")

to("Ennekem" "me@localhost")

subject("[SYSLOG ALERT]: eth0 link down")

body("Syslog received an alert:\n$MSG")

);

};

log {

source(s_local);

filter(f_linkdown);

destination(d_alert);

};

7.11.1. smtp() destination options

The smtp() sends e-mail messages using SMTP, without needing external applications. The smtp() destinationhas the following options:

body()stringType:n/aDefault:

Description: The BODY field of the e-mail. You can also use macros in the string. Use \n to start a new line.For example:

body("syslog-ng PE received the following alert from $HOST:\n$MSG")

260syslog-ng.com

smtp() destination options

bcc()stringType:n/aDefault:

Description: The BCC recipient of the e-mail (contents of the BCC field). You can specify the e-mail address,or the name and the e-mail address. Set the bcc() option multiple times to send the e-mail to multiple recipients.For example:

bcc("admin@example.com")

or

bcc("Admin" "admin@example.com")

or

bcc("Admin" "admin@example.com")

bcc("Admin2" "admin2@example.com")

cc()stringType:n/aDefault:

Description: The CC recipient of the e-mail (contents of the CC field). You can specify the e-mail address, orthe name and the e-mail address. Set the cc() option multiple times to send the e-mail to multiple recipients.For example:

cc("admin@example.com")

or

cc("Admin" "admin@example.com")

or

cc("Admin" "admin@example.com")

cc("Admin2" "admin2@example.com")

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

261syslog-ng.com

smtp() destination options

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

262syslog-ng.com

smtp() destination options

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.30. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

263syslog-ng.com

smtp() destination options

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

from()stringType:n/aDefault:

Description: The sender of the e-mail (contents of the FROM field). You can specify the e-mail address, orthe name and the e-mail address. For example:

from("admin@example.com")

or

from("Admin" "admin@example.com")

If you specify the from() option multiple times, the last value will be used. Instead of the from() option, youcan also use sender(), which is just an alias of the from() option.

header()stringType:n/aDefault:

Description: Adds an extra header to the e-mail with the specified name and content. The first parameter setsthe name of the header, the second one its value. The value of the header can contain macros. Set the header()option multiple times to add multiple headers. For example:

header("X-Program", "$PROGRAM")

When using the header option, note the following points:

■ Do not use the header() option to set the values of headers that have dedicated options. Use it onlyto add extra headers.

264syslog-ng.com

smtp() destination options

■ If you set the same custom header multiple times, only the first will be added to the e-mail, otheroccurrences will be ignored.

■ It is not possible to set the DATE, Return-Path, Original-Recipient, Content-*, MIME-*, Resent-*,Received headers.

host()hostname or IP addressType:n/aDefault:

Description: Hostname or IP address of the SMTP server.

NoteIf you specify host="localhost", syslog-ng PE will use a socket to connect to the local SMTP server. Usehost="127.0.0.1" to force TCP communication between syslog-ng PE and the local SMTP server.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

port()numberType:25Default:

Description: The port number of the SMTP server.

reply-to()stringType:n/aDefault:

Description: Replies of the recipient will be sent to this address (contents of the REPLY-TO field). You canspecify the e-mail address, or the name and the e-mail address. Set the reply-to() option multiple times tosend the e-mail to multiple recipients. For example:

reply-to("admin@example.com")

or

reply-to("Admin" "admin@example.com")

or

reply-to("Admin" "admin@example.com")

reply-to("Admin2" "admin2@example.com")

265syslog-ng.com

smtp() destination options

retries()numberType:3Default:

Description: Number of times a message send operation is retried upon failure.

sender()stringType:n/aDefault:

Description: An alias for the from() option

subject()stringType:n/aDefault:

Description: The SUBJECT field of the e-mail. You can also use macros. For example:

subject("[SYSLOG ALERT]: Critical error message received from $HOST")

If you specify the subject() option multiple times, the last value will be used.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

266syslog-ng.com

smtp() destination options

to()stringType:localhostDefault:

Description: The recipient of the e-mail (contents of the TO field). You can specify the e-mail address, or thename and the e-mail address. Set the to() option multiple times to send the e-mail to multiple recipients. Forexample:

to("admin@example.com")

or

to("Admin" "admin@example.com")

or

to("Admin" "admin@example.com")

to("Admin2" "admin2@example.com")

7.11.2. Error handling

If the connection fails, syslog-ng PE repeatedly attempts to reconnect in intervals specified in time-reopen().

If the connection is successful, but the message is not accepted, syslog-ng PE attempts to resend the messageN times regardless of the reject being a hard or a soft reject. The number of attempts is specified with the optionretries().

NoteIf the message was sent to two addresses and only one address fails, the retry will still apply for both addresses. Theassumption is that the destination will be used with a local SMTP server acting as a relay, and the local server will handleall the edge cases.

7.12. Sending SNMP traps

The snmp() driver sends SNMP traps using the Simple Network Management Protocol version 2c or version3. Incoming log messages can be converted to SNMP traps, as the fields of the SNMP messages can becustomized using syslog-ng PE macros.

NoteThe snmp() driver is available in syslog-ng PE version 4 F1 and later.

NoteThe snmp destination driver currently supports sending SNMP traps only using the UDP transport protocol.

267syslog-ng.com

Error handling

The snmp() driver requires the host(), trap-obj(), and snmp-obj() options to be set, as well as theengine-id() option in case the SNMPv3 protocol is used. For the list of available optional parameters, seeSection 7.12.2, snmp() destination options (p. 271).

Declaration:

@module snmp

destination d_snmp {snmp(host() trap-obj() snmp-obj() ...);};

By default, syslog-ng PE does not load the snmp() module. If you want to use the snmp() destination, includethe following line in your syslog-ng PE configuration file before defining the destination to load the snmp()module. This line is needed only once, even if you use multiple SNMP destinations. For details on modules,see Section 5.6.1, Loading modules (p. 80).

@module snmp

WarningIf syslog-ng PE cannot resolve the destination hostname during startup, it will try to resolve the hostname again whenthe next message to be sent as an SNMP trap is received. However, if this name resolution fails, the trap will be dropped.

NoteThe snmp() destination driver does not generate MARK signals itself, but can receive and forward MARK signals.

Example 7.31. Using the snmp() destination driverThe following example defines an SNMP destination that uses the SNMPv2c protocol.

@module snmp

destination d_snmpv2c{

snmp(

version('v2c')

host('192.168.1.1')

trap-obj('.1.3.6.1.6.3.1.1.4.1.0', 'Objectid', '.1.3.6.1.4.1.18372.3.1.1.1.2.1')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.1.0', 'Octetstring', 'Test SNMP trap')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.2.0', 'Octetstring', 'admin')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.3.0', 'Ipaddress', '192.168.1.1')

);

};

The following example defines an SNMP destination that uses the SNMPv3 protocol and uses macros to fill the valuesof the SNMP objects.

@module snmp

destination d_snmpv3{

snmp(

version('v3')

host('192.168.1.1')

port(162)

engine-id('0xdeadbeefde')

auth-username('myusername')

auth-password('password')

enc-algorithm('AES')

enc-password('password')

trap-obj('.1.3.6.1.6.3.1.1.4.1.0', 'Objectid', '.1.3.6.1.4.1.18372.3.1.1.1.2.1')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.1', 'Octetstring', '${MESSAGE}')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.2', 'Octetstring', 'admin')

268syslog-ng.com

Sending SNMP traps

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.3', 'Ipaddress', '${SOURCEIP}')

);

};

7.12.1. Converting Cisco syslog messages to "clogMessageGenerated" SNMP traps

Starting with version 4 F1, syslog-ng PE can convert the syslog messages sent by Cisco devices to Cisco-specificSNMP traps defined by the CISCO-SYSLOG-MIB (enterprises.cisco.ciscoMgmt.ciscoCiscoMIB)is also supported (such traps are also referred to as clogMessageGenerated notifications). That way, theincoming log messages can be forwarded to devices used to process and analyze Cisco-specific SNMP traps.For this to work correctly, the following requirements must be met:

■ The snmp module of syslog-ng PE must be loaded, that is, the syslog-ng PE configuration file mustinclude the following line:

@module snmp

■ The syslog-ng Source Configuration Library (SCL) must be included in the syslog-ng PE configurationfile:

@include "scl.conf"

■ The pattern database described in Section 7.12.1.1, Parsing Cisco-specific message fields withpatterndb (p. 269) must be used to parse the incoming log messages.

To accomplish this, syslog-ng PE has to use a special pattern database to parse the Cisco-specific syslogmessages, because these messages do not comply with the standard syslog formats.

For details on the Cisco-specific SNMP trap format, see CISCO-SYSLOG-MIB on the Cisco website.

7.12.1.1. Parsing Cisco-specific message fields with patterndb

The ${PROGRAM} part of the syslog messages sent by Cisco devices contain not only the program name, butother important protocol information part as well. The ${PROGRAM} of these messages contains the Facility,Severity, and the Mnemonic (the Cisco name) of the message. The following pattern database parses thesevalues and makes them available as the .cisco.Facility, .cisco.Severity, and .cisco.MsgName,respectively. The actual log message is available as .cisco.MsgText.

<patterndb version="4" pub_date="2011-05-03">

<ruleset name="cisco snmp ruleset1" xml:id="480de478-d4a6-4a7f-bea4-0c0245d361e3">

<description>Pattern for Cisco messages having BSD timestamps, for example:

Jul 01 2010 00:32:59: %SYS-5-CONFIG_I: Configured from console by

console</description>

<url>http://balabit.com</url>

<pattern>%@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ANYSTRING:.cisco.MsgName@</pattern>

<rules>

<rule xml:id="09944c71-95eb-4bc0-8575-936931d85713" provider="balabit"

class="system">

269syslog-ng.com

Converting Cisco syslog messages to "clogMessageGenerated" SNMP traps

<patterns>

<pattern> @ANYSTRING:.cisco.MsgText@</pattern>

</patterns>

</rule>

</rules>

</ruleset>

<ruleset name="cisco snmp ruleset2" xml:id="480de478-d4a6-4a7f-bea4-0c0245d361e3">

<description>Pattern for Cisco messages having cisco-specific timestamps, for

example: 18: Jan 22 10:45:44.543: %SYS-5-CONFIG_I: Configured from console by

console</description>

<url>http://balabit.com</url>

<rules>

<rule xml:id="09944c71-95eb-4bc0-8575-936931d85714" provider="balabit"

class="system">

<patterns>

<pattern>%@ESTRING:.cisco.Facility:-@@ESTRING:.cisco.Severity:-@@ESTRING:.cisco.MsgName::@

@ANYSTRING:.cisco.MsgText@</pattern>

</patterns>

</rule>

</rules>

</ruleset>

</patterndb>

7.12.1.2. Sending clogMessageGenerated SNMP traps

To send out clogMessageGenerated SNMP traps, use the cisco_snmp() destination driver. The cisco-snmp()destination is actually a modified version of the snmp() destination driver.

NoteThe cisco-snmp() driver is actually an element of the syslog-ng Source Configuration Library (SCL), a reusableconfiguration snippet tailored to handle process accounting logs. For details on using or writing SCLs, see Section 5.7.2,Reusing configuration blocks (p. 82).

The cisco-snmp() driver has the same requirements and options as the snmp() destination driver, butautomatically fills the clogMessageGenerated-specific fields with the data received from parsing theCisco-specific syslog messages using the pattern database. For details on the , see the <INSTALLDIR>/

share/include/scl/snmp/plugin.conf file.

Declaration:

destination d_cisco_snmp {cisco-snmp(host(<hostname>));};

Example 7.32. Defining a Cisco-specific SNMP destinationThe following example defines an SNMP destination that sends out clogMessageGenerated messages using the SNMPv3protocol.

destination d_cisco_snmp {cisco-snmp(host("192.168.1.1")

version("v3")

engine-id("'0xdeadbeefde'")

auth-username('myusername')

270syslog-ng.com

Converting Cisco syslog messages to "clogMessageGenerated" SNMP traps

auth-password('password')

enc-password('password'));};

7.12.2. snmp() destination options

This driver sends SNMP traps using the SNMP v2c or v3 protocol.

The snmp() destination has the following options:

auth-algorithm()SHA|shaType:SHADefault:

Description: The authentication method to use. Lowercase values (for example, sha) can be used as well.

This option is used with the SNMPv3 protocol.

auth-password()stringType:empty stringDefault:

Description: The password used for authentication. If the auth-username() option is set but theauth-password() is empty, syslog-ng PE will try to authenticate with an empty password.

This option is used with the SNMPv3 protocol.

auth-username()stringType:

Default:

Description: The username used to authenticate on the SNMP server. If this parameter is set, syslog-ng PEwill try to authenticate on the SNMP server.

This option is used with the SNMPv3 protocol.

community()stringType:publicDefault:

Description: The community string used for SNMPv2c authentication.

This option is used with the SNMPv2c protocol.

271syslog-ng.com

snmp() destination options

enc-algorithm()AES|aesType:AESDefault:

Description: The encryption method used to encrypt the SNMP traffic. Lowercase values (for example, aes)can be used as well.

This option is used with the SNMPv3 protocol.

enc-password()stringType:

Default:

Description: The password used for the encryption. Encryption is used only if the enc-password() is notempty.

This option is used with the SNMPv3 protocol.

engine-id()number (hexadecimal number)Type:

Default:

Description: The engine ID is a hexadecimal number at least 10 digits long, starting with 0x. For example0xABABABABAB.

This option is a required parameter when using the SNMPv3 protocol.

host()hostname or IP addressType:n/aDefault:

Description: Hostname of the SNMP server.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

port()number (port number)Type:162Default:

Description: The port number to connect to.

272syslog-ng.com

snmp() destination options

snmp-obj()<oid_of_the_object>, <type_of_the_object>, <value_of_the_object>Type:

Default:

Description: The snmp-obj() option can be used to create custom SNMP trap elements. To create a trapelement, specify the OID, type, and value of the element in the snmp-obj() option. To send SNMP traps, atleast one snmp-obj() option must be defined. The snmp-obj() option requires the following parameters.Note that syslog-ng PE does not validate the values of these elements.

■ <oid_of_the_object>: The object id of the SNMP object, for example,.1.3.6.1.4.1.18372.3.1.1.1.1.1.

■ <type_of_the_object>: The type of the object specified as an ASN.1 primitive. One of: Integer,Timeticks, Octetstring, Counter32, Ipaddress, Objectid. The type names are notcase sensitive.

■ <value_of_the_object>: The value of the object as a string. The macros of syslog-ng PE canbe used to set these values, making it possible to transfer the content and other metadata from thethe syslog message to the SNMP trap. Note that if the value of an Integer, Counter32 orTimeticks object is not a number (for example, is an empty string or other not-number string),syslog-ng PE will automatically replace the value with 0. The values of other types of objects arenot validated.

Example 7.33. Defining SNMP objectsThe following are SNMP object definitions:

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.3', 'Ipaddress', '192.168.1.1')

snmp-obj('.1.3.6.1.4.1.18372.3.1.1.1.1.2', 'Octetstring', '${MESSAGE}')

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

trap-obj()<oid_of_the_object>, "Objectid", <value_of_the_object>Type:

Default:

273syslog-ng.com

snmp() destination options

Description: The trap-obj() is a specialized version of the snmp-obj() option that is used to identify theSNMP trap object. The type of the trap object is always Objectid. The <oid_of_the_object> and the<value_of_the_object> parameters are identical to the respective parameters of the snmp-obj() option.For details on these parameters, see Section snmp-obj() (p. 273).

NoteUsing the trap-obj() object is equivalent to using the snmp-obj() with the Objectid type.

version()v2c|v3Type:v2cDefault:

Description: Specifies which version of the SNMP protocol to use.

NoteThe syslog-ng PE application will accept any valid option for the snmp() destination, but will only use the ones relevantto the selected protocol version, any other option will be ignored. For example, if the version("v2c")

engine-id("0xABABABABAB") community("mycommunity") options are set, syslog-ng PE will accept every option,but process only the community() option, because engine-id() applies only to SNMPv3.

7.13. Storing messages in an SQL database

The sql() driver sends messages into an SQL database. Currently the Microsoft SQL (MSSQL), MySQL,Oracle, PostgreSQL, and SQLite databases are supported.

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

sql(database_type host_parameters database_parameters [options]);

The sql() driver has the following required parameters: type(), database(), table(), columns(), andvalues().

274syslog-ng.com

Storing messages in an SQL database

WarningThe syslog-ng application requires read and write access to the SQL table, otherwise it cannot verify that the destinationtable exists.

Currently the syslog-ng application has default schemas for the different databases and uses these defaults if the databaseschema (for example columns and column types) is not defined in the configuration file. However, these schemas willbe deprecated and specifying the exact database schema will be required in later versions of syslog-ng.

NoteIn addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed.These packages are automatically installed by the binary syslog-ng installer.

The sql() driver is currently not available for every platform that is supported by syslog-ng. For a list of platforms thatsupport the sql() driver, visit this link.

The table and value parameters can include macros to create tables and columns dynamically (for details,see Section 14.1.5, Macros of syslog-ng PE (p. 383)).

WarningWhen using macros in table names, note that some databases limit the maximum allowed length of table names. Consultthe documentation of the database for details.

Inserting the records into the database is performed by a separate thread. The syslog-ng application automaticallyperforms the escaping required to insert the messages into the database.

Example 7.34. Using the sql() driverThe following example sends the log messages into a PostgreSQL database running on the logserver host. The messagesare inserted into the logs database, the name of the table includes the exact date and the name of the host sending themessages. The syslog-ng application automatically creates the required tables and columns, if the user account used toconnect to the database has the required privileges.

destination d_sql {

sql(type(pgsql)

host("logserver") username("syslog-ng") password("password")

database("logs")

table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")

columns("datetime", "host", "program", "pid", "message")

values("{$R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid", "message"));

};

The following example specifies the type of the database columns as well:

destination d_sql {

sql(type(pgsql)

host("logserver") username("syslog-ng") password("password")

database("logs")

table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")

columns("datetime varchar(16)", "host varchar(32)", "program varchar(20)", "pid

varchar(8)", "message varchar(200)")

values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid", "message"));

};

275syslog-ng.com

Storing messages in an SQL database

7.13.1. Using the sql() driver with an Oracle database

The Oracle sql destination has some special aspects that are important to note.

■ The hostname of the database server is set in the tnsnames.ora file, not in the host parameter ofthe sql() destination.

If the tnsnames.ora file is not located in the /etc directory (or in the /var/opt/oracle directory onSolaris), set the following Oracle-related environment variables, so syslog-ng PE will find the file:ORACLE_BASE, ORACLE_HOME, and ORACLE_SID. For details, see the documentation of the OracleInstant Client.

■ You cannot use the same database() settings in more than one destination, because the database()option of the SQL driver is just a reference to the connection string of the tnsnames.ora file. Toovercome this problem, you can duplicate the connections in the tnsnames.ora file under a differentname, and use a different table in each Oracle destination in syslog-ng PE.

■ As certain database versions limit the maximum length of table names, macros in the table namesshould be used with care.

■ In the current version of syslog-ng PE, the types of database columns must be explicitly set for theOracle destination. The column used to store the text part of the syslog messages should be able tostore messages as long as the longest message permitted by syslog-ng, therefore it is usuallyrecommended to use the varchar2 or clob column type. (The maximum length of the messagescan be set using the log-msg-size() option.) For details, see the following example.

■ The Oracle Instant Client used by syslog-ng PE supports only the following character sets:

• Single-byte character sets: US7ASCII, WE8DEC, WE8MSWIN1252, and WE8ISO8859P1

• Unicode character sets: UTF8, AL16UTF16, and AL32UTF8

Example 7.35. Using the sql() driver with an Oracle databaseThe following example sends the log messages into an Oracle database running on the logserver host, which must beset in the /etc/tnsnames.ora file. The messages are inserted into the LOGS database, the name of the table includesthe exact date when the messages were sent. The syslog-ng application automatically creates the required tables andcolumns, if the user account used to connect to the database has the required privileges.

destination d_sql {

sql(type(oracle)

username("syslog-ng") password("password")

database("LOGS")

table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")

columns("datetime varchar(16)", "host varchar(32)", "program varchar(32)", "pid

varchar(8)", "message varchar2")

values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid", "message"));

};

The Oracle Instant Client retrieves the address of the database server from the /etc/tnsnames.ora file. Edit or createthis file as needed for your configuration. A sample is provided below.

LOGS =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)

(HOST = logserver)

(PORT = 1521))

)

276syslog-ng.com

Using the sql() driver with an Oracle database

(CONNECT_DATA =

(SERVICE_NAME = EXAMPLE.SERVICE)

)

)

7.13.2. Using the sql() driver with a Microsoft SQL database

The mssql database driver can access Microsoft SQL (MSSQL) destinations. This driver has some specialaspects that are important to note.

■ The date format used by the MSSQL database must be explicitly set in the /etc/locales.conffile of the syslog-ng server. For details, see the following example.

■ As certain database versions limit the maximum length of table names, macros in the table namesshould be used with care.

■ In the current version of syslog-ng PE, the types of database columns must be explicitly set for theMSSQL destination.

WarningThe following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.

■ The column used to store the text part of the syslog messages should be able to store messages aslong as the longest message permitted by syslog-ng. The varchar column type can store maximum4096 bytes-long messages. The maximum length of the messages can be set using thelog-msg-size() option. For details, see the following example.

■ Remote access for SQL users must be explicitly enabled on the Microsoft Windows host runningthe Microsoft SQL Server. For details, see Procedure 3.13, Configuring Microsoft SQL Server toaccept logs from syslog-ng (p. 62).

Example 7.36. Using the sql() driver with an MSSQL databaseThe following example sends the log messages into an MSSQL database running on the logserver host. The messagesare inserted into the syslogng database, the name of the table includes the exact date when the messages were sent. Thesyslog-ng application automatically creates the required tables and columns, if the user account used to connect to thedatabase has the required privileges.

destination d_mssql {

sql(type(mssql) host("logserver") port("1433")

username("syslogng") password("syslogng") database("syslogng")

table("msgs_${R_YEAR}${R_MONTH}${R_DAY}")columns("datetime varchar(16)", "host

varchar(32)",

"program varchar(32)", "pid varchar(8)", "message varchar(4096)")

values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid"));

};

The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of the syslog-ngserver. Edit or create this file as needed for your configuration. A sample is provided below.

[default]

date = "%Y-%m-%d %H:%M:%S"

277syslog-ng.com

Using the sql() driver with a Microsoft SQL database

7.13.3. The way syslog-ng interacts with the database

Used SQL operations by syslog-ng.

Create table:

■ If the given table does not exist, syslog-ng tries to create it with the given column types.

■ The syslog-ng PE application automatically creates the required tables and columns, if the useraccount used to connect to the database has the required privileges.

■ If syslog-ng cannot create or alter a table, it tries to do it again when it reaches the nexttime-reopen().

Alter table:

■ If the table structure is different from given structure in an existing table, syslog-ng tries to addcolumns in this table but never will delete or modify an existing column.

■ If syslog-ng PE cannot create or alter a table, it tries to do it again when reach the nexttime-reopen().

■ The syslog-ng PE application requires read and write access to the SQL table, otherwise it cannotverify that the destination table exists.

Insert table:

■ Insert new records in a table.

■ Inserting the records into the database is performed by a separate thread.

■ The syslog-ng PE application automatically performs the escaping required to insert the messagesinto the database.

■ If insert returns with error, syslog-ng tries to insert the message +two times by default, then dropsit. Retrying time is the value of time-reopen().

Encoding.

The syslog-ng PE application uses UTF-8 by default when writes logs into database.

Start/stop and reload.

Start:

■ The syslog-ng PE application will connect to database automatically after starting regardless existingincoming messages.

Stop:

■ The syslog-ng PE application will close the connection to database before shutting down.

Possibility of losing logs:

■ The syslog-ng PE application cannot lose logs during shutting down if disk buffer was given and itis not full yet.

278syslog-ng.com

The way syslog-ng interacts with the database

■ The syslog-ng PE application cannot lose logs during shutting down if disk buffer was not given.

Reload:

■ The syslog-ng PE application will close the connection to database if it received SIGHUP signal(reload).

■ It will reconnect to the database when it tries to send a new message to this database again.

Macros:

The value of ${SEQNUM} macro will be overrided by sql driver regardless of local or relayed incomingmessage.

It will be grown continously.

7.13.4. MySQL-specific interaction methods

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for exampleMYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

7.13.5. MsSQL-specific interaction methods

In SQL Server 2005 this restriction is lifted - kind of. The total length of all key columns in an index cannotexceed 900 bytes.

If you are using null() in your configuration, be sure that the columns allow NULL to insert. Give the columnas the following example: "datetime varchar(16) NULL".

The date format used by the MSSQL database must be explicitly set in the /etc/locales.conf file of thesyslog-ng server. [default] date = "%Y-%m-%d %H:%M:%S".

7.13.6. Supported SQL destinations by platform

OracleSQLitePgSQLMySQLMSSQLPlatform \ Database-✔✔✔✔AIX

✔✔✔✔✔Linux (linux_glibc236):CentOS 5

✔✔✔✔✔Linux (linux_glibc236):CentOS 6

✔✔✔✔✔Linux (linux_glibc236):CentOS 7

✔✔✔✔✔Linux (linux_glibc236):Debian 7

✔✔✔✔✔Linux (linux_glibc236):openSUSE 11

✔✔✔✔✔Linux (linux_glibc236): OracleLinux 5

279syslog-ng.com

MySQL-specific interaction methods

OracleSQLitePgSQLMySQLMSSQLPlatform \ Database✔✔✔✔✔Linux (linux_glibc236): Oracle

Linux 6

✔✔✔✔✔Linux (linux_glibc236): OracleLinux 7

✔✔✔✔✔Linux (linux_glibc236): SLES11

✔✔✔✔✔Linux (linux_glibc236): RedHat EL 5

✔✔✔✔✔Linux (linux_glibc236): RedHat EL 6

✔✔✔✔✔Linux (linux_glibc236): RedHat EL 7

✔✔✔✔✔Linux (linux_glibc236):Ubuntu 12.04

✔✔✔✔✔Linux (linux_glibc236):Ubuntu 14.04

-✔✔✔✔FreeBSD 8

-✔✔✔✔FreeBSD 9

-✔✔✔✔FreeBSD 10

✔✔-✔✔HP-UX 11v2_on IA64

✔✔✔✔✔Solaris 10 on SPARC andSPARC64

✔✔✔✔✔Solaris 11 on SPARC andSPARC64

✔✔-✔✔Solaris 10 on x86_64

✔✔-✔✔Solaris 11 on x86_64Table 7.2. Supported SQL destinations by platform

7.13.7. sql() destination options

NoteIn order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ngPremium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode (p. 11).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

280syslog-ng.com

sql() destination options

This driver sends messages into an SQL database. The sql() destination has the following options:

columns()string listType:"date", "facility", "level", "host", "program", "pid", "message"Default:

Description:Name of the columns storing the data in fieldname [dbtype] format. The [dbtype] parameteris optional, and specifies the type of the field. By default, syslog-ng PE creates text columns. Note that notevery database engine can index text fields.

WarningThe following column types cannot be used in MSSQL destinations: nchar, nvarchar, ntext, and xml.

database()stringType:logsDefault:

Description: Name of the database that stores the logs. Macros cannot be used in database name. Also, whenusing an Oracle database, you cannot use the same database() settings in more than one destination.

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

281syslog-ng.com

sql() destination options

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

282syslog-ng.com

sql() destination options

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.37. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

flags()list of flagsType:empty stringDefault:

Description: Flags related to the sql() destination.

283syslog-ng.com

sql() destination options

■ dont-create-tables: Enable this flag to prevent syslog-ng PE from creating non-existing databasetables automatically. The syslog-ng PE application typically has to create tables if you use macrosin the table names. Available in syslog-ng PE version 4.0 and later.

■ explicit-commits: By default, syslog-ng PE commits every log message to the target databaseindividually. When the explicit-commits option is enabled, messages are committed in batches.This improves the performance, but results in some latency, as the messages are not immediatelysent to the database. The size and frequency of batched commits can be set using the flush-lines()parameter. The explicit-commits option is available in syslog-ng PE version 4.0 and later.

Example 7.38. Setting flags for SQL destinationsThe following example sets the dont-create-tables and explicit-commits flags for an sql() destination.

flags(dont-create-tables,explicit-commits)

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing the

284syslog-ng.com

sql() destination options

fractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

host()hostname or IP addressType:n/aDefault:

Description: Hostname of the database server. Note that Oracle destinations do not use this parameter, butretrieve the hostname from the /etc/tnsnames.ora file.

NoteIf you specify host="localhost", syslog-ng will use a socket to connect to the local database server. Usehost="127.0.0.1" to force TCP communication between syslog-ng and the local database server.

To specify the socket to use, set and export the MYSQL_UNIX_PORT environment variable, for exampleMYSQL_UNIX_PORT=/var/lib/mysql/mysql.sock; export MYSQL_UNIX_PORT.

indexes()string listType:"date", "facility", "host", "program"Default:

Description: The list of columns that are indexed by the database to speed up searching. To disable indexingfor the destination, include the empty indexes() parameter in the destination, simply omitting the indexesparameter will cause syslog-ng to request indexing on the default columns.

The syslog-ng PE application will create the name of indexes automaticaly with the following method:

■ In case of MsSQL, PostgreSQL, MySQL or SQLite or (Oracle but tablename < 30 characters):{table}_{column}_idx.

■ In case of Oracle and tablename > 30 characters: md5sum of {table}_{column}-1 and the firstcharacter will be replaced by "i" character and the md5sum will be truncated to 30 characters.

local-time-zone()name of the timezone, or the timezone offsetType:The local timezone.Default:

Description: Sets the timezone used when expanding filename and tablename templates.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

285syslog-ng.com

sql() destination options

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

null()stringType:

Default:

Description: If the content of a column matches the string specified in the null() parameter, the contents ofthe column will be replaced with an SQL NULL value. If unset (by default), the option does not match on anystring. For details, see the Example 7.39, Using SQL NULL values (p. 286).

Example 7.39. Using SQL NULL valuesThe null() parameter of the SQL driver can be used to replace the contents of a column with a special SQL NULLvalue. To replace every column that contains an empty string with NULL, use the null("") option, for example

destination d_sql {

sql(type(pgsql)

host("logserver") username("syslog-ng") password("password")

database("logs")

table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")

columns("datetime", "host", "program", "pid", "message")

values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid", "message")

null(""));

};

To replace only a specific column (for example pid) if it is empty, assign a default value to the column, and use thisdefault value in the null() parameter:

destination d_sql {

sql(type(pgsql)

host("logserver") username("syslog-ng") password("password")

database("logs")

table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")

columns("datetime", "host", "program", "pid", "message")

values("${R_DATE}", "${HOST}", "${PROGRAM}", "${PID:-@@NULL@@}", "${MSGONLY}")

indexes("datetime", "host", "program", "pid", "message")

null("@@NULL@@"));

};

Ensure that the default value you use does not appear in the actual log messages, because other occurrences of this stringwill be replaced with NULL as well.

password()stringType:n/aDefault:

Description: Password of the database user.

286syslog-ng.com

sql() destination options

port()number (port number)Type:1433 TCP for MSSQL, 3306 TCP for MySQL, 1521 for Oracle, and 5432 TCP for PostgreSQLDefault:

Description: The port number to connect to.

retriesnumber (insertion attempts)Type:3Default:

Description: The number of insertion attempts. If syslog-ng PE could not insert a message into the database,it will repeat the attempt until the number of attempts reaches retries, then drops the connection to thedatabase. For example, syslog-ng PE will try to insert a message maximum three times by default (once forfirst insertion and twice if the first insertion was failed).

session-statements()comma-separated list of SQL statementsType:empty stringDefault:

Description: Specifies one or more SQL-like statement which is executed after syslog-ng PE has successfullyconnected to the database. For example:

session-statements("SET COLLATION_CONNECTION='utf8_general_ci'")

WarningThe syslog-ng PE application does not validate or limit the contents of customized queries. Consequently, queries performedwith a user with write-access can potentially modify or even harm the database. Use customized queries with care, andonly for your own responsibility.

table()stringType:messagesDefault:

Description: Name of the database table to use (can include macros). When using macros, note that somedatabases limit the length of table names.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

287syslog-ng.com

sql() destination options

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

type()mssql, mysql, oracle, pgsql, or sqlite3Type:mysqlDefault:

Description: Specifies the type of the database, that is, the DBI database driver to use. Use the mssql optionto send logs to an MSSQL database. For details, see the examples of the databases on the following sections.

username()stringType:n/aDefault:

Description: Name of the database user.

values()string listType:"${R_YEAR}-${R_MONTH}-${R_DAY}, ${R_HOUR}:${R_MIN}:${R_SEC}", "${FACILITY}","${LEVEL}", "${HOST}", "${PROGRAM}", "${PID}", "${MSGONLY}"

Default:

Description: The parts of the message to store in the fields specified in the columns() parameter.

It is possible to give a special value calling: default (without quotation marks).It means that the value will beused that is the default of the column type of this value.

Example 7.40. Value: default

columns("date datetime", "host varchar(32)", "row_id serial")

values("${R_DATE}", "${HOST}", default)

7.14. Sending messages to a remote log server using the IETF-syslog protocol

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the localintranet or internet using the new standard syslog protocol developed by IETF (for details about the new protocol,see Section 2.10.2, IETF-syslog messages (p. 21)). The protocol supports sending messages using the UDP,TCP, or the encrypted TLS networking protocols.

The required arguments of the driver are the address of the destination host (where messages should be sent).The transport method (networking protocol) is optional, syslog-ng uses the TCP protocol by default. For thelist of available optional parameters, see Section 7.14.1, syslog() destination options (p. 289).

Declaration:

syslog(host transport [options]);

288syslog-ng.com

Sending messages to a remote log server using the IETF-syslog protocol

NoteNote that the syslog destination driver has required parameters, while the source driver defaults to the local bind address,and every parameter is optional.

The udp transport method automatically sends multicast packets if a multicast destination address is specified.The tcp and tls methods do not support multicasting.

NoteThe default ports for the different transport protocols are as follows: UDP — 514, TCP — 601, TLS — 6514.

Example 7.41. Using the syslog() driver

destination d_tcp { syslog("10.1.2.3" transport("tcp") port(1999) localport(999)); };

If name resolution is configured, the hostname of the target server can be used as well.

destination d_tcp { syslog("target_host" transport("tcp") port(1999) localport(999)); };

Send the log messages using TLS encryption and use mutual authentication. For details on the encryption and authenticationoptions, see Section 10.4, TLS options (p. 356).

destination d_syslog_tls{

syslog("10.100.20.40"

transport("tls")

port(6514)

tls(peer-verify(required-trusted)

ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')

key-file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')

cert-file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem')

)

);

};

NoteIf a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the$MESSAGE part of the log), the structure of the header is fixed.

7.14.1. syslog() destination options

The syslog() driver sends messages to a remote host (for example a syslog-ng server or relay) on the localintranet or internet using the RFC5424 syslog protocol developed by IETF (for details about the protocol, seeSection 2.10.2, IETF-syslog messages (p. 21)). The protocol supports sending messages using the RLTP™,UDP, TCP, or the encrypted TLS networking protocols.

These destinations have the following options:

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

289syslog-ng.com

syslog() destination options

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

290syslog-ng.com

syslog() destination options

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.42. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

291syslog-ng.com

syslog() destination options

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

failover-servers()list of IP addresses and fully-qualified domain namesType:0Default:

Description:Available only in syslog-ng Premium Edition version 3.2 and later. Specifies a secondary destinationserver where log messages are sent if the primary server becomes unaccessible. To list several failover servers,separate the address of the servers with comma. The time syslog-ng PE waits for the a server before switchingto the next failover server is set in the time-reopen() option. For details about how client-side failover works,see Section 8.4, Client-side failover (p. 324).

WarningThe failover servers must be accessible on the same port as the primary server.

NoteThis option is not available for the connection-less UDP protocol, because in this case the client does not detect that thedestination becomes unaccessible.

Example 7.43. Specifying failover servers for syslog() destinationsThe following example specifies two failover servers for a simple syslog() destination.

destination d_syslog_tcp{

syslog("10.100.20.40"

transport("tcp")

port(6514)

failover-servers("10.2.3.4", "myfailoverserver")

);};

The following example specifies a failover server for a network() destination that uses TLS encryption.

destination d_syslog_tls{

network("10.100.20.40"

transport("tls")

port(6514)

failover-servers("10.2.3.4", "myfailoverserver")

tls(peer-verify(required-trusted)

ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')

key-file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')

cert-file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem'))

);};

flags()no_multi_line, syslog-protocolType:empty setDefault:

292syslog-ng.com

syslog() destination options

Description: Flags influence the behavior of the destination driver.

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message headeris formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

293syslog-ng.com

syslog() destination options

ip-tos()number (type of service)Type:0Default:

Description: Specifies the Type-of-Service value of outgoing packets.

ip-ttl()number (hops)Type:0Default:

Description: Specifies the Time-To-Live value of outgoing packets.

keep-alive()yes or noType:yesDefault:

Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Notethat this applies to the client (destination) side of the syslog-ng connections, server-side (source) connectionsare always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

localip()stringType:0.0.0.0Default:

Description: The IP address to bind to before connecting to target.

localport()number (port number)Type:0Default:

Description: The port number to bind to. Messages are sent from this port.

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

mark-freq()number (seconds)Accepted values:1200Default:

294syslog-ng.com

syslog() destination options

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

295syslog-ng.com

syslog() destination options

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

port() or localport()number (port number)Type:UDP — 514, TCP — 601, TLS — 6514.Default:

Description: The port number to connect to.

so-broadcast()yes or noType:noDefault:

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messagesto a broadcast address. For details, see the socket(7) manual page.

so-keepalive()yes or noType:noDefault:

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

296syslog-ng.com

syslog() destination options

so-sndbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

spoof-interface()string (the name of the interface)Type:emptyDefault:

Description: Specifies the name of the interface that is used to send the spoofed messages. Using this optionis required only on Microsoft Windows platforms. The name of the interface is the internal device identifierused by WinPcap, which has absolutely no relation to any other possible device identifiers used by Windows.To determine the identifier of the interface, download and install the WinPcap application fromhttps://www.winpcap.org/, then execute the iflist.exe file that is installed with syslog-ng PE. A sample ofthe iflist.exe output is shown below. The first line is the identifier of the interface, but note that you needuse an ext ra backs lash charac ter. For example , i f the ident i f ie r i s\Device\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}, you must usespoof-interface("\\Device\\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}" in the configurationfile of syslog-ng PE.

\Device\NPF_{ADE8ECA1-183A-43A2-AF01-9C81ABB4A4C1}

Description: VMware Accelerated AMD PCNet Adapter (Microsoft's Packet

Scheduler)

Loopback: no

Address Family: #2

Address Family Name: AF_INET

Address: 192.168.250.1

Netmask: 255.255.255.0

Broadcast Address: 255.255.255.255

Example 7.44. Spoofing the source address on Microsoft WindowsThe following UDP destination spoofs the address of the original client. On Microsoft Windows platforms, thespoof-interface() option must be also set.

destination d_udp {

network("10.30.0.32" port(5555)

transport("udp")

spoof-source(yes)

spoof-interface("\\Device\\NPF_{578D4231-71C1-470F-AFFE-2CB1E5BC35A8}")

);

};

WarningTo use spoofing, you must also use the spoof-source(yes) option.

297syslog-ng.com

syslog() destination options

spoof-source()yes or noType:noDefault:

Description: Enables source address spoofing. This means that the host running syslog-ng generates UDPpackets with the source IP address matching the original sender of the message. This is useful when you wantto perform some kind of preprocessing via syslog-ng, then forward messages to your central log managementsolution with the source address of the original sender. This option only works for UDP destinations, thoughthe original message can be received by TCP as well.

WarningTo use spoofing on Microsoft Windows platforms, you must also set the spoof-interface() option as well.

When using the spoof-source option, syslog-ng PE automatically truncates long messages to 1024 bytes, regardlessof the settings of log-msg-size().

suppress()secondsType:0 (disabled)Default:

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

NoteIf a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the$MESSAGE part of the log), the structure of the header is fixed.

template-escape()yes or noType:noDefault:

298syslog-ng.com

syslog() destination options

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

tls()tls optionsType:n/aDefault:

Description: This option sets various options related to TLS encryption, for example, key/certificate files andtrusted CA locations. TLS can be used only with tcp-based transport protocols. For details, see Section 10.4,TLS options (p. 356).

transport()rltp, udp, tcp, or tlsType:tcpDefault:

Description: Specifies the protocol used to send messages to the destination server.

If you use the udp transport, syslog-ng PE automatically sends multicast packets if a multicast destinationaddress is specified. The tcp transport does not support multicasting.

299syslog-ng.com

syslog() destination options

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.15. Sending messages to a remote log server using the legacy BSD-syslog protocol(tcp(), udp() drivers)

NoteThe tcp(), tcp6(), udp(), and udp6() drivers are obsolete. Use the network() source and the network() destinationinstead. For details, see Section 6.5, Collecting messages using the RFC3164 protocol (network() driver) (p. 101) andSection 7.8, Sending messages to a remote log server using the RFC3164 protocol (network() driver) (p. 234), respectively.

To convert your existing tcp(), tcp6(), udp(), udp6() source drivers to use the network() driver, seeProcedure 7.15.1.1, Change an old destination driver to the network() driver (p. 300).

The tcp(), tcp6(), udp(), and udp6() drivers send messages to another host (for example a syslog-ngserver or relay) on the local intranet or internet using the UDP or TCP protocol. The tcp6() and udp6()

drivers use the IPv6 network protocol.

7.15.1. tcp(), tcp6(), udp(), and udp6() destination options

NoteThe tcp(), tcp6(), udp(), and udp6() drivers are obsolete. Use the network() source and the network() destinationinstead. For details, see Section 6.5, Collecting messages using the RFC3164 protocol (network() driver) (p. 101) andSection 7.8, Sending messages to a remote log server using the RFC3164 protocol (network() driver) (p. 234), respectively.

To convert your existing tcp(), tcp6(), udp(), udp6() source drivers to use the network() driver, seeProcedure 7.15.1.1, Change an old destination driver to the network() driver (p. 300).

7.15.1.1. Procedure – Change an old destination driver to the network() driver

To replace your existing tcp(), tcp6(), udp(), udp6() destinations with a network() destination, completethe following steps.

Step 1. Replace the driver with network. For example, replace udp( with network(

Step 2. Set the transport protocol.

■ If you used TLS-encryption, add the transport("tls") option, then continue with thenext step.

■ If you used the tcp or tcp6 driver, add the transport("tcp") option.

■ If you used the udp or udp driver, add the transport("udp") option.

Step 3. If you use IPv6 (that is, the udp6 or tcp6 driver), add the ip-protocol("6") option.

300syslog-ng.com

Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers)

Step 4. If you did not specify the port used in the old driver, check Section 7.8.1, network() destinationoptions (p. 235) and verify that your clients send the messages to the default port of the transport protocolyou use. Otherwise, set the appropriate port number in your source using the port() option.

Step 5. All other options are identical. Test your configuration with the syslog-ng --syntax-only

command.The following configuration shows a simple tcp destination.

destination d_old_tcp {

tcp(

"127.0.0.1" port(1999)

tls(

peer-verify("required-trusted")

key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")

cert-file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt')

)

);

};

When replaced with the network() driver, it looks like this.

destination d_new_network_tcp {

network(

"127.0.0.1"

port(1999)

transport("tls")

tls(

peer-verify("required-trusted")

key-file("/opt/syslog-ng/etc/syslog-ng/syslog-ng.key")

cert-file('/opt/syslog-ng/etc/syslog-ng/syslog-ng.crt')

)

);

};

7.16. Sending messages to UNIX domain sockets

The unix-stream() and unix-dgram() drivers send messages to a UNIX domain socket in eitherSOCK_STREAM or SOCK_DGRAM mode.

Both drivers have a single required argument specifying the name of the socket to connect to. For the list ofavailable optional parameters, see Section 7.16.1, unix-stream() and unix-dgram() destination options (p. 302).

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

Declaration:

301syslog-ng.com

Sending messages to UNIX domain sockets

unix-stream(filename [options]);

unix-dgram(filename [options]);

Example 7.45. Using the unix-stream() driver

destination d_unix_stream { unix-stream("/var/run/logs"); };

7.16.1. unix-stream() and unix-dgram() destination options

These drivers send messages to a unix socket in either SOCK_STREAM or SOCK_DGRAM mode. Theunix-stream() and unix-dgram() destinations have the following options.

WarningThis feature is currently not available when running the syslog-ng PE application on Microsoft Windows platforms. Fora complete list of limitations, see Section 1.6.1, Limitations on Microsoft Windows platforms (p. 5).

disk-buffer()Description: This option enables putting outgoing messages into the disk buffer of the destination to avoidmessage loss in case of a system failure on the destination side. It has the following options:

reliable()yes|noType:noDefault:Description: If set to yes, syslog-ng PE cannot lose logs in case ofreload/restart, unreachable destination or syslog-ng PE crash. This solutionprovides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. If set to no, thenormal disk-buffer will be used. This provides a faster, but less reliabledisk-buffer option.

WarningHazard of data loss! If you change the value of reliable() option whenthere are messages in the disk-buffer, the messages stored in thedisk-buffer will be lost.

302syslog-ng.com

unix-stream() and unix-dgram() destination options

dir()stringType:N/ADefault:Description: Defines the folder where the disk-buffer files are stored. Thisoption has priority over --qdisk-dir=.

WarningWhen creating a new dir() option for a disk buffer, or modifying anexisting one, make sure you delete the persist file, or at least remove therelevant persist-entry.

syslog-ng PE creates disk-buffer files based on the path recorded in thepersist file. Therefore, if the persist file or the relevant entry is not deletedafter modifying the dir() option, then following a restart, syslog-ng PEwill look for or create disk-buffer files in their old location. To ensurethat syslog-ng PE uses the new dir() setting, the persist file must notcontain any information about the destinations which the disk-buffer filein question belongs to.

disk-buf-size()number (bytes)Type:

Default:Description: This is a required option. The maximum size of the disk-bufferin bytes. The minimum value is 1048576 bytes. If you set a smaller value,the minimum value will be used automatically. It replaces the oldlog-disk-fifo-size() option.

mem-buf-length()number (messages)Type:10000Default:Description: Use this option if the option reliable() is set to no. Thisoption contains the number of messages stored in overflow queue. It replacesthe old log-fifo-size() option. It inherits the value of the globallog-fifo-size() option if provided. If it is not provided, the defaultvalue is 10000 messages. Note that this option will be ignored if the optionreliable() is set to yes.

303syslog-ng.com

unix-stream() and unix-dgram() destination options

mem-buf-size()number (bytes)Type:163840000Default:Description: Use this option if the option reliable() is set to yes. Thisoption contains the size of the messages in bytes that is used in the memorypart of the disk buffer. It replaces the old log-fifo-size() option. It doesnot inherit the value of the global log-fifo-size() option, even if it isprovided. Note that this option will be ignored if the option reliable()

is set to no.

quot-size()number (messages)Type:64Default:Description: The number of messages stored in the output buffer of thedestination.

Options reliable() and disk-buf-size() are required options.

Example 7.46. Examples for using disk-buffer()In the following case reliable disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

dir("/tmp/disk-buffer")

)

);

};

In the following case normal disk-buffer() is used.

destination d_demo {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

dir("/tmp/disk-buffer")

)

);

};

flags()no_multi_line, syslog-protocolType:empty setDefault:

Description: Flags influence the behavior of the destination driver.

304syslog-ng.com

unix-stream() and unix-dgram() destination options

■ no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire messageis converted to a single line.

■ syslog-protocol: The syslog-protocol flag instructs the driver to format the messages accordingto the new IETF syslog protocol standard (RFC5424), but without the frame header. If this flag isenabled, macros used for the message have effect only for the text of the message, the message headeris formatted to the new standard. Note that this flag is not needed for the syslog driver, and thatthe syslog driver automatically adds the frame header to the messages.

flush-lines()number (messages)Type:Use global setting.Default:

Description: Specifies how many lines are sent to a destination at a time. The syslog-ng PE application waitsfor this number of lines to accumulate and sends them off in a single batch. Setting this number high increasesthroughput as fully filled frames are sent to the destination, but also increases message latency.

For optimal performance when sending messages to an syslog-ng PE server, make sure that the flush-lines()is smaller than the window size set using the log-iw-size() option in the source of your server.

flush-timeout() (OBSOLETE)time in millisecondsType:Use global setting.Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

305syslog-ng.com

unix-stream() and unix-dgram() destination options

log-fifo-size()number (messages)Type:Use global setting.Default:

Description: The number of messages that the output queue can store.

keep-alive()yes or noType:yesDefault:

Description: Specifies whether connections to destinations should be closed when syslog-ng is reloaded. Notethat this applies to the client (destination) side of the syslog-ng connections, server-side (source) connectionsare always reopened after receiving a HUP signal unless the keep-alive option is enabled for the source.

so-broadcast()yes or noType:noDefault:

Description:This option controls the SO_BROADCAST socket option required to make syslog-ng send messagesto a broadcast address. For details, see the socket(7) manual page.

so-keepalive()yes or noType:noDefault:

Description: Enables keep-alive messages, keeping the socket open. This only effects TCP and UNIX-streamsockets. For details, see the socket(7) manual page.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

306syslog-ng.com

unix-stream() and unix-dgram() destination options

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

307syslog-ng.com

unix-stream() and unix-dgram() destination options

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

so-rcvbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket receive buffer in bytes. For details, see the socket(7) manualpage.

so-sndbuf()number (bytes)Type:0Default:

Description: Specifies the size of the socket send buffer in bytes. For details, see the socket(7) manual page.

suppress()secondsType:0 (disabled)Default:

Description: If several identical log messages would be sent to the destination without any other messagesbetween the identical messages (for example, an application repeated an error message ten times), syslog-ngcan suppress the repeated messages and send the message only once, followed by the Last message repeated

n times. message. The parameter of this option specifies the number of seconds syslog-ng waits for identicalmessages.

template()stringType:A format conforming to the default logfile format.Default:

Description: Specifies a template defining the logformat to be used in the destination. Macros are describedin Section 14.1.5, Macros of syslog-ng PE (p. 383). Please note that for network destinations it might not beappropriate to change the template as it changes the on-wire format of the syslog protocol which might not betolerated by stock syslog receivers (like syslogd or syslog-ng itself). For network destinations make sure thereceiver can cope with the custom format defined.

template-escape()yes or noType:noDefault:

308syslog-ng.com

unix-stream() and unix-dgram() destination options

Description: Turns on escaping for the ', ", and backspace characters in templated output files. This is usefulfor generating SQL statements and quoting string contents so that parts of the log message are not interpretedas commands to the SQL server.

throttle()number (messages per second)Type:0Default:

Description: Sets the maximum number of messages sent to the destination per second. Use thisoutput-rate-limiting functionality only when using disk-buffer as well to avoid the risk of losing messages.Specifying 0 or a lower value sets the output limit to unlimited.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

ts-format()rfc3164, bsd, rfc3339, isoType:Use the global option (which defaults to rfc3164)Default:

Description:Override the global timestamp format (set in the global ts-format() parameter) for the specificdestination. For details, see Section 2.5.2, A note on timezones and timestamps (p. 14).

7.17. Sending messages to a user terminal — usertty() destination

This driver writes messages to the terminal of a logged-in user.

The usertty() driver has a single required argument, specifying a username who should receive a copy ofmatching messages. Use the asterisk * to specify every user currently logged in to the system.

Declaration:

usertty(username);

The usertty() does not have any further options nor does it support templates.

309syslog-ng.com

Sending messages to a user terminal — usertty() destination

Example 7.47. Using the usertty() driver

destination d_usertty { usertty("root"); };

310syslog-ng.com

Sending messages to a user terminal — usertty() destination

Chapter 8. Routing messages: log paths,reliability, and filters

8.1. Log paths

Log paths determine what happens with the incoming log messages. Messages coming from the sources listedin the log statement and matching all the filters are sent to the listed destinations.

To define a log path, add a log statement to the syslog-ng configuration file using the following syntax:

log {

source(s1); source(s2); ...

optional_element(filter1|parser1|rewrite1);

optional_element(filter2|parser2|rewrite2);

...

destination(d1); destination(d2); ...

flags(flag1[, flag2...]);

};

WarningLog statements are processed in the order they appear in the configuration file, thus the order of log paths may influencewhat happens to a message, especially when using filters and log flags.

NoteThe order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

Example 8.1. A simple log statementThe following log statement sends all messages arriving to the localhost to a remote server.

source s_localhost { network(ip(127.0.0.1) port(1999)); };

destination d_tcp { network("10.1.2.3" port(1999) localport(999)); };

log { source(s_localhost); destination(d_tcp); };

All matching log statements are processed by default, and the messages are sent to every matching destinationby default. So a single log message might be sent to the same destination several times, provided the destinationis listed in several log statements, and it can be also sent to several different destinations.

This default behavior can be changed using the flags() parameter. Flags apply to individual log paths, theyare not global options. For details and examples on the available flags, see Section 8.1.2, Log path flags (p. 314).The effect and use of the flow-control flag is detailed in Section 8.2, Managing incoming and outgoingmessages with flow-control (p. 316).

311syslog-ng.com

Log paths

8.1.1. Embedded log statements

Starting from version 3.0, syslog-ng can handle embedded log statements (also called log pipes). Embeddedlog statements are useful for creating complex, multi-level log paths with several destinations and use filters,parsers, and rewrite rules.

For example, if you want to filter your incoming messages based on the facility parameter, and then use furtherfilters to send messages arriving from different hosts to different destinations, you would use embedded logstatements.

Figure 8.1. Embedded log statement

Embedded log statements include sources — and usually filters, parsers, rewrite rules, or destinations — andother log statements that can include filters, parsers, rewrite rules, and destinations. The following rules applyto embedded log statements:

■ Only the beginning (also called top-level) log statement can include sources.

■ Embedded log statements can include multiple log statements on the same level (that is, a top-levellog statement can include two or more log statements).

■ Embedded log statements can include several levels of log statements (that is, a top-level log statementcan include a log statement that includes another log statement, and so on).

■ After an embedded log statement, you can write either another log statement, or the flags() optionof the original log statement. You cannot use filters or other configuration objects. This also meansthat flags (except for the flow-control flag) apply to the entire log statement, you cannot use themonly for the embedded log statement.

■ Embedded log statements that are on the same level receive the same messages from the higher-levellog statement. For example, if the top-level log statement includes a filter, the lower-level logstatements receive only the messages that pass the filter.

312syslog-ng.com

Embedded log statements

Figure 8.2. Embedded log statements

Embedded log filters can be used to optimize the processing of log messages, for example, to re-use the resultsof filtering and rewriting operations.

8.1.1.1. Using embedded log statements

Embedded log statements (for details, see Section 8.1.1, Embedded log statements (p. 312)) re-use the resultsof processing messages (for example the results of filtering or rewriting) to create complex log paths. Embeddedlog statements use the same syntax as regular log statements, but they cannot contain additional sources. Todefine embedded log statements, use the following syntax:

log {

source(s1); source(s2);...

optional_element(filter1|parser1|rewrite1);

optional_element(filter2|parser2|rewrite2);

...

destination(d1); destination(d2); ...

#embedded log statement

log {

optional_element(filter1|parser1|rewrite1);

optional_element(filter2|parser2|rewrite2);

...

destination(d1); destination(d2); ...

#another embedded log statement

log {

optional_element(filter1|parser1|rewrite1);

optional_element(filter2|parser2|rewrite2);

...

destination(d1); destination(d2);

...

};

};

#set flags after the embedded log statements

flags(flag1[, flag2...]);

};

313syslog-ng.com

Embedded log statements

Example 8.2. Using embedded log pathsThe following log path sends every message to the configured destinations: both the d_file1 and the d_file2 destinationsreceive every message of the source.

log { source(s_localhost); destination(d_file1); destination(d_file2); };

The next example is equivalent with the one above, but uses an embedded log statement.

log { source(s_localhost); destination(d_file1);

log {destination(d_file2); };

};

The following example uses two filters:

■ messages coming from the host 192.168.1.1 are sent to the d_file1 destination, and

■ messages coming from the host 192.168.1.1 and containing the string example are sent to the d_file2destination.

filter filter_host { host(192.168.1.1) };

filter filter_example { message("example") };

log { source(s_localhost); filter(filter_host); destination(d_file1);

log {filter(filter_example); destination(d_file2); };

};

The following example collects logs from multiple source groups and uses the source() filter in the embedded logstatement to select messages of the s_network source group.

log { source(s_localhost); source(s_network); destination(d_file1);

log {source(s_network); destination(d_file2); };

};

8.1.2. Log path flags

Flags influence the behavior of syslog-ng, and the way it processes messages. The following flags may be usedin the log paths, as described in Section 8.1, Log paths (p. 311).

DescriptionFlagThis flag means that the source of the message is ignored, only the filters of the log path aretaken into account when matching messages. A log statement using the catchall flag processesevery message that arrives to any of the defined sources.

catchall

This flag makes a log statement 'fallback'. Fallback log statements process messages that werenot processed by other, 'non-fallback' log statements.

fallback

'Processed' means that every filter of a log path matched the message. Note that in case ofembedded log paths, the message is considered to be processed if it matches the filters of theouter log path, even if it does not match the filters of the embedded log path. For details, seeExample 8.3, Using log path flags (p. 315).

This flag means that the processing of log messages processed by the log statement ends here,other log statements appearing later in the configuration file will not process the messages

final

processed by the log statement labeled as 'final'. Note that this does not necessarily mean thatmatching messages will be stored only once, as there can be matching log statements processedbefore the current one (syslog-ng PE evaluates log statements in the order they appear in theconfiguration file).

314syslog-ng.com

Log path flags

DescriptionFlag'Processed' means that every filter of a log path matched the message. Note that in case ofembedded log paths, the message is considered to be processed if it matches the filters of theouter log path, even if it does not match the filters of the embedded log path. For details, seeExample 8.3, Using log path flags (p. 315).

Enables flow-control to the log path, meaning that syslog-ng will stop reading messages fromthe sources of this log statement if the destinations are not able to process the messages at the

flow-control

required speed. If disabled, syslog-ng will drop messages if the destination queues are full. Ifenabled, syslog-ng will only drop messages if the destination queues/window sizes are improperlysized. For details, see Section 8.2, Managing incoming and outgoing messages withflow-control (p. 316).

Table 8.1. Log statement flags

WarningThe final, fallback, and catchall flags apply only for the top-level log paths, they have no effect on embedded logpaths.

Example 8.3. Using log path flagsLet's suppose that you have two hosts (myhost_A and myhost_B) that run two applications each (application_A andapplication_B), and you collect the log messages to a central syslog-ng server. On the server, you create two log paths:

one that processes only the messages sent by myhost_A, and■

■ one that processes only the messages sent by application_A.

This means that messages sent by application_A running on myhost_A will be processed by both log paths, and themessages of application_B running on myhost_B will not be processed at all.

■ If you add the final flag to the first log path, then only this log path will process the messages of myhost_A,so the second log path will receive only the messages of application_A running on myhost_B.

■ If you create a third log path that includes the fallback flag, it will process the messages not processedby the first two log paths, in this case, the messages of application_B running on myhost_B.

■ Adding a fourth log path with the catchall flag would process every message received by the syslog-ngserver.

log { source(s_localhost); destination(d_file); flags(catchall); };

The following example shows a scenario that can result in message loss. Do NOT use such a configuration, unless youknow exactly what you are doing. The problem is if a message matches the filters in the first part of the first log path,syslog-ng PE treats the message as 'processed'. Since the first log path includes the final flag, syslog-ng PE will notpass the message to the second log path (the one with the fallback flag). As a result, syslog-ng PE drops messages thatdo not match the filter of the embedded log path.

# Do not use such a configuration, unless you know exactly what you are doing.

log {

source(s_network);

# Filters in the external log path.

# If a message matches this filter, it is treated as 'processed'

filter(f_program);

filter(f_message);

log {

# Filter in the embedded log path.

# If a message does not match this filter, it is lost, it will not be processed

by the 'fallback' log path

filter(f_host);

destination(d_file1);

315syslog-ng.com

Log path flags

};

flags(final);

};

log {

source(s_network);

destination(d_file2);

flags(fallback);

};

8.2. Managing incoming and outgoing messages with flow-control

This section describes the internal message-processing model of syslog-ng, as well as the flow-control featurethat can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particularlog path.

The syslog-ng application monitors (polls) the sources defined in its configuration file, periodically checkingeach source for messages. When a log message is found in one of the sources, syslog-ng polls every source andreads the available messages. These messages are processed and put into the output buffer of syslog-ng (alsocalled fifo). From the output buffer, the operating system sends the messages to the appropriate destinations.

In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng readsonly a fixed number of messages from each source. The log-fetch-limit() option specifies the number ofmessages read during a poll loop from a single source.

Figure 8.3. Managing log messages in syslog-ng

NoteThe log-fetch-limit() parameter can be set as a global option, or for every source individually.

Every destination has its own output buffer. The output buffer is needed because the destination might not beable to accept all messages immediately. The log-fifo-size() parameter sets the size of the output buffer.The output buffer must be larger than the log-fetch-limit() of the sources, to ensure that every messageread during the poll loop fits into the output buffer. If the log path sends messages to a destination from multiplesources, the output buffer must be large enough to store the incoming messages of every source.

TCP and unix-stream sources can receive the logs from several incoming connections (for example manydifferent clients or applications). For such sources, syslog-ng reads messages from every connection, thus thelog-fetch-limit() parameter applies individually to every connection of the source.

316syslog-ng.com

Managing incoming and outgoing messages with flow-control

Figure 8.4. Managing log messages of TCP sources in syslog-ng

The flow-control of syslog-ng introduces a control window to the source that tracks how many messages cansyslog-ng accept from the source. Every message that syslog-ng reads from the source lowers the window sizeby one, every message that syslog-ng successfully sends from the output buffer increases the window size byone. If the window is full (that is, its size decreases to zero), syslog-ng stops reading messages from the source.The initial size of the control window is by default 1000: the log-fifo-size()must be larger than this valuein order for flow-control to have any effect. If a source accepts messages from multiple connections, all messagesuse the same control window.

NoteIf the source can handle multiple connections (for example, network()), the size of the control window is divided bythe value of the max-connections() parameter and this smaller control window is applied to each connection of thesource.

When flow-control is used, every source has its own control window. As a worst-case situation, the outputbuffer of the destination must be set to accommodate all messages of every control window, that is, thelog-fifo-size() of the destination must be greater than number_of_sources*log-iw-size(). Thisapplies to every source that sends logs to the particular destination. Thus if two sources having several connectionsand heavy traffic send logs to the same destination, the control window of both sources must fit into the outputbuffer of the destination. Otherwise, syslog-ng does not activate the flow-control, and messages may be lost.

The syslog-ng application handles outgoing messages the following way:

Figure 8.5. Handling outgoing messages in syslog-ng PE

317syslog-ng.com

Managing incoming and outgoing messages with flow-control

■ Output queue: Messages from the output queue are sent to the target syslog-ng server. The syslog-ngapplication puts the outgoing messages directly into the output queue, unless the output queue isfull. The output queue can hold 64 messages, this is a fixed value and cannot be modified.

■ Disk buffer: If the output queue is full and disk-buffering is enabled, syslog-ng Premium Editionputs the outgoing messages into the disk buffer of the destination.

■ Overflow queue: If the output queue is full and the disk buffer is disabled or full, syslog-ng puts theoutgoing messages into the overflow queue of the destination. (The overflow queue is identical tothe output buffer used by other destinations.) The log-fifo-size() parameter specifies the numberof messages stored in the overflow queue. For details on sizing the log-fifo-size() parameter,see Section 8.2, Managing incoming and outgoing messages with flow-control (p. 316).

There are two types of flow-control: Hard flow-control and soft flow-control.

■ Soft flow-control: In case of soft flow-control there is no message lost if the destination can acceptmessages, but it is possible to lose messages if it cannot accept messages (for example non-writeablefile destination, or the disk becomes full), and all buffers are full. Soft flow-control cannot beconfigured, it is automatically available for file and logstore destinations.

Example 8.4. Soft flow-control

source s_file {file("/tmp/input_file.log");};

destination d_file {file("/tmp/output_file.log");};

destination d_tcp { network("127.0.0.1" port(2222) log-fifo-size(1000)); };

log{ source(s_file); destination(d_file); destination(d_tcp);};

WarningHazard of data loss! For destinations other than file and logstore, soft flow-control is not available. Thus,it is possible to lose log messages on those destinations. To avoid data loss on those destinations, use hardflow-control.

■ Hard flow-control: In case of hard flow-control there is no message lost. To use hard flow-control,enable the flow-control flag in the log path. Hard flow-control is available for all destinations.

Example 8.5. Hard flow-control

source s_file {file("/tmp/input_file.log");};

destination d_file {file("/tmp/output_file.log");};

destination d_tcp { network("127.0.0.1" port(2222) log-fifo-size(1000)); };

log{ source(s_file); destination(d_file); destination(d_tcp)

flags(flow-control);};

8.2.1. Flow-control and multiple destinations

Using flow-control on a source has an important side-effect if the messages of the source are sent to multipledestinations. If flow-control is in use and one of the destinations cannot accept the messages, the other destinations

318syslog-ng.com

Flow-control and multiple destinations

do not receive any messages either, because syslog-ng stops reading the source. For example, if messages froma source are sent to a remote server and also stored locally in a file, and the network connection to the serverbecomes unavailable, neither the remote server nor the local file will receive any messages.

NoteCreating separate log paths for the destinations that use the same flow-controlled source does not avoid the problem.

If you use flow-control and reliable disk-based buffering together with multiple destinations, the flow-controlstarts slowing down the source only when:

■ one destination is down, and

■ the number of messages stored in the disk buffer of the destination reaches (disk-buf-size()minus mem-buf-size()).

8.2.2. Configuring flow-control

For details on how flow-control works, see Section 8.2, Managing incoming and outgoing messages withflow-control (p. 316). The summary of the main points is as follows:

■ The syslog-ng application normally reads a maximum of log-fetch-limit() number of messagesfrom a source.

■ From TCP and unix-stream sources, syslog-ng reads a maximum of log-fetch-limit() fromevery connection of the source. The number of connections to the source is set using themax-connections() parameter.

■ Every destination has an output buffer (log-fifo-size()).

■ Flow-control uses a control window to determine if there is free space in the output buffer for newmessages. Every source has its own control window, the log-iw-size() parameter sets the sizeof the control window.

■ When a source accepts multiple connections, the size of the control window is divided by the valueof the max-connections() parameter and this smaller control window is applied to each connectionof the source.

■ The output buffer must be larger than the control window of every source that logs to the destination.

■ If the control window is full, syslog-ng stops reading messages from the source until some messagesare successfully sent to the destination.

■ If the output buffer becomes full, and neither disk-buffering nor flow-control is used, messages maybe lost.

319syslog-ng.com

Configuring flow-control

WarningIf you modify the max-connections() or the log-fetch-limit() parameter, do not forget to adjust thelog-iw-size() and log-fifo-size() parameters accordingly.

Example 8.6. Sizing parameters for flow-controlSuppose that syslog-ng has a source that must accept up to 300 parallel connections. Such situation can arise when anetwork source receives connections from many clients, or if many applications log to the same socket. Therefore, setthe max-connections() parameter of the source to 300. However, the log-fetch-limit() (default value: 10)parameter applies to every connection of the source individually, while the log-iw-size() (default value: 1000)parameter applies to the source. In a worst-case scenario, the destination does not accept any messages, while all 300connections send at least log-fetch-limit() number of messages to the source during every poll loop. Therefore, thecontrol window must accommodate at least max-connections()*log-fetch-limit() messages to be able to readevery incoming message of a poll loop. In the current example this means that (log-iw-size() should be greater than300*10=3000. If the control window is smaller than this value, the control window might fill up with messages fromthe first connections — causing syslog-ng to read only one message of the last connections in every poll loop.

The output buffer of the destination must accommodate at least log-iw-size() messages, but use a greater value: inthe current example 3000*10=30000messages. That way all incoming messages of ten poll loops fit in the output buffer.If the output buffer is full, syslog-ng does not read any messages from the source until some messages are successfullysent to the destination.

source s_localhost {

network(ip(127.0.0.1) port(1999) max-connections(300)); };

destination d_tcp {

network("10.1.2.3" port(1999); localport(999)); log-fifo-size(30000); };

log { source(s_localhost); destination(d_tcp); flags(flow-control); };

If other sources send messages to this destination, than the output buffer must be further increased. For example, if anetwork host with maximum 100 connections also logs into the destination, than increase the log-fifo-size() by10000.

source s_localhost {

network(ip(127.0.0.1) port(1999) max-connections(300)); };

source s_tcp {

network(ip(192.168.1.5) port(1999) max-connections(100)); };

destination d_tcp {

network("10.1.2.3" port(1999); localport(999)); log-fifo-size(40000); };

log { source(s_localhost); destination(d_tcp); flags(flow-control); };

8.3. Using disk-based and memory buffering

The syslog-ng Premium Edition application can store messages on the local hard disk if the destination (forexample, the central log server) or the network connection to the destination becomes unavailable. The syslog-ngPE application automatically sends the stored messages to the destination when the connection is reestablished.The disk buffer is used as a queue: when the connection to the destination is reestablished, syslog-ng PE sendsthe messages to the destination in the order they were received.

NoteDisk-based buffering can be used in conjunction with flow-control. For details on flow-control, see Section 8.2, Managingincoming and outgoing messages with flow-control (p. 316).

The following destination drivers can use disk-based buffering: elasticsearch(), elasticsearch2(),hdfs(), kafka(), mongodb(), program(), smtp(), snmp(), sql(), unix-dgram(), and unix-stream().The network(), syslog(), tcp(), and tcp6() destination drivers can also use disk-based buffering, except

320syslog-ng.com

Using disk-based and memory buffering

when using the udp transport method. (The other destinations or protocols do not provide the necessary feedbackmechanisms required for disk-based buffering.)

Every such destination uses a separate disk buffer (similarly to the output buffers controlled bylog-fifo-size()). The hard disk space is not pre-allocated, so ensure that there is always enough free spaceto store the disk buffers even when the disk buffers are full.

If syslog-ng PE is restarted (using the /etc/init.d/syslog-ng restart command, or another appropriatecommand on your platform), it automatically saves any unsent messages from the disk buffer and the outputqueue. After the restart, syslog-ng PE sends the saved messages to the destination. In other words, the diskbuffer is persistent. The disk buffer is also resistant to syslog-ng PE crashes.

The syslog-ng PE application supports two types of disk buffering: reliable and normal. For details, see Section8.3.1, Enabling reliable disk-based buffering (p. 322) and Section 8.3.2, Enabling normal disk-basedbuffering (p. 323), respectively.

Message handling and normal disk-based buffering. When you use disk-based buffering, and thereliable() option is set to no, syslog-ng PE handles outgoing messages the following way:

Figure 8.6. Handling outgoing messages in syslog-ng PE

■ Output queue: Messages from the output queue are sent to the destination (for example, your centrallog server). The syslog-ng PE application puts the outgoing messages directly into the output queue,unless the output queue is full. By default, the output queue can hold 64 messages (you can adjustit using the quot-size() option).

■ Disk buffer: If the output queue is full, disk-buffering is enabled, and reliable() is set to no,syslog-ng PE puts the outgoing messages into the disk buffer of the destination. (The disk buffer isenabled if the log-disk-fifo-size() parameter of the destination is larger than 0. This optionspecifies the size of the disk buffer in bytes.)

■ Overflow queue: If the output queue is full and the disk buffer is disabled or full, syslog-ng PE putsthe outgoing messages into the overflow queue of the destination. (The overflow queue is identicalto the output buffer used by other destinations.) The log-fifo-size() parameter specifies the

321syslog-ng.com

Using disk-based and memory buffering

number of messages stored in the overflow queue. For details on sizing the log-fifo-size()

parameter, see also Section 8.2, Managing incoming and outgoingmessages with flow-control (p. 316).

NoteUsing disk buffer can significantly decrease performance.

Message handling and reliable disk-based buffering. When you use disk-based buffering, and thereliable() option is set to yes, syslog-ng PE handles outgoing messages the following way.

The mem-buf-size() option determines when flow-control is triggered. All messages arriving to the log paththat includes the destination using the disk-buffer are written into the disk-buffer, until the size of the disk-bufferreaches (disk-buf-size() minus mem-buf-size()). Above that size, messages are written into both thedisk-buffer and the memory-buffer, indicating that flow-control needs to slow down the message source. Thesemessages are not taken out from the control window (governed by log-iw-size()), causing the controlwindow to fill up. If the control window is full, the flow-control completely stops reading incoming messagesfrom the source. (As a result, mem-buf-size() must be at least as large as log-iw-size().)

8.3.1. Enabling reliable disk-based buffering

The following destination drivers can use disk-based buffering: elasticsearch(), elasticsearch2(),hdfs(), kafka(), mongodb(), program(), smtp(), snmp(), sql(), unix-dgram(), and unix-stream().The network(), syslog(), tcp(), and tcp6() destination drivers can also use disk-based buffering, exceptwhen using the udp transport method. (The other destinations or protocols do not provide the necessary feedbackmechanisms required for disk-based buffering.)

To enable reliable disk-based buffering, use the disk-buffer(reliable(yes)) parameter in the destination.Use reliable disk-based buffering if you do not want to lose logs in case of reload/restart, unreachable destinationor syslog-ng PE crash. This solution provides a slower, but reliable disk-buffer option. It is created and initializedat startup and gradually grows as new messages arrive. The filename of the reliable disk buffer file is thefollowing: <syslog-ng path>/var/syslog-ng-00000.rqf.

Example 8.7. Example for using reliable disk-based buffering

destination d_BSD {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-size(10000)

disk-buf-size(2000000)

reliable(yes)

)

);

};

322syslog-ng.com

Enabling reliable disk-based buffering

8.3.2. Enabling normal disk-based buffering

The following destination drivers can use disk-based buffering: elasticsearch(), elasticsearch2(),hdfs(), kafka(), mongodb(), program(), smtp(), snmp(), sql(), unix-dgram(), and unix-stream().The network(), syslog(), tcp(), and tcp6() destination drivers can also use disk-based buffering, exceptwhen using the udp transport method. (The other destinations or protocols do not provide the necessary feedbackmechanisms required for disk-based buffering.)

To enable normal disk-based buffering, use the disk-buffer(reliable(no)) parameter in the destination.Use normal disk-based buffering if you want a solution that is faster than the reliable disk-based buffering. Inthis case, disk buffering will be less reliable and it is possible to lose logs in case of syslog-ng PE crash. Thefilename of the normal disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

It is possible to use this option without using the disk-buffer plugin. In this case, use thelog-disk-fifo-size() parameter in the destination.

Example 8.8. Example for using normal disk-based bufferingWhen using the disk-buffer plugin

destination d_BSD {

network(

"127.0.0.1"

port(3333)

disk-buffer(

mem-buf-length(10000)

disk-buf-size(2000000)

reliable(no)

)

);

};

Without disk-buffer plugin

destination d_BSD {

network(

"127.0.0.1"

port(3333)

log-disk-fifo-size(2000000)

log-fifo-size(10000)

);

};

8.3.3. Enabling memory buffering

To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination driverscan use memory buffering. Use memory buffering if you want to send logs to destinations where disk-basedbuffering is not available. Or if you want the fastest solution, and if syslog-ng PE crash or network downtimeis never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logsare stored only in the memory.

Example 8.9. Example for using memory buffering

destination d_BSD {

network(

"127.0.0.1"

port(3333)

log-fifo-size(10000)

323syslog-ng.com

Enabling normal disk-based buffering

);

};

8.4. Client-side failover

syslog-ng PE can detect if the remote server of a network destination becomes unaccessible, and start sendingmessages to a secondary server. Multiple failover servers can be configured, so if the secondary server becomesunaccessible as well, syslog-ng PE will switch to the third server in the list, and so on. If there are no morefailover servers left, syslog-ng PE returns to the beginning of a list and attempts to connect to the primary server.

When syslog-ng PE starts up, it will always try to connect to the primary server first, but once it fails over toa secondary server, it will not automatically attempt to return to the primary server even if it becomes available.If syslog-ng PE is restarted, it will attempt to connect the primary server. Reloading the configuration ofsyslog-ng PE will not cause syslog-ng PE to return to the primary server, unless the configuration of thedestination has changed.

If syslog-ng PE uses TLS-encryption to communicate with the remote server, syslog-ng PE checks the certificateof the failover server as well. The certificates of the failover servers should match their domain names or IPaddresses — for details, see Section 10.2, Encrypting log messages with TLS (p. 351). Note that when mutualauthentication is used, the syslog-ng PE client sends the same certificate to every server.

The primary server and the failover servers must be accessible with the same communication method: it is notpossible to use different destination drivers or options for the different servers.

NoteClient-side failover works only for TCP-based connections (including TLS-encrypted connections), that is, the syslog()and network() destination drivers (excluding UDP transport).

Client-side failover is not supported in the sql() driver, even though it may use a TCP connection to access a remotedatabase.

For details on configuring failover servers, see Section 7.8.1, network() destination options (p. 235) and Section7.14.1, syslog() destination options (p. 289).

8.5. Filters

The following sections describe how to select and filter log messages.

■ Section 8.5.1, Using filters (p. 325) describes how to configure and use filters.

■ Section 8.5.2, Combining filters with boolean operators (p. 325) shows how to create complex filtersusing boolean operators.

■ Section 8.5.3, Comparing macro values in filters (p. 326) explains how to evaluate macros in filters.

■ Section 8.5.4, Using wildcards, special characters, and regular expressions in filters (p. 327) providestips on using regular expressions.

■ Section 8.5.5, Tagging messages (p. 328) explains how to tag messages and how to filter on the tags.

324syslog-ng.com

Client-side failover

■ Section 8.5.6, Filter functions (p. 328) is a detailed description of the filter functions available insyslog-ng PE.

8.5.1. Using filters

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for theparticular message. If a log statement includes filters, the messages are sent to the destinations only if they passall filters of the log path. For example, a filter can select only the messages originating from a particular host.Complex filters can be created using filter functions and logical boolean expressions.

To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:

filter <identifier> { <filter_type>("<filter_expression>"); };

Example 8.10. A simple filter statementThe following filter statement selects the messages that contain the word deny and come from the host example.

filter demo_filter { host("example") and match("deny" value("MESSAGE")) };

For the filter to have effect, include it in a log statement:

log {

source(s1);

filter(demo_filter);

destination(d1);};

8.5.2. Combining filters with boolean operators

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only ifall filters are true for the message. In other words, the filters are connected with the logical AND operator. Inthe following example, no message arrives to the destination, because the filters are exclusive (the hostnameof a client cannot be example1 and example2 at the same time):

filter demo_filter1 { host("example1"); };

filter demo_filter2 { host("example2"); };

log {

source(s1); source(s2);

filter(demo_filter1); filter(demo_filter2);

destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

filter demo_filter { host("example1") or host("example2"); };

log {

source(s1); source(s2);

filter(demo_filter);

destination(d1); destination(d2); };

Use the not operator to invert filters, for example, to select the messages that were not sent by host example1:

filter demo_filter { not host("example1"); };

325syslog-ng.com

Using filters

However, to select the messages that were not sent by host example1 or example2, you have to use the andoperator (that's how boolean logic works):

filter demo_filter { not host("example1") and not host("example2"); };

Alternatively, you can use parentheses to avoid this confusion:

filter demo_filter { not (host("example1") or host("example2")); };

For a complete description on filter functions, see Section 8.5.6, Filter functions (p. 328).

The following filter statement selects the messages that contain the word deny and come from the host example.

filter demo_filter { host("example") and match("deny" value("MESSAGE")); };

The value() parameter of the match function limits the scope of the function to the text part of the message(that is, the part returned by the ${MESSAGE} macro). For details on using the match() filter function, seeSection match() (p. 331).

TipFilters are often used together with log path flags. For details, see Section 8.1.2, Log path flags (p. 314).

8.5.3. Comparing macro values in filters

Starting with syslog-ng PE version 4 F1, it is also possible to compare macro values and templates as numericaland string values. String comparison is alphabetical: it determines if a string is alphabetically greater or equalto another string. Use the following syntax to compare macro values or templates. For details on macros andtemplates, see Section 14.1, Customizing message format (p. 378).

filter <filter-id>

{"<macro-or-template>" operator "<value-or-macro-or-template>"};

Example 8.11. Comparing macro values in filtersThe following expression selects log messages containing a PID (that is, ${PID} macro is not empty):

filter f_pid {"${PID}" !=""};

The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument ofthe operator and compares the values as strings:

filter f_pid {"${HOST}${PID}" eq "${HOST}"};

The following example selects messages with priority level 4 or higher.

filter f_level {"${LEVEL_NUM}" > "5"};

The following filter selects messages which have the collector word in the soc@0.device structured data field.

filter f_fwd_collectors {"${.SDATA.soc@0.device}" eq "collector"};

This filter expresison selects messages, which has the FW string in the soc@0.class structured data field.

filter f_debug { match("FW" value(".SDATA.soc@0.class") type("string")); };

326syslog-ng.com

Comparing macro values in filters

Note that:

■ The macro or template must be enclosed in double-quotes.

■ The $ character must be used before macros.

■ Using comparator operators can be equivalent to using filter functions, but is somewhat slower. Forexample, using"${HOST}" eq "myhost" is equivalent to usinghost("myhost" type(string)).

■ You can use any macro in the expression, including user-defined macros from parsers and resultsof pattern database classifications.

■ The results of filter functions are boolean values, so they cannot be compared to other values.

■ You can use boolean operators to combine comparison expressions.

The following operators are available:

MeaningString operatorNumerical operatorEqualseq==

Not equal tone!=

Greater thangt>

Less thanlt<

Greater than or equalge>=

Less than or equalle=<Table 8.2. Numerical and string comparison operators

8.5.4. Using wildcards, special characters, and regular expressions in filters

The host(), match(), and program() filter functions accept regular expressions as parameters. The exacttype of the regular expression to use can be specified with the type() option. By default, syslog-ng PE usesPOSIX regular expressions.

To use other expression types, add the type() option after the regular expression. For example:

message("^(.+)\\1$" type("pcre"))

In regular expressions, the asterisk (*) character means 0, 1 or any number of the previous expression. Forexample, in the f*ilter expression the asterisk means 0 or more f letters. This expression matches for thefollowing strings: ilter, filter, ffilter, and so on. To achieve the wildcard functionality commonlyrepresented by the asterisk character in other applications, use .* in your expressions, for example f.*ilter.

Alternatively, if you do not need regular expressions, only wildcards, use type(glob) in your filter:

Example 8.12. Filtering with widcardsThe following filter matches on hostnames starting with the myhost string, for example, on myhost-1, myhost-2, andso on.

filter f_wildcard {host("myhost*" type(glob));};

327syslog-ng.com

Using wildcards, special characters, and regular expressions in filters

For details on using regular expressions in syslog-ng PE, see Section 14.3, Regular expressions (p. 403).

To filter for special control characters like the carriage return (CR), use the \r escape prefix in syslog-ng PEversion 3.0 and 3.1. In syslog-ng PE 3.2 and later, you can also use the \x escape prefix and the ASCII codeof the character. For example, to filter on carriage returns, use the following filter:

filter f_carriage_return {match("\x0d" value ("MESSAGE"));};

8.5.5. Tagging messages

You can label the messages with custom tags. Tags are simple labels, identified by their names, which must beunique. Currently syslog-ng PE can tag a message at two different places:

■ at the source when the message is received, and

■ when the message matches a pattern in the pattern database. For details on using the pattern database,see Section 16.2, Using pattern databases (p. 421), for details on creating tags in the pattern database,see Section 16.5.2, The syslog-ng pattern database format (p. 433).

When syslog-ng receives a message, it automatically adds the .source.<id_of_the_source_statement>tag to the message. Use the tags() option of the source to add custom tags, and the tags() option of thefilters to select only specific messages.

NoteTagging messages and also filtering on the tags is very fast, much faster than other types of filters.■

■ Tags are available locally, that is, if you add tags to a message on the client, these tags will not be availableon the server.

■ To include the tags in the message, use the ${TAGS} macro in a template. Alternatively, if you are usingthe IETF-syslog message format, you can include the ${TAGS} macro in the .SDATA.meta part of themessage. Note that the ${TAGS} macro is available only in syslog-ng PE 3.1.1 and later.

For an example on tagging, see Example 8.14, Adding tags and filtering messages with tags (p. 333).

8.5.6. Filter functions

The following functions may be used in the filter statement, as described in Section 8.5, Filters (p. 324).

DescriptionNameFilter messages based on the sending facility.facility()

Call another filter function.filter()

Filter messages based on the sending host.host()

File-based whitelisting and blacklisting.inlist()

Filter messages based on their priority.level() or priority()

Use a regular expression to filter messages based on aspecified header or content field.

match()

328syslog-ng.com

Tagging messages

DescriptionNameUse a regular expression to filter messages based ontheir content.

message()

Filter messages based on the IP address of the sendinghost.

netmask()

Filter messages based on the sending application.program()

Select messages of the specified syslog-ng PE sourcestatement.

source()

Select messages having the specified tag.tags()Table 8.3. Filter functions available in syslog-ng PE

facility()facility(<facility-name>) or facility(<facility-code>) or facility(<facility-name>..<facility-name>)Synopsis:

Description: Match messages having one of the listed facility codes.

The facility() filter accepts both the name and the numerical code of the facility or the importance level.Facility codes 0-23 are predefined and can be referenced by their usual name. Facility codes above 24 are notdefined.

You can use the facility filter the following ways:

■ Use a single facility name, for example, facility(user)

■ Use a single facility code, for example, facility(1)

■ Use a facility range (works only with facility names), for example, facility(local0..local5)

The syslog-ng application recognizes the following facilities: (Note that some of these facilities are availableonly on specific platforms.)

FacilityFacility nameNumerical Codekernel messageskern0

user-level messagesuser1

mail systemmail2

system daemonsdaemon3

security/authorization messagesauth4

messages generated internally bysyslogd

syslog5

line printer subsystemlpr6

network news subsystemnews7

UUCP subsystemuucp8

clock daemoncron9

security/authorization messagesauthpriv10

329syslog-ng.com

Filter functions

FacilityFacility nameNumerical CodeFTP daemonftp11

NTP subsystemntp12

log auditsecurity13

log alertconsole14

clock daemonsolaris-cron15

locally used facilities (local0-local7)local0..local716-23Table 8.4. syslog Message Facilities recognized by the facility() filter

filter()filter(filtername)Synopsis:

Description: Call another filter rule and evaluate its value.

host()host(regexp)Synopsis:

Description: Match messages by using a regular expression against the hostname field of log messages. Notethat you can filter only on the actual content of the HOST field of the message (or what it was rewritten to).That is, syslog-ng PE will compare the filter expression to the content of the ${HOST} macro. This means thatfor the IP address of a host will not match, even if the IP address and the hostname field refers to the same host.To filter on IP addresses, use the netmask() filter.

filter demo_filter { host("example") };

inlist()in-list("</path/to/file.list>", value("<field-to-filter>"));Synopsis:

Description: Matches the value of the specified field to a list stored in a file, allowing you to do simple,file-based black- and whitelisting. The file must be a plain-text file, containing one entry per line. The syslog-ngPE application loads the entire file, and compares the value of the specified field (for example, ${PROGRAM})to entries in the file. When you use the in-list filter, note the following points:

■ Comparing the values is case-sensitive.

■ Only exact matches are supported, partial and substring matches are not.

■ If you modify the list file, reload the configuration of syslog-ng PE for the changes to take effect.

Available in syslog-ng PE and later.

Example 8.13. Selecting messages using the in-list filterCreate a text file that contains the programs (as in the ${PROGRAM} field of their log messages) you want to select. Forexample, you want to forward only the logs of a few applications from a host: kernel, sshd, and sudo. Create the/etc/syslog-ng/programlist.list file with the following contents:

330syslog-ng.com

Filter functions

kernel

sshd

sudo

The following filter selects only the messages of the listed applications:

filter f_whitelist { in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); };

Create the appropriate sources and destinations for your environment, then create a log path that uses the previous filterto select only the log messages of the applications you need:

log {

source(s_all);

filter(f_whitelist);

destination(d_logserver);};

To create a blacklist filter, simply negate the in-list filter:

filter f_blacklist { not in-list("/etc/syslog-ng/programlist.list", value("PROGRAM")); };

level() or priority()level(<priority-level>) or level(<priority-level>..<priority-level>)Synopsis:

Description: The level() filter selects messages corresponding to a single importance level, or a level-range.To select messages of a specific level, use the name of the level as a filter parameter, for example use thefollowing to select warning messages:

level(warning)

To select a range of levels, include the beginning and the ending level in the filter, separated with two dots(..). For example, to select every message of error or higher level, use the following filter:

level(err..emerg)

The level() filter accepts the following levels: emerg, alert, crit, err, warning, notice, info, debug.

match()match(regexp)Synopsis:

Description: Match a regular expression to the headers and the message itself (that is, the values returned bythe MSGHDR and MSG macros). Note that in syslog-ng version 2.1 and earlier, the match() filter was appliedonly to the text of the message, excluding the headers. This functionality has been moved to the message()filter.

To limit the scope of the match to a specific part of the message (identified with a macro), use the match(regexpvalue("MACRO")) syntax. Do not include the $ sign in the parameter of the value() option.

The value() parameter accepts both built-in macros and user-defined ones created with a parser or using apattern database. For details on macros and parsers, see Section 14.1.2, Templates and macros (p. 379), Section15.1, Parsing messages with comma-separated and similar values (p. 407), and Section 16.2.1, Using parserresults in filters and templates (p. 422).

message()message(regexp)Synopsis:

331syslog-ng.com

Filter functions

Description: Match a regular expression to the text of the log message, excluding the headers (that is, the valuereturned by the MSG macros). Note that in syslog-ng version 2.1 and earlier, this functionality was performedby the match() filter.

netmask()netmask(ipv4/mask)Synopsis:

Description: Select only messages sent by a host whose IP address belongs to the specified IPv4 subnet. Notethat this filter checks the IP address of the last-hop relay (the host that actually sent the message to syslog-ngPE), not the contents of the HOST field of the message. You can use both the dot-decimal and the CIDR notationto specify the netmask. For example, 192.168.5.0/255.255.255.0 or 192.168.5.0/24. To filter IPv6addresses, see Section netmask6() (p. 332).

netmask6()netmask6(ipv6/mask)Synopsis:

Description: Select only messages sent by a host whose IP address belongs to the specified IPv6 subnet. Notethat this filter checks the IP address of the last-hop relay (the host that actually sent the message to syslog-ngPE), not the contents of the HOST field of the message. You can use both the regular and the compressed formatto specify the IP address, for example, 1080:0:0:0:8:800:200C:417A or 1080::8:800:200C:417A. Ifyou do not specify the address, localhost is used. Use the netmask (also called prefix) to specify how manyof the leftmost bits of the address comprise the netmask (values 1-128 are valid). For example, the followingspecify a 60-bit prefix: 12AB:0000:0000:CD30:0000:0000:0000:0000/60 or 12AB::CD30:0:0:0:0/60.Note that if you set an IP address and a prefix, syslog-ng PE will ignore the bits of the address after the prefix.To filter IPv4 addresses, see Section netmask() (p. 332).

The netmask6() filter is available in syslog-ng PE 5.0.8 and 5.2.2 and later.

WarningIf the IP address is not syntactically correct, the filter will never match. The syslog-ng PE application currently does notsend a warning for such configuration errors.

program()program(regexp)Synopsis:

Description: Match messages by using a regular expression against the program name field of log messages.

source()source idSynopsis:

Description: Select messages of a source statement. This filter can be used in embedded log statements if theparent statement contains multiple source groups — only messages originating from the selected source groupare sent to the destination of the embedded log statement.

332syslog-ng.com

Filter functions

tags()tagSynopsis:

Description: Select messages labeled with the specified tag. Every message automatically has the tag of itssource in .source.<id_of_the_source_statement> format. This option is available only in syslog-ng3.1 and later.

Example 8.14. Adding tags and filtering messages with tags

source s_tcp {

network(ip(192.168.1.1) port(1514) tags("tcp", "router"));

};

Use the tags() option of the filters to select only specific messages:

filter f_tcp {

tags(".source.s_tcp");

};

filter f_router {

tags("router");

};

NoteThe syslog-ng PE application automatically adds the class of the message as a tag using the.classifier.<message-class> format. For example, messages classified as "system" receive the.classifier.system tag. Use the tags() filter function to select messages of a specific class.

filter f_tag_filter {tags(".classifier.system");};

8.6. Dropping messages

To skip the processing of a message without sending it to a destination, create a log statement with the appropriatefilters, but do not include any destination in the statement, and use the final flag.

Example 8.15. Skipping messagesThe following log statement drops all debug level messages without any further processing.

filter demo_debugfilter { level(debug); };

log { source(s_all); filter(demo_debugfilter); flags(final); };

333syslog-ng.com

Dropping messages

Chapter 9. Global options of syslog-ng PE

9.1. Configuring global syslog-ng options

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used,and other general points. Each option may have parameters, similarly to driver specifications. To set globaloptions, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };

Example 9.1. Using global optionsTo disable domain name resolving, add the following line to the syslog-ng configuration file:

options { use-dns(no); };

For a detailed list of the available options, see Section 9.2, Global options (p. 334). For important global optionsand recommendations on their use, see Chapter 20, Best practices and examples (p. 460).

9.2. Global options

The following options can be specified in the options statement, as described in Section 9.1, Configuring globalsyslog-ng options (p. 334).

bad-hostname()regular expressionAccepted values:noDefault:

Description: A regexp containing hostnames which should not be handled as hostnames.

chain-hostnames()yes | noAccepted values:noDefault:

Description: Enable or disable the chained hostname format. If a client sends the log message directly to thesyslog-ng PE server, the chain-hostnames() option is enabled on the server, and the client sends a hostnamein the message that is different from its DNS hostname (as resolved from DNS by the syslog-ng PE server),then the server can append the resolved hostname to the hostname in the message (separated with a / character)when the message is written to the destination.

For example, consider a client-server scenario with the following hostnames:client-hostname-from-the-message, client-hostname-resolved-on-the-server,server-hostname. The hostname of the log message written to the destination depends on the

334syslog-ng.com

Configuring global syslog-ng options

keep-hostname() and thechain-hostnames() options. Howkeep-hostname() andchain-hostnames()options are related is described in the following table.

keep-hostname() setting on the servernoyes

client-hostname-from-the-message/client-hostname-resolved-on-the-server

client-hostname-from-the-messageyeschain-hostnames()setting on the server

client-hostname-resolved-on-the-serverclient-hostname-from-the-messageno

If the log message is forwarded to the syslog-ng PE server via a syslog-ng PE relay, the hostname depends onthe settings of the keep-hostname() and the chain-hostnames() options both on the syslog-ng PE relayand the syslog-ng PE server.

For example, consider a client-relay-server scenario with the following hostnames:client-hostname-from-the-message, client-hostname-resolved-on-the-relay,client-hostname-resolved-on-the-server, relay-hostname-resolved-on-the-server. Howkeep-hostname() and chain-hostnames() options are related is described in the following table.

335syslog-ng.com

chain-hostnames()

chain-hostnames() setting on the servernoyes

keep-hostname() setting on theserver

keep-hostname() setting on theserver

noyesnoyes

relay-hostname-resolved-on-the-server

client-hostname-from-the-message

client-hostname-from-the-message/ relay-hostname-resolved-on-the-server

client-hostname-from-the-message

yes

k e e p -hostname()setting onthe relay

yes

c h a i n -hostnames()setting onthe relay

client-hostname-from-the-message/client-hostname-resolved-on-the-relay

client-hostname-from-the-message/client-hostname-resolved-on-the-relay / relay-hostname-resolved-on-the-server

client-hostname-from-the-message/client-hostname-resolved-on-the-relay

no

client-hostname-from-the-message

client-hostname-from-the-message/ relay-hostname-resolved-on-the-server

client-hostname-from-the-message

yes

k e e p -hostname()setting onthe relay

noclient-hostname-resolved-on-the-relay

client-hostname-resolved-on-the-relay / relay-hostname-resolved-on-the-server

client-hostname-resolved-on-the-relay

no

The chain-hostnames() option of syslog-ng can interfere with the way syslog-ng PE counts the log sourcehosts, causing syslog-ng to think there are more hosts logging to the central server, especially if the clientssends a hostname in the message that is different from its real hostname (as resolved from DNS). Disable thechain-hostnames() option on your log sources to avoid any problems related to license counting.

check-hostname()yes | noAccepted values:noDefault:

Description: Enable or disable checking whether the hostname contains valid characters.

create-dirs()yes | noAccepted values:noDefault:

336syslog-ng.com

check-hostname()

Description: Enable or disable directory creation for destination files.

custom-domain()stringAccepted values:empty stringDefault:

Description: Use this option to specify a custom domain name that is appended after the short hostname toreceive the FQDN. This option affects every outgoing message: eventlog sources, file sources, MARK messagesand internal messages of syslog-ng PE.

■ If the hostname is a short hostname, the custom domain name is appended after the hostname (forexample mypc becomes mypc.customcompany.local).

■ If the hostname is an FQDN, the domain name part is replaced with the custom domain name (forexample if the FQDN in the forwarded message is mypc.mycompany.local and the custom domainname is customcompany.local, the hostname in the outgoing message becomesmypc.customcompany.local).

dir-group()groupidAccepted values:rootDefault:

Description: The default group for newly created directories.

dir-owner()useridAccepted values:rootDefault:

Description: The default owner of newly created directories.

dir-perm()permission valueAccepted values:0700Default:

Description: The permission mask of directories created by syslog-ng. Log directories are only created if a fileafter macro expansion refers to a non-existing directory, and directory creation is enabled (see also thecreate-dirs() option). For octal numbers prefix the number with 0, for example use 0755 for rwxr-xr-x.

To preserve the original properties of an existing directory, use the option without specifying an attribute:dir-perm(). Note that when creating a new directory without specifying attributes for dir-perm(), thedefault permission of the directories is masked with the umask of the parent process (typically 0022).

337syslog-ng.com

custom-domain()

dns-cache()yes | noAccepted values:yesDefault:

Description: Enable or disable DNS cache usage.

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

dns-cache-expire()number (seconds)Accepted values:3600Default:

Description: Number of seconds while a successful lookup is cached.

dns-cache-expire-failed()number (seconds)Accepted values:60Default:

Description: Number of seconds while a failed lookup is cached.

dns-cache-hosts()filenameAccepted values:unsetDefault:

Description: Name of a file in /etc/hosts format that contains static IP->hostname mappings. Use thisoption to resolve hostnames locally without using a DNS. Note that any change to this file triggers a reload insyslog-ng and is instantaneous.

dns-cache-size()number of hostnamesAccepted values:1007Default:

Description: Number of hostnames in the DNS cache.

file-template()stringAccepted values:

Default:

Description: Specifies a template that file-like destinations use by default. For example:

338syslog-ng.com

dns-cache()

template t_isostamp { template("$ISODATE $HOST $MSGHDR$MSG\n"); };

options { file-template(t_isostamp); };

flush-lines()number (messages)Accepted values:1Default:

Description: Specifies how many lines are flushed to a destination at a time. The syslog-ng PE applicationwaits for this number of lines to accumulate and sends them off in a single batch. Setting this number highincreases throughput as fully filled frames are sent to the network, but also increases message latency.

flush-timeout() (OBSOLETE)time in millisecondsAccepted values:10000Default:

Description: This is an obsolete option. Specifies the time syslog-ng waits for lines to accumulate in its outputbuffer. For details, see the flush-lines() option.

NoteThis option will be removed from the list of acceptable options. After that, your configuration will become invalid if itstill contains the flush-timeout() option. To avoid future problems, remove this option from your configuration.

frac-digits()number (digits of fractions of a second)Type:Value of the global option (which defaults to 0)Default:

Description: The syslog-ng application can store fractions of a second in the timestamps according to theISO8601 format. The frac-digits() parameter specifies the number of digits stored. The digits storing thefractions are padded by zeros if the original timestamp of the message specifies only seconds. Fractions canalways be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO8601timestamps as well.

group()groupidAccepted values:rootDefault:

Description: The default group of output files. By default, syslog-ng changes the privileges of accessed files(for example /dev/null) to root.root 0600. To disable modifying privileges, use this option with the -1value.

339syslog-ng.com

flush-lines()

keep-hostname()yes or noType:noDefault:

Description: Enable or disable hostname rewriting.

■ If enabled (keep-hostname(yes)), syslog-ng PE assumes that the incoming log message was sentby the host specified in the HOST field of the message.

■ If disabled (keep-hostname(no)), syslog-ng PE rewrites the HOST field of the message, either tothe IP address (if the use-dns() parameter is set to no), or to the hostname (if the use-dns()

parameter is set to yes and the IP address can be resolved to a hostname) of the host sending themessage to syslog-ng PE. For details on using name resolution in syslog-ng PE, see Section 20.3,Using name resolution in syslog-ng (p. 461).

NoteIf the log message does not contain a hostname in its HOST field, syslog-ng PE automatically adds a hostname to themessage.

For messages received from the network, this hostname is the address of the host that sent the message (thismeans the address of the last hop if the message was transferred via a relay).

■

■ For messages received from the local host, syslog-ng PE adds the name of the host.

This option can be specified globally, and per-source as well. The local setting of the source overrides the globaloption if available.

NoteWhen relaying messages, enable this option on the syslog-ng PE server and also on every relay, otherwise syslog-ng PEwill treat incoming messages as if they were sent by the last relay.

keep-timestamp()yes or noType:yesDefault:

Description: Specifies whether syslog-ng should accept the timestamp received from the sending applicationor client. If disabled, the time of reception will be used instead. This option can be specified globally, andper-source as well. The local setting of the source overrides the global option if available.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of syslog-ng PE).

340syslog-ng.com

keep-hostname()

log-fifo-size()number (messages)Accepted values:10000Default:

Description: The number of messages that the output queue can store.

log-msg-size()number (bytes)Accepted values:65535Default:

Description:Maximum length of a message in bytes. This length includes the entire message (the data structureand individual fields). The maximal value that can be set is 268435456 bytes (256MB). For messages usingthe IETF-syslog message format (RFC5424), the maximal size of the value of an SDATA field is 64kB.

logstore-journal-shmem-threshold()number (bytes)Type:536870912Default:

Description: If the size of memory (in bytes) required by journal files increases above this value, syslog-ngPE maps only a single block of every logstore journal into the memory. Default value: 536870912 (512 MB).

If the memory required for the journal files exceeds the logstore-journal-shmem-threshold() limit,syslog-ng PE will store only a single journal block of every journal file in the memory, and — if more blocksare needed for a journal — store the additional blocks on the hard disk. Opening new logstore files meansallocating memory for one new journal block for every new file. In extreme situations involving large traffic,this can lead to syslog-ng PE consuming the entire memory of the system. Adjust the journal-block-size()and your file-naming conventions as needed to avoid such situations. For details on logstore journals, see Section7.6.2, Journal files (p. 217).

Example 9.2. Calculating memory usage of logstore journalsIf you are using the default settings (4 journal blocks for every logstore journal, one block is 1MB,logstore-journal-shmem-threshold() is 512MB), this means that syslog-ng PE will allocate 4MB memory forevery open logstore file, up to 512MB if you have 128 open logstore files. Opening a new logstore file would require 4more megabytes of memory for journaling, bringing the total required memory to 516MB, which is above thelogstore-journal-shmem-threshold(). In this case, syslog-ng PE switches to storing only a single journal blockin the memory, lowering the memory requirements of journaling to 129MB. However, opening more and more logstorefiles will require more and more memory, and this is not limited, except when syslog-ng PE reaches the maximum numberof files that can be open (as set in the --fd-limit command-line option).

Example 9.3. Limiting the memory use of journal filesThe following example causes syslog-ng PE to map only a single journal block into the host's memory if the total memoryrange used by logstore journals would be higher than 32 MB.

options {

logstore-journal-shmem-threshold(33554432);

};

destination d_messages { logstore("/var/log/messages_logstore.lgs"

journal-block-size(19660800)

journal-block-count(5)

341syslog-ng.com

log-fifo-size()

);

};

mark() (DEPRECATED)number (seconds)Accepted values:1200Default:

Description: The mark-freq() option is an alias for the deprecated mark() option. This is retained forcompatibility with syslog-ng version 1.6.x.

mark-freq()number (seconds)Accepted values:1200Default:

Description: An alias for the obsolete mark() option, retained for compatibility with syslog-ng version 1.6.x.The number of seconds between two MARKmessages. MARKmessages are generated when there was no messagetraffic to inform the receiver that the connection is still alive. If set to zero (0), no MARK messages are sent. Themark-freq() can be set for global option and/or every MARK capable destination driver if mark-mode() isperiodical or dst-idle or host-idle. If mark-freq() is not defined in the destination, then the mark-freq()will be inherited from the global options. If the destination uses internal mark-mode(), then the globalmark-freq() will be valid (does not matter what mark-freq() set in the destination side).

mark-mode()internal | dst-idle | host-idle | periodical | none | globalAccepted values:internal for pipe, program drivers

none for file, unix-dgram, unix-stream drivers

global for syslog, tcp, udp destinations

host-idle for global option

Default:

Description: The mark-mode() option can be set for the following destination drivers: file(), program(),unix-dgram(), unix-stream(), network(), pipe(), syslog() and in global option.

■ internal: When internal mark mode is selected, internal source should be placed in the log pathas this mode does not generate mark by itself at the destination. This mode only yields the markmessages from internal source. This is the mode as syslog-ng PE 3.x worked. MARKwill be generatedby internal source if there was NO traffic on local sources:

file(), pipe(), unix-stream(), unix-dgram(), program()

■ dst-idle: Sends MARK signal if there was NO traffic on destination drivers. MARK signal frominternal source will be dropped.

342syslog-ng.com

mark() (DEPRECATED)

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ host-idle: Sends MARK signal if there was NO local message on destination drivers. For exampleMARK is generated even if messages were received from tcp. MARK signal from internal source willbe dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ periodical: Sends MARK signal perodically, regardless of traffic on destination driver. MARK signalfrom internal source will be dropped.

MARK signal can be sent by the following destination drivers: network(), syslog(), program(),file(), pipe(), unix-stream(), unix-dgram().

■ none: Destination driver drops all MARK messages. If an explicit mark-mode() is not given to thedrivers where none is the default value, then none will be used.

■ global: Destination driver uses the global mark-mode() setting. The syslog-ng interprets syntaxerror if the global mark-mode() is global.

NoteIn case of dst-idle, host-idle and periodical, the MARK message will not be written in the destination, if it is notopen yet.

Available in syslog-ng PE 4 LTS and later.

normalize-hostnames()yes | noAccepted values:noDefault:

Description: If enabled (normalize-hostnames(yes)), syslog-ng PE converts the hostnames to lowercase.

NoteThis setting applies only to hostnames resolved from DNS. It has no effect if the keep-hostname() option is enabled,and the message contains a hostname.

on-error()drop-message|drop-property|fallback-to-string|silently-drop-message|silently-drop-property|silently-fallback-to-stringAccepted values:drop-messageDefault:

343syslog-ng.com

normalize-hostnames()

Description: Controls what happens when type-casting fails and syslog-ng PE cannot convert some data to thespecified type. By default, syslog-ng PE drops the entire message and logs the error. Currently thevalue-pairs() option uses the settings of on-error().

■ drop-message: Drop the entire message and log an error message to the internal() source. Thisis the default behavior of syslog-ng PE.

■ drop-property: Omit the affected property (macro, template, or message-field) from the logmessage and log an error message to the internal() source.

■ fallback-to-string: Convert the property to string and log an error message to the internal()source.

■ silently-drop-message: Drop the entire message silently, without logging the error.

■ silently-drop-property: Omit the affected property (macro, template, or message-field) silently,without logging the error.

■ silently-fallback-to-string: Convert the property to string silently, without logging theerror.

owner()useridAccepted values:rootDefault:

Description: The default owner of output files. By default, syslog-ng changes the privileges of accessed files(for example /dev/null) to root.root 0600. To disable modifying privileges, use this option with the -1value.

perm()permission valueAccepted values:0600Default:

Description: The default permission for output files. By default, syslog-ng changes the privileges of accessedfiles (for example /dev/null) to root.root 0600. To disable modifying privileges, use this option withthe -1 value.

proto-template()name of a templateAccepted values:The default message format of the used protocolDefault:

Description: Specifies a template that protocol-like destinations (for example, network() and syslog()) use bydefault. For example:

344syslog-ng.com

owner()

template t_isostamp { template("$ISODATE $HOST $MSGHDR$MSG\n"); };

options { proto-template(t_isostamp); };

recv-time-zone()name of the timezone, or the timezone offsetAccepted values:local timezoneDefault:

Description: Specifies the time zone associated with the incoming messages, if not specified otherwise in themessage or in the source driver. For details, see also Section 2.5, Timezones and daylight saving (p. 13) andSection 2.5.2, A note on timezones and timestamps (p. 14).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

send-time-zone()name of the timezone, or the timezone offsetAccepted values:local timezoneDefault:

Description: Specifies the time zone associated with the messages sent by syslog-ng, if not specified otherwisein the message or in the destination driver. For details, see Section 2.5, Timezones and daylight saving (p. 13).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

stats-freq()number (seconds)Accepted values:600Default:

Description: The period between two STATS messages in seconds. STATS are log messages sent by syslog-ng,containing statistics about dropped log messages. Set to 0 to disable the STATS messages.

stats-level()0 | 1 | 2 | 3Accepted values:0Default:

Description: Specifies the detail of statistics syslog-ng collects about the processed messages.

■ Level 0 collects only statistics about the sources and destinations

345syslog-ng.com

recv-time-zone()

■ Level 1 contains details about the different connections and log files, but has a slight memory overhead

■ Level 2 contains detailed statistics based on the hostname.

■ Level 3 contains detailed statistics based on various message parameters like facility, severity, ortags.

Note that level 2 and 3 increase the memory requirements and CPU load. For details on message statistics, seeChapter 17, Statistics and metrics of syslog-ng (p. 446).

sync() or sync-freq() (DEPRECATED)number (messages)Accepted values:0Default:

Description: Obsolete aliases for flush-lines()

threaded()yes|noAccepted values:yesDefault:

Description: Enable syslog-ng PE to run in multithreaded mode and use multiple CPUs. Available only insyslog-ng Premium Edition 4 F1 and later. SeeChapter 18, Multithreading and scaling in syslog-ng PE (p. 449)for details.

time-reap()number (seconds)Accepted values:60Default:

Description: The time to wait in seconds before an idle destination file is closed. Note that only destinationfiles having macros in their filenames are closed automatically.

time-reopen()number (seconds)Accepted values:60Default:

Description: The time to wait in seconds before a dead connection is reestablished.

time-sleep() (DEPRECATED)number (milliseconds)Accepted values:0Default:

Description: The time to wait in milliseconds between each invocation of the poll() iteration.

346syslog-ng.com

sync() or sync-freq() (DEPRECATED)

timestamp-freq()number (seconds)Type:0Default:

Description: The minimum time (in seconds) that should expire between two timestamping requests. Whensyslog-ng closes a chunk, it checks how much time has expired since the last timestamping request: if it ishigher than the value set in the timestamp-freq() parameter, it requests a new timestamp from the authorityset in the timestamp-url() parameter.

By default, timestamping is disabled: the timestamp-freq() global option is set to 0. To enable timestamping,set it to a positive value.

timestamp-url()stringAccepted values:

Default:

Description: The URL of the Timestamping Authority used to request timestamps to sign logstore chunks.Note that syslog-ng PE currently supports only Timestamping Authorities that conform to RFC3161 InternetX.509 Public Key Infrastructure Time-Stamp Protocol, other protocols likeMicrosoft Authenticode Timestampingare not supported.

timestamp-policy()stringAccepted values:

Default:

Description: If the Timestamping Server has timestamping policies configured, specify the OID of the policyto use into the Timestamping policy field. syslog-ng PE will include this ID in the timestamping requests sentto the TSA. This option is available in syslog-ng PE 3.1 and later.

time-zone()name of the timezone, or the timezone offsetType:unspecifiedDefault:

Description: Convert timestamps to the timezone specified by this option. If this option is not set, then theoriginal timezone information in the message is used. Converting the timezone changes the values of alldate-related macros derived from the timestamp, for example, HOUR. For the complete list of such macros, seeSection 14.1.3, Date-related macros (p. 381).

The timezone can be specified as using the name of the (for example time-zone("Europe/Budapest")),or as the timezone offset in +/-HH:MM format (for example +01:00). On Linux and UNIX platforms, the validtimezone names are listed under the /usr/share/zoneinfo directory.

347syslog-ng.com

timestamp-freq()

ts-format()rfc3164 | bsd | rfc3339 | isoAccepted values:rfc3164Default:

Description: Specifies the timestamp format used when syslog-ng itself formats a timestamp and nothing elsespecifies a format (for example: STAMP macros, internal messages, messages without original timestamps). Fordetails, see also Section 2.5.2, A note on timezones and timestamps (p. 14).

By default, timestamps include only seconds. To include fractions of a second (for example, milliseconds) usethe frac-digits() option. For details, see Section frac-digits() (p. 339).

NoteThis option applies only to file and file-like destinations. Destinations that use specific protocols (for example, network(),or syslog()) ignore this option. For protocol-like destinations, use a template locally in the destination, or use theproto-template option.

use-dns()yes, no, persist_onlyType:yesDefault:

Description: Enable or disable DNS usage. The persist_only option attempts to resolve hostnames locallyfrom file (for example from /etc/hosts). The syslog-ng PE application blocks on DNS queries, so enablingDNS may lead to a Denial of Service attack. To prevent DoS, protect your syslog-ng network endpoint withfirewall rules, and make sure that all hosts which may get to syslog-ng are resolvable. This option can bespecified globally, and per-source as well. The local setting of the source overrides the global option if available.

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

use-fqdn()yes or noType:noDefault:

Description: Add Fully Qualified Domain Name instead of short hostname. This option can be specifiedglobally, and per-source as well. The local setting of the source overrides the global option if available.

NoteThis option has no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message containsa hostname.

348syslog-ng.com

ts-format()

use-rcptid() (DEPRECATED)yes | noAccepted values:noDefault:

Description: The receipt ID function is disabled by default due to performance issues. For details, see alsoSection RCPTID (p. 388). This function is now deprecated. Use the use-uniqid() option instead, for details,see Section use-uniqid() (p. 349).

use-time-recvd() (DEPRECATED)yes | noAccepted values:noDefault:

WarningThis option is not available in syslog-ng PE version 3.2 and later. Use the R_ prefixed version of the respective macroinstead. Starting with syslog-ng PE version 3.2, the DATE macro equals the S_DATE macro.

Description: This option controls how the time related macros are expanded in filename and content templates.If set to yes, then the non-prefixed versions of the time related macros (for example: HOUR instead of R_HOURand S_HOUR) refer to the time when the message was received, otherwise it refers to the timestamp which is inthe message.

NoteThe timestamps in the messages are generated by the originating host and might not be accurate.

This option is deprecated as many users assumed that it controls the timestamp as it is written tologfiles/destinations, which is not the case. To change how messages are formatted, specify a content-templatereferring to the appropriate prefixed (S_ or R_) time macro.

use-uniqid()yes | noAccepted values:noDefault:

Description: This option enables generating a globally unique ID. It is generated from the HOSTID and theRCPTID in the format of HOSTID@RCPTID. It has a fixed length: 16+@+8 characters. You can include theunique ID in the message by using the macro. For details, see Section UNIQID (p. 391).

Enabling this option automatically generates the HOSTID. The HOSTID is a persistent, 32-bits-longcryptographically secure pseudo random number, that belongs to the host that the syslog-ng is running on. Ifthe persist file is damaged, the HOSTID might change.

Enabling this option automatically enables the RCPTID functionality. For details, see Section RCPTID (p. 388)

349syslog-ng.com

use-rcptid() (DEPRECATED)

Chapter 10. TLS-encrypted message transfer

10.1. Secure logging using TLS

The syslog-ng application can send and receive log messages securely over the network using the TransportLayer Security (TLS) protocol using the network() and syslog() drivers.

TLS uses certificates to authenticate and encrypt the communication, as illustrated on the following figure:

Figure 10.1. Certificate-based authentication

The client authenticates the server by requesting its certificate and public key. Optionally, the server can alsorequest a certificate from the client, thus mutual authentication is also possible.

In order to use TLS encryption in syslog-ng, the following elements are required:

■ A certificate on the syslog-ng server that identifies the syslog-ng server.

■ The certificate of the Certificate Authority that issued the certificate of the syslog-ng server (or theself-signed certificate of the syslog-ng server) must be available on the syslog-ng client.

When using mutual authentication to verify the identity of the clients, the following elements are required:

■ A certificate must be available on the syslog-ng client. This certificate identifies the syslog-ng client.

■ The certificate of the Certificate Authority that issued the certificate of the syslog-ng client must beavailable on the syslog-ng server.

Mutual authentication ensures that the syslog-ng server accepts log messages only from authorized clients.

For details on configuring TLS communication in syslog-ng, see Section 10.2, Encrypting log messages withTLS (p. 351).

350syslog-ng.com

Secure logging using TLS

10.2. Encrypting log messages with TLS

This section describes how to configure TLS encryption in syslog-ng. For the concepts of using TLS in syslog-ng,see Section 10.1, Secure logging using TLS (p. 350).

Create an X.509 certificate for the syslog-ng server.

NoteThe subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty)of the server's certificate must contain the hostname or the IP address (as resolved from the syslog-ng clients and relays)of the server (for example syslog-ng.example.com).

Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, for example*.example.com.

Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IPaddress in the subject_alt_name parameter.

10.2.1. Procedure – Configuring TLS on the syslog-ng clients

Purpose:

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacyBSD-syslog protocol (using the network() driver) and the new IETF-syslog protocol standard (using thesyslog() driver):

Steps:

Step 1. Copy the CA certificate (for example cacert.pem) of the Certificate Authority that issued the certificateof the syslog-ng server (or the self-signed certificate of the syslog-ng server) to the syslog-ng clienthosts, for example into the /opt/syslog-ng/etc/syslog-ng/ca.d directory.Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem

The result is a hash (for example 6d2962a8), a series of alphanumeric characters based on theDistinguished Name of the certificate.

Note that by default, syslog-ng PE expects SHA-1 hashes. If you want to use MD5 hashes for somereason, use the ca_dir_layout(md5-based) option in your configuration (for details, see Sectionca-dir-layout() (DEPRECATED) (p. 357)).

Issue the following command to create a symbolic link to the certificate that uses the hash returned bythe previous command and the .0 suffix.

ln -s cacert.pem 6d2962a8.0

On Microsoft Windows, use the following command: mklink 6d2962a8.0 cacert.pem

Step 2. Add a destination statement to the syslog-ng configuration file that uses the tls(

ca-dir(path_to_ca_directory) ) option and specify the directory using the CA certificate. Thedestination must use the network() or the syslog() destination driver, and the IP address and portparameters of the driver must point to the syslog-ng server.

351syslog-ng.com

Encrypting log messages with TLS

Example 10.1. A destination statement using TLSThe following destination encrypts the log messages using TLS and sends them to the 6514/TCP port of thesyslog-ng server having the 10.1.2.3 IP address.

destination demo_tls_destination {

network("10.1.2.3" port(6514)

transport("tls")

tls( ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d"))

);

};

A similar statement using the IETF-syslog protocol and thus the syslog() driver:

destination demo_tls_syslog_destination {

syslog("10.1.2.3" port(6514)

transport("tls")

tls(ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d"))

);

};

Step 3. Include the destination created in Step 2 in a log statement.

WarningThe encrypted connection between the server and the client fails if the Common Name or thesubject_alt_name parameter of the server certificate does not contain the hostname or the IP address (asresolved from the syslog-ng clients and relays) of the server.

Do not forget to update the certificate files when they expire.

10.2.2. Procedure – Configuring TLS on the syslog-ng server

Purpose:

Complete the following steps on the syslog-ng server:

Steps:

Step 1. Copy the certificate (for example syslog-ng.cert) of the syslog-ng server to the syslog-ng serverhost, for example into the /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificatemust be a valid X.509 certificate in PEM format.

Step 2. Copy the private key (for example syslog-ng.key) matching the certificate of the syslog-ng serverto the syslog-ng server host, for example into the /opt/syslog-ng/etc/syslog-ng/key.d

directory. The key must be in PEM format, and must not be password-protected.

Step 3. Add a source statement to the syslog-ng configuration file that uses the tls(

key-file(key_file_fullpathname) cert-file(cert_file_fullpathname) ) option andspecify the key and certificate files. The source must use the source driver (network() or syslog())matching the destination driver used by the syslog-ng client.

Example 10.2. A source statement using TLSThe following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of anyinterface of the syslog-ng server.

source demo_tls_source {

network(ip(0.0.0.0) port(1999)

352syslog-ng.com

Encrypting log messages with TLS

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert"))

);

};

A similar source for receiving messages using the IETF-syslog protocol:

source demo_tls_syslog_source {

syslog(ip(0.0.0.0) port(1999)

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert"))

);

};

Step 4. Disable mutual authentication for the source by setting the following TLS option in the source statement:tls( peer-verify(optional-untrusted);

For details on how to configure mutual authentication, see Section 10.3, Mutual authentication usingTLS (p. 353).

For the details of the available tls() options, see Section 10.4, TLS options (p. 356).

Example 10.3. Disabling mutual authenticationThe following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of anyinterface of the syslog-ng server. The identity of the syslog-ng client is not verified.

source demo_tls_source {

network(ip(0.0.0.0) port(1999)

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")

peer-verify(optional-untrusted))

);

};

A similar source for receiving messages using the IETF-syslog protocol:

source demo_tls_syslog_source {

syslog(ip(0.0.0.0) port(1999)

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")

peer-verify(optional-untrusted))

);

};

WarningDo not forget to update the certificate and key files when they expire.

10.3. Mutual authentication using TLS

This section describes how to configure mutual authentication between the syslog-ng server and the client.Configuring mutual authentication is similar to configuring TLS (for details, see Section 10.2, Encrypting log

353syslog-ng.com

Mutual authentication using TLS

messages with TLS (p. 351)), but the server verifies the identity of the client as well. Therefore, each client musthave a certificate, and the server must have the certificate of the CA that issued the certificate of the clients.For the concepts of using TLS in syslog-ng, see Section 10.1, Secure logging using TLS (p. 350).

10.3.1. Procedure – Configuring TLS on the syslog-ng clients

Purpose:

Complete the following steps on every syslog-ng client host. Examples are provided using both the legacyBSD-syslog protocol (using the network() driver) and the new IETF-syslog protocol standard (using thesyslog() driver):

Steps:

Step 1. Create an X.509 certificate for the syslog-ng client.

Step 2. Copy the certificate (for example client_cert.pem) and the matching private key (for exampleclient.key) to the syslog-ng client host, for example into the/opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificate must be a valid X.509certificate in PEM format and must not be password-protected.

Step 3. Copy the CA certificate of the Certificate Authority (for example cacert.pem) that issued the certificateof the syslog-ng server (or the self-signed certificate of the syslog-ng server) to the syslog-ng clienthosts, for example into the /opt/syslog-ng/etc/syslog-ng/ca.d directory.Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem

The result is a hash (for example 6d2962a8), a series of alphanumeric characters based on theDistinguished Name of the certificate.

Note that by default, syslog-ng PE expects SHA-1 hashes. If you want to use MD5 hashes for somereason, use the ca_dir_layout(md5-based) option in your configuration (for details, see Sectionca-dir-layout() (DEPRECATED) (p. 357)).

Issue the following command to create a symbolic link to the certificate that uses the hash returned bythe previous command and the .0 suffix.

ln -s cacert.pem 6d2962a8.0

On Microsoft Windows, use the following command: mklink 6d2962a8.0 cacert.pem

Step 4. Add a destination statement to the syslog-ng configuration file that uses the tls(

ca-dir(path_to_ca_directory) ) option and specify the directory using the CA certificate. Thedestination must use the network() or the syslog() destination driver, and the IP address and portparameters of the driver must point to the syslog-ng server. Include the client's certificate and privatekey in the tls() options.

Example 10.4. A destination statement using mutual authenticationThe following destination encrypts the log messages using TLS and sends them to the 1999/TCP port of thesyslog-ng server having the 10.1.2.3 IP address. The private key and the certificate file authenticating theclient is also specified.

destination demo_tls_destination {

network("10.1.2.3" port(1999)

transport("tls")

tls( ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d")

354syslog-ng.com

Mutual authentication using TLS

key-file("/opt/syslog-ng/etc/syslog-ng/key.d/client.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem"))

); };

destination demo_tls_syslog_destination {

syslog("10.1.2.3" port(1999)

transport("tls")

tls( ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d")

key-file("/opt/syslog-ng/etc/syslog-ng/key.d/client.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/client_cert.pem"))

); };

Step 5. Include the destination created in Step 2 in a log statement.

WarningThe encrypted connection between the server and the client fails if the Common Name or thesubject_alt_name parameter of the server certificate does not the hostname or the IP address (as resolvedfrom the syslog-ng clients and relays) of the server.

Do not forget to update the certificate files when they expire.

10.3.2. Procedure – Configuring TLS on the syslog-ng server

Purpose:

Complete the following steps on the syslog-ng server:

Steps:

Step 1. Copy the certificate (for example syslog-ng.cert) of the syslog-ng server to the syslog-ng serverhost, for example into the /opt/syslog-ng/etc/syslog-ng/cert.d directory. The certificatemust be a valid X.509 certificate in PEM format.

Step 2. Copy the CA certificate (for example cacert.pem) of the Certificate Authority that issued the certificateof the syslog-ng clients to the syslog-ng server, for example into the/opt/syslog-ng/etc/syslog-ng/ca.d directory.Issue the following command on the certificate: openssl x509 -noout -hash -in cacert.pem

The result is a hash (for example 6d2962a8), a series of alphanumeric characters based on theDistinguished Name of the certificate.

Note that by default, syslog-ng PE expects SHA-1 hashes. If you want to use MD5 hashes for somereason, use the ca_dir_layout(md5-based) option in your configuration (for details, see Sectionca-dir-layout() (DEPRECATED) (p. 357)).

Issue the following command to create a symbolic link to the certificate that uses the hash returned bythe previous command and the .0 suffix.

ln -s cacert.pem 6d2962a8.0

On Microsoft Windows, use the following command: mklink 6d2962a8.0 cacert.pem

355syslog-ng.com

Mutual authentication using TLS

Step 3. Copy the private key (for example syslog-ng.key) matching the certificate of the syslog-ng serverto the syslog-ng server host, for example into the /opt/syslog-ng/etc/syslog-ng/key.d

directory. The key must be in PEM format, and must not be password-protected.

Step 4. Add a source statement to the syslog-ng configuration file that uses the tls(

key-file(key_file_fullpathname) cert-file(cert_file_fullpathname) ) option andspecify the key and certificate files. The source must use the source driver (network() or syslog())matching the destination driver used by the syslog-ng client. Also specify the directory storing thecertificate of the CA that issued the client's certificate.

For the details of the available tls() options, see Section 10.4, TLS options (p. 356).

Example 10.5. A source statement using TLSThe following source receives log messages encrypted using TLS, arriving to the 1999/TCP port of anyinterface of the syslog-ng server.

source demo_tls_source {

network(ip(0.0.0.0) port(1999)

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")

ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ); };

A similar source for receiving messages using the IETF-syslog protocol:

source demo_tls_syslog_source {

syslog(ip(0.0.0.0) port(1999)

transport("tls")

tls( key-file("/opt/syslog-ng/etc/syslog-ng/key.d/syslog-ng.key")

cert-file("/opt/syslog-ng/etc/syslog-ng/cert.d/syslog-ng.cert")

ca-dir("/opt/syslog-ng/etc/syslog-ng/ca.d")) ); };

WarningDo not forget to update the certificate and key files when they expire.

10.4. TLS options

The syslog-ng application can encrypt incoming and outgoing syslog message flows using TLS if you use thenetwork() or syslog() drivers.

NoteThe format of the TLS connections used by syslog-ng is similar to using syslog-ng and stunnel, but the source IP informationis not lost.

To encrypt connections, use the transport("tls") and tls() options in the source and destination statements.

The tls() option can include the following settings:

356syslog-ng.com

TLS options

allow-compress()yes | noAccepted values:noDefault:

Description: Enable on-the-wire compression in TLS communication. Note that this option must be enabledboth on the server and the client side to have any effect. Enabling compression can significantly reduce thebandwidth required to transport the messages, but can slightly decrease the performance of syslog-ng PE,reducing the number of transferred messages during a given period.

ca-dir()Directory nameAccepted values:noneDefault:

Description:Name of a directory, that contains a set of trusted CA certificates in PEM format. The CA certificatefiles have to be named after the 32-bit hash of the subject's name. This naming can be created using the c_rehashutility in openssl.

ca-dir-layout() (DEPRECATED)sha1-basedAccepted values:sha1-basedDefault:

Description: The type of the hash used for the CA certificates. NOTE: This option is deprecated.

WarningIf you are upgrading to syslog-ng PE version 6.x from a version earlier than 5.0, you must rehash the trusted CA certificates.

cert-file()FilenameAccepted values:noneDefault:

Description: Name of a file, that contains an X.509 certificate (or a certificate chain) in PEM format, suitableas a TLS certificate, matching the private key. If the file contains a certificate chain, the file must begin withthe certificate of the host, followed by the CA certificate that signed the certificate of the host, and any othersigning CAs in order.

cert-subject()stringAccepted values:noneDefault:

Description:Only available on Windows systems. The subject of the certificate you wish to use for authenticatingthe local endpoint of the TLS connection. If set, syslog-ng PE will retrieve the certificate that has the specified

357syslog-ng.com

allow-compress()

subject from the Local Computer > Personal certificate store. Therefore, if this option is used, first import therespective certificate and the corresponding private key to the appropriate certificate store as a .pfx file.

To get the format for the cert-subject() string right, check the certificate's subject first. Concatenate thedifferent key-value pairs in the subject, going from bottom to top, omitting the keys and equals signs, and addingcommas for newline characters.

In the example provided in Figure 10.2, Getting details of the certificate subject for the cert-subject()string (p. 358), the cert-subject() string would look like this:

cert-subject("HU, Some-State, Budapest, Balabit, syslog-ng team, client")

Figure 10.2. Getting details of the certificate subject for the cert-subject() string

cipher-suite()Name of a cipher, or a colon-separated listAccepted values:aes-128-cbcDefault:

Description: Specifies the cipher, hash, and key-exchange algorithms used for the encryption, for example,ECDHE-ECDSA-AES256-SHA384. The list of available algorithms depends on the version of OpenSSL usedto compile syslog-ng PE. To specify multiple ciphers, separate the cipher names with a colon, and enclose thelist between double-quotes, for example:

cipher-suite("ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384")

The cipher-suite() option also determines the encryption protocol used in the connection: to disable SSLv3,use an algorithm that is available only in TLSv1.2, and that both the client and the server supports.

358syslog-ng.com

cipher-suite()

■ Supports TLS v1.2: ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-SHA384, ECDHE-ECDSA-AES256-SHA384, DH-DSS-AES256-GCM-SHA384,DHE-DSS-AES256-GCM-SHA384,DH-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-GCM-SHA384,DHE-RSA-AES256-SHA256, DHE-DSS-AES256-SHA256, DH-RSA-AES256-SHA256,DH-DSS-AES256-SHA256, ADH-AES256-GCM-SHA384, ADH-AES256-SHA256,ECDH-RSA-AES256-GCM-SHA384, ECDH-ECDSA-AES256-GCM-SHA384,ECDH-RSA-AES256-SHA384, ECDH-ECDSA-AES256-SHA384, AES256-GCM-SHA384,AES256-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-SHA256, ECDHE-ECDSA-AES128-SHA256, DH-DSS-AES128-GCM-SHA256,DHE-DSS-AES128-GCM-SHA256,DH-RSA-AES128-GCM-SHA256,DHE-RSA-AES128-GCM-SHA256,DHE-RSA-AES128-SHA256, DHE-DSS-AES128-SHA256, DH-RSA-AES128-SHA256,DH-DSS-AES128-SHA256, ADH-AES128-GCM-SHA256, ADH-AES128-SHA256,ECDH-RSA-AES128-GCM-SHA256, ECDH-ECDSA-AES128-GCM-SHA256,ECDH-RSA-AES128-SHA256, ECDH-ECDSA-AES128-SHA256, AES128-GCM-SHA256,AES128-SHA256, NULL-SHA256

■ Supports SSL v3: ECDHE-RSA-AES256-SHA, ECDHE-ECDSA-AES256-SHA,SRP-DSS-AES-256-CBC-SHA, SRP-RSA-AES-256-CBC-SHA, SRP-AES-256-CBC-SHA,DHE-RSA-AES256-SHA, DHE-DSS-AES256-SHA, DH-RSA-AES256-SHA, DH-DSS-AES256-SHA,DHE-RSA-CAMELLIA256-SHA, DHE-DSS-CAMELLIA256-SHA, DH-RSA-CAMELLIA256-SHA,DH-DSS-CAMELLIA256-SHA, AECDH-AES256-SHA, ADH-AES256-SHA, ADH-CAMELLIA256-SHA,ECDH-RSA-AES256-SHA, ECDH-ECDSA-AES256-SHA, AES256-SHA, CAMELLIA256-SHA,PSK-AES256-CBC-SHA, ECDHE-RSA-AES128-SHA, ECDHE-ECDSA-AES128-SHA,SRP-DSS-AES-128-CBC-SHA, SRP-RSA-AES-128-CBC-SHA, SRP-AES-128-CBC-SHA,DHE-RSA-AES128-SHA, DHE-DSS-AES128-SHA, DH-RSA-AES128-SHA, DH-DSS-AES128-SHA,DHE-RSA-SEED-SHA, DHE-DSS-SEED-SHA, DH-RSA-SEED-SHA, DH-DSS-SEED-SHA,DHE-RSA-CAMELLIA128-SHA, DHE-DSS-CAMELLIA128-SHA, DH-RSA-CAMELLIA128-SHA,DH-DSS-CAMELLIA128-SHA, AECDH-AES128-SHA, ADH-AES128-SHA, ADH-SEED-SHA,ADH-CAMELLIA128-SHA, ECDH-RSA-AES128-SHA, ECDH-ECDSA-AES128-SHA, AES128-SHA,SEED-SHA, CAMELLIA128-SHA, PSK-AES128-CBC-SHA, ECDHE-RSA-RC4-SHA,ECDHE-ECDSA-RC4-SHA, AECDH-RC4-SHA, ADH-RC4-MD5, ECDH-RSA-RC4-SHA,ECDH-ECDSA-RC4-SHA, RC4-SHA, RC4-MD5, PSK-RC4-SHA, ECDHE-RSA-DES-CBC3-SHA,ECDHE-ECDSA-DES-CBC3-SHA, SRP-DSS-3DES-EDE-CBC-SHA, SRP-RSA-3DES-EDE-CBC-SHA,SRP-3DES-EDE-CBC-SHA, EDH-RSA-DES-CBC3-SHA, EDH-DSS-DES-CBC3-SHA,DH-RSA-DES-CBC3-SHA, DH-DSS-DES-CBC3-SHA, AECDH-DES-CBC3-SHA, ADH-DES-CBC3-SHA,ECDH-RSA-DES-CBC3-SHA, ECDH-ECDSA-DES-CBC3-SHA, DES-CBC3-SHA,PSK-3DES-EDE-CBC-SHA, ECDHE-RSA-NULL-SHA, ECDHE-ECDSA-NULL-SHA, AECDH-NULL-SHA,ECDH-RSA-NULL-SHA, ECDH-ECDSA-NULL-SHA, NULL-SHA, NULL-MD5

crl-dir()Directory nameAccepted values:noneDefault:

Description: Name of a directory that contains the Certificate Revocation Lists for trusted CAs. Similarly toca-dir() files, use the 32-bit hash of the name of the issuing CAs as filenames. The extension of the filesmust be .r0.

359syslog-ng.com

crl-dir()

If the crl-dir() is set, and the peer certificate has been revoked, syslog-ng PE rejects the connection. If thepeer certificate has not been revoked, or syslog-ng PE cannot access the CRL, syslog-ng PE accepts theconnection.

curve-list()string (colon-separated list)Accepted values:noneDefault:

Description: A colon-separated list that specifies the curves that are permitted in the connection when usingElliptic Curve Cryptography (ECC). The syslog-ng PE application uses automatically the highest preferencecurve that both peers support. If not specified, the list includes every supported curve. For example:

curve-list('prime256v1:secp521r1')

The syslog-ng Premium Edition application currently supports the following curves: sect163k1, sect163r1,sect163r2, sect193r1, sect193r2,, sect233k1, sect233r1, sect239k1, sect283k1, sect283r1,,sect409k1, sect409r1, sect571k1, sect571r1, secp160k1,, secp160r1, secp160r2, secp192k1,prime192v1, secp224k1,, secp224r1, secp256k1, prime256v1, secp384r1, secp521r1,,brainpoolP256r1, brainpoolP384r1, brainpoolP512r1.

dhparam-file()string (filename)Accepted values:noneDefault:

Description: Specifies a file containing Diffie-Hellman parameters, generated using the openssl dhparam

utility. Note that syslog-ng PE supports only DH parameter files in the PEM format. If you do not set thisparameter, syslog-ng PE uses the 2048-bit MODP Group, as described in RFC 3526.

key-file()FilenameAccepted values:noneDefault:

Description: Name of a file, that contains an unencrypted private key in PEM format, suitable as a TLS key.

peer-verify()optional-trusted | optional-untrusted | required-trusted |required-untrusted

Accepted values:

required-trustedDefault:

Description: Verification method of the peer, the four possible values is a combination of two properties ofvalidation:

■ whether the peer is required to provide a certificate (required or optional prefix), and

■ whether the certificate provided needs to be valid or not.

The following table summarizes the possible options and their results depending on the certificate of the peer.

360syslog-ng.com

curve-list()

The remote peer has:valid certificateinvalid certificateno certificate

TLS-encryptionTLS-encryptionTLS-encryptionoptional-untrusted

Local peer-verify()setting

TLS-encryptionrejected connectionTLS-encryptionoptional-trustedTLS-encryptionTLS-encryptionrejected connectionrequired-untrustedTLS-encryptionrejected connectionrejected connectionrequired-trusted

For untrusted certificates only the existence of the certificate is checked, but it does not have to be valid —syslog-ng accepts the certificate even if it is expired, signed by an unknown CA, or its CN and the name of themachine mismatches.

WarningWhen validating a certificate, the entire certificate chain must be valid, including the CA certificate. If any certificate ofthe chain is invalid, syslog-ng PE will reject the connection.

trusted-dn()list of accepted distinguished namesAccepted values:noneDefault:

Description: To accept connections only from hosts using certain certificates signed by the trusted CAs, listthe distinguished names of the accepted certificates in this parameter. For example using trusted-dn("*,

O=Example Inc, ST=Some-State, C=*") will accept only certificates issued for the Example Inc

organization in Some-State state.

trusted-keys()list of accepted SHA-1 fingerprintsAccepted values:noneDefault:

Description:To accept connections only from hosts using certain certificates having specific SHA-1 fingerprints,list the fingerprints of the accepted certificates in this parameter. For exampletrusted-keys("SHA1:00:EF:ED:A4:CE:00:D1:14:A4:AB:43:00:EF:00:91:85:FF:89:28:8F",

"SHA1:0C:42:00:3E:B2:60:36:64:00:E2:83:F0:80:46:AD:00:A8:9D:00:15").

To find the fingerprint of a certificate, you can use the following command: openssl x509 -in

<certificate-filename> -sha1 -noout -fingerprint

NoteWhen using the trusted-keys() and trusted-dn() parameters at the same time, note the following:

■ If the fingerprint of the peer is listed in the trusted-keys() parameter and the DN of the peer is listedin the trusted-dn() parameter, then the certificate validation is performed.

■ If either the fingerprint of the peer is not listed in the trusted-keys() parameter or the DN of the peeris not listed in the trusted-dn() parameter, then the authentication of the peer fails and the connectionis closed.

361syslog-ng.com

trusted-dn()

Chapter 11. FIPS-compliant syslog-ng

Starting from syslog-ng PE version 5.0.4, LTS versions are Federal Information Processing Standards(FIPS)-compliant. Note that only the LTS versions of syslog-ng PE are FIPS-compliant, feature releases arenot. For details on the versioning policy of syslog-ng PE see Section 2.6, Versions and releases of syslog-ngPE (p. 14).

11.1. Installing FIPS-compliant syslog-ng PE

For the FIPS-compliant version of syslog-ng PE, you are required to use the new license file. It uses SHA-512algorithm, and therefore is accepted by the FIPS-compliant version. The non-FIPS-compliant syslog-ng PEwill accept both the previously released license files and the new license files as well.

The following Linux platforms are supported:

■ debian-etch, debian-lenny, debian-squeeze

■ suse-10.1, sles-10.2, sles-11.0, sles-11.1

■ rhel-4, rhel-5, rhel-6

■ ubuntu-hardy, ubuntu-lucid

The FIPS-compliant version of syslog-ng PE is installed using a dedicated installer. The FIPS-compliantsyslog-ng PE installer will replace the existing non-FIPS-compliant syslog-ng PE and the tools shipped withit to its FIPS-compliant counterpart. For the supported plaforms both .run and native installers are available.The name of the FIPS-compliant syslog-ng PE installers includes the 'fips' specifier, for example:syslog-ng-premium-edition-fips-5.0.4-linux-glibc2.3.6-amd64.run.

Upgrading from a non-FIPS-compliant version of syslog-ng PE to a FIPS-compliant version is simple, in mostcases you can use your old configuration files. After starting the FIPS-compliant version of syslog-ng PE, itwill print those options in your configuration that cannot be used with the FIPS-compliant version of syslog-ngPE.

To check the version of syslog-ng PE, use the -V command line option. It will display version details, modules,and enabled options. In the FIPS-compliant version the Enable-FIPS option is set to on while in thenon-FIPS-compliant version it is set to off.

The startup message will also indicate this, the FIPS-compliant version of syslog-ng PE will display the following:FIPS-mode='enabled' .

11.2. Limitations of the FIPS-compliant syslog-ng PE

The FIPS-compliant syslog-ng PE is compatible with the non-FIPS-compliant version with the followinglimitations:

■ The current FIPS-compliant version can only be run on Linux platforms.

362syslog-ng.com

Installing FIPS-compliant syslog-ng PE

■ The FIPS-compliant version of syslog-ng PE uses only those cryptographic algorithms that areconsidered to be secure according to the FIPS-140 standard, therefore the support of some hashingand encryption algorithms have been removed (for example MD5 and DES).

■ Currently there is no database support in the FIPS-compliant version (neither as source nor asdestination).

■ The layout of the Certiface Authority directory (ca-dir) and the Certificate Revocation List directory(crl-dir) cannot be hashed with MD5. If the layout of your Certiface Authority directory (ca-dir) orthe Certificate Revocation List directory is hashed with MD5, you must rehash it with SHA-1 to beable to use it with the FIPS-compliant version of syslog-ng PE.

■ To use verified TLS/SSL communtication in the FIPS-compliant version of syslog-ng PE, you arerequired to use certificates that are using one of the SHA-1 or SHA-2 algorithms ('SHA-1', 'SHA-224','SHA-256', 'SHA-384' or 'SHA-512').

■ The FIPS-compliant version of syslog-ng PE is unable to use DES or MD5 during theSNMP-authentication.

■ To use logstore with the FIPS-compliant version of syslog-ng PE you are required to set the cipherto one of the AES-based algorithms and to set the digest to one of the SHA-1 or SHA-2 algorithms('SHA-1', 'SHA-224', 'SHA-256', 'SHA-384' or 'SHA-512').

■ With the FIPS-compliant version of syslog-ng PE, you will be unable to use the previously generatedlogstores that do not meet the abovementioned requirements. However, previously generated logstoresthat meet those requirements can be used.

For details on FIPS security functions, see Security Requirements for Cryptographic Modules.

11.3. Legal Notice of FIPS Compliance of Syslog-ng Premium Edition

Syslog-ng Premium Edition version 5.0.4 and the subsequent releases may claim FIPS compliance by use ofthe OpenSSL FIPS Object Module v2.0.5 cryptographic component. This component was certified by the OpenSource Software Institute and obtained the FIPS-140-2 validation Level 1 (certificate number 1747) from theU.S. National Institute of Standards and Technology (NIST) and the Communications Security EstablishmentCanada (CSEC).

The FIPS-compliant version of syslog-ng Premium Edition v5.0.4 and the subsequent releases exclusively usethe OpenSSL FIPS Object Module indicated above for crypthograpy. The build process of the FIPS-compliantversion of syslog-ng Premium Edition complies with the OpenSSL FIPS 140-2 Security Policy Version 2.0.5f.

363syslog-ng.com

Legal Notice of FIPS Compliance of Syslog-ng Premium Edition

Chapter 12. Reliable Log Transfer Protocol™

12.1. Logging using RLTP™

The syslog-ng PE application can send and receive log messages in a reliable way over the TCP transport layerusing the Reliable Log Transfer Protocol™ (RLTP™). RLTP™ is a proprietary transport protocol that preventsmessage loss during connection breaks. The transport is used between syslog-ng PE hosts (for example, a clientand a server, or a client-relay-server), and interoperates with the flow-control and reliable disk-buffer mechanismsof syslog-ng PE, thus providing the best way to prevent message loss. The sender detects which messages hasthe receiver successfully received. If messages are lost during the transfer, the sender resends the missingmessages, starting from the last successfully received message. Therefore, messages are not duplicated at thereceiving end in case of a connection break (however, in failover mode this is not completely ensured). RLTP™also allows to receive encrypted and non-encrypted connections on the same port, using a single source driver.

NoteBecause of the communication overhead, the RLTP™ protocol is slower than other transport protocols, which might bea problem if you need to collect a high amount (over 200000 messages per second) of log messages on your log server.For performance details of syslog-ng PE see the syslog-ng Premium Edition Performance Guideline at the syslog-ngDocumentation page.

NoteMake sure that you have set the value of the log_msg_size() parameter large enough in your configuration. If its sizeis less than the size of the sent messages, it might result in disk fill-up and no incoming logs.

WarningIn the following cases, it is possible to lose log messages even if you use RLTP™:

If you use RLTP™ together with non-reliable disk-buffer, it is possible to lose logs.■

■ When sending logs through a relay that is using a non-reliable disk-buffer, it is possible to lose logs if therelay crashes.

■ When sending logs through a relay that is using a non-reliable disk-buffer, it is possible that logs areduplicated if the relay crashes, or it is stopped.

■ If the underlying disk system of syslog-ng PE fails to write the log messages to the disk, but it does notreturn a write error, or some other hardware or operating-system error happens.

NoteTo minimize the chances of losing messages, it is recommended to set theflush-size() parameter of the receiver to 1 (however, note that dependingon your syslog-ng PE configuration, this can decrease the performance ofthe receiver).

The RLTP™ protocol works on top of TCP, and can use STARTTLS for encryption. RLTP™ supports IPv4and IPv6 addresses. Inside the RLTP™ message, the message can use any format, for example, RFC3164(BSD-syslog) or RFC5424 (IETF-syslog). The default port of RLTP™ is 35514.

364syslog-ng.com

Logging using RLTP™

RLTP™ can be added to the configuration like a transport protocol within the syslog() driver and thenetwork() driver.

12.1.1. Procedure – How RLTP™ connections work

Purpose:

This procedure summarizes how two syslog-ng PE hosts (a sender and a receiver) communicate using theReliable Log Transfer Protocol™ (RLTP™).

Prerequisites:

The sender (also called the client) is the host that has RLTP™ configured in its destination driver. The receiver(also called the server) is the host that has RLTP™ configured in its source driver.

Steps:

Step 1. The sender initiates the connection to the receiver.

Step 2. The sender and the receiver negotiate whether to encrypt the connection and to use compression ornot.

Step 3. If the connection should be encrypted, the sender and the receiver perform authentication (as configuredin the tls() options of their configuration).

Step 4. If the sender and the receiver have communicated earlier using RLTP™, the receiver indicates whichwas the last message received from the sender.

Step 5. The sender starts sending messages in batches. Batch size depends on the flush-lines() parameterof the sender.

For optimal performance when sending messages to an syslog-ng PE server, make sure that theflush-lines() is smaller than the window size set using the log-iw-size() option in the sourceof your server.

Step 6. When the receiver has successfully processed the messages in the batch, it sends an acknowledgementof the processed messages to the sender.What "successfully processed" means depends on the configuration of the receiver, for example, writtento disk in a destination, forwarded to a remote destination using not RLTP™, dropped because of filtersettings, or written to the disk-buffer. (If the messages are forwarded using RLTP™, see Section 12.1.2,Using RLTP™ in a client-relay-server scenario (p. 365).)

Step 7. After receiving the acknowledgement, the sender sends another batch of messages.

12.1.2. Using RLTP™ in a client-relay-server scenario

You can use RLTP™ between multiple syslog-ng PE hosts, for example, in a client-relay-server scenario. Insuch case, the communication described in Procedure 12.1.1, How RLTP™ connections work (p. 365) appliesboth between the client and the relay, and the relay and the server. However, note the following points:

■ Unless you use disk-buffer on the relay, the relay waits for acknowledgement from the server beforeacknowledging the messages to the client. If you send the messages in large batches, and the server

365syslog-ng.com

Using RLTP™ in a client-relay-server scenario

can process the messages slowly (or the network connection is slow), you might have to adjust themessage-acknowledgement-timeout() on the client.

■ If you use reliable disk-buffer on the relay, the relay will acknowledge the messages when themessages are written to the disk-buffer. That way, the client does not have to wait while the serveracknowledges the messages.

12.2. RLTP™ options

The following options are specific to the RLTP™ protocol. Note that when using RLTP™ in a source or adestination, the options of the syslog() or the network() driver can be used as well.

allow-compress()yes | noAccepted values:noDefault:

Description: Enable on-the-wire compression in the RLTP communication. Note that this option must beenabled both on the server and the client side to have any effect. Enabling compression can significantly reducethe bandwidth required to transport the messages, but can slightly decrease the performance of syslog-ng PE,reducing the number of transferred messages. The allow-compress() option can be used in source anddestination drivers as well. Available in syslog-ng PE 5.0 and later.

message-acknowledgement-timeout()number (seconds)Type:900Default:

Description: When the receiver (syslog-ng PE server) receives and successfully processes a message, it sendsan acknowledgement to the sender (the syslog-ng PE client). If the receiver does not acknowledge receivingthe messages within this period, the sender terminates the connection with the receiver. Use this option onlyin destination drivers.

response-timeout()number (seconds)Type:60Default:

Description: If syslog-ng PE does not receive any message (acknowledgement, or other message related to theRLTP™ protocol) in the given timeframe, syslog-ng PE terminates the connection with the peer, and the"Connection broken" message appears in the logs of the sender (the syslog-ng PE client). This is normal, andhappens when the sender does not send any new message to the receiver.

Under normal circumstances, you should not change the value of this option. The response-timeout()

option can be used in source and destination drivers as well.

366syslog-ng.com

RLTP™ options

tls-required()yes, optional, noType:optionalDefault:

Description: Determines whether STARTTLS is to be used during communication. If the option is set to yes,you must also configure the tls() option to specify other parameters of the TLS connection (for example, theauthentication of the server and the client).

The tls-required() option can be used in source and destination drivers as well.

For example, if you configure tls-required(yes) on server side and tls-required(no) on client side,the connection is dropped. If one of them is set to optional, the configuration of the other side will decide ifTLS is used or not. If both sides are set to optional, and the tls() option is properly configured, TLSencryption will be used. The following table summarizes the possible options and their results.

Note that the various parameters of the tls() option are considered in the connection only if thetls-required() settings of the peers result in TLS-encryption in the following table. In other words: thetls-required() option of RLTP™ determines if TLS should be used at all, while the peer-verify()

option of the tls()setting determines if the TLS connection can be actually established.

tls-required() setting on the serveroptionalnoyes

TLS-encryptionrejected connectionTLS-encryptionyes

tls-required() settingon the client

unencrypted connectionunencrypted connectionrejected connectionnoTLS-encryption if thetls() option is set,unencrypted connectionotherwise

unencrypted connectionTLS-encryptionoptional

Setting tls-required(optional) on your server allows you to receive both encrypted and unencryptedconnections on the same port.

12.3. Examples for using RLTP™

Example 12.1. Simple RLTP™ connectionThe sender and the receiver use RLTP™ over the network() protocol. Since the tls() option is not configured neitheron the sender nor on the receiver, the communication will be unencrypted.

Receiver configuration (syslog-ng PE server):

source s_network_rltp {

network(

ip("127.0.0.1")

port("5555")

transport(rltp)

ip-protocol(4)

);

};

Sender configuration (syslog-ng PE client):

367syslog-ng.com

tls-required()

destination d_network_rltp {

network(

"127.0.0.1"

port("5555")

transport(rltp)

ip-protocol(4)

);

};

Example 12.2. RLTP™ with TLS encryptionThe following example configure a sender and a receiver to communicate using RLTP™. Since the tls-required()option is set to optional on the receiver and yes on the sender, and the tls() option is configured, the communicationwill be TLS-encrypted. For the sender (syslog-ng PE client), reliable disk-buffering is enabled to prevent data loss.

Receiver configuration (syslog-ng PE server):

source s_syslog_rltp {

syslog(

ip("127.0.0.1")

port("4444")

transport(rltp(tls-required(optional)))

ip-protocol(4)

tls(

peer-verify(required-trusted)

ca-dir("/var/tmp/client/")

key-file("/var/tmp/server/server_priv.key")

cert-file("/var/tmp/server/server.crt")

)

);

};

Sender configuration (syslog-ng PE client):

destination d_syslog_rltp {

syslog(

"127.0.0.1"

port("4444")

transport(rltp(tls-required(yes)))

ip-protocol(4)

disk-buffer( mem-buf-size(200000) disk-buf-size(2000000) reliable(yes) )

tls(

peer-verify(required-trusted)

ca-dir("/var/tmp/server/")

key-file("/var/tmp/client/client_priv.key")

cert-file("/var/tmp/client/client.crt")

)

);

};

368syslog-ng.com

Examples for using RLTP™

Chapter 13. Reliability and minimizing the loss oflog messages

13.1. Introduction

Reliable Log Transfer Protocol™ (RLTP™) interacts with flow control and disk buffering to ensure that theloss of log messages is minimized or is prevented completely. This section explains how each loss preventionmethod contributes to reliability and minimizing log message loss. Flow control, disk buffering, and RLTP™are explained in detail elsewhere in the document. In this section, we present a high-level overview of all ofthese mechanisms and highlight considerations such as:

■ What best practices exist in various scenarios, how to set key parameters

■ When is a log message considered "delivered"

■ Under what circumstances can log loss occur

Each of the following sections discusses a different scenario and uses figures to aid comprehension.

NoteEach figure depicts a scenario in which the volume of incoming messages makes it necessary to use all buffers and controlwindows at maximum capacity.

Important information:

Any of the mechanisms that syslog-ng PE uses to prevent or minimize the loss of log messages only works ifthe hardware and operating system work normally. When there is an issue with the hardware or operating systemthat the application and syslog-ng PE run on, log loss may occur. Issues include operating system crash (forexample, kernel panic), memory errors, disk errors, power outage, and so on.

13.2. Flow control, no disk buffering, no RLTP™

How it works:

log-iw-size() sets a control window that tracks how many messages syslog-ng PE can accept. Every sourcehas its own control window. If the window gets full, syslog-ng PE stops reading messages from the sourcesuntil some messages are successfully sent to the destination(s).

369syslog-ng.com

Introduction

Figure 13.1. Flow control, no disk buffering, no RLTP™

How to set key parameters:

Set flags(flow-control) in the log path.

The output buffer must be large enough to store the incoming messages of every connected source:

log-fifo-size() > sum of log-iw-size() of sources connected to this destination

Benefits:

This configuration minimizes the loss of log messages in the following situations:

■ Unreachable destination server(s): Only as many incoming log messages are read as can be"delivered". When flow control is used, those messages are considered delivered that have beenwritten to the output buffer. When the output buffer is full, syslog-ng PE stops reading messagesfrom the connected sources. This means that no log messages get lost.

NoteIn case the application is sending its log messages through a blocking I/O socket, then it is the applicationthat stops sending new log messages and waits until the previous batch has been delivered. If the applicationis not sending logs through a blocking I/O socket, then it will keep sending messages (regardless of whetheror not the previous batch has been delivered), and this can result in the loss of log messages. For example,it is not possible to apply flow control in the case of a UDP source.

Drawbacks:

While this configuration gives you the fastest processing time, it has some limitations. It does not provideprotection against the loss of log messages in the following situations:

■ TCP error: In the case of a TCP connection, when messages are sent from the destination drivers tothe destination servers, messages are written to the TCP socket. The TCP socket sends anacknowledgement to the destination drivers once it has successfully processed messages. A messageis considered "delivered" when no error occurs during the process of writing the data to the socket,and the acknowledgement is received. Note, however, that if something goes wrong after messages

370syslog-ng.com

Flow control, no disk buffering, no RLTP™

have been successfully written to the TCP socket, log messages can still get lost. Also note that TCPerrors can occur on both the source and the destination side, and both can cause the loss of logmessages.

■ Message loss outside of syslog-ng PE: Because syslog-ng PE stores only a small number of logmessages in the memory, it is possible to lose messages outside of syslog-ng. For example, if theoutput buffer is full because the server is not reachable, syslog-ng PE will not read the source,meaning that the external application that generates the logs can drop the logs. If you want to minimizethe risk, use disk buffering. For details, see Section 13.3, Flow control, normal disk buffering, noRLTP™ (p. 371) and Section 13.4, Flow control, reliable disk buffering, no RLTP™ (p. 373).

■ Message loss when syslog-ng PE is stopped or restarted: When syslog-ng is stopped or restarted,the contents of the output buffers are lost. If you want to minimize the risk, use disk buffering. Fordetails, see Section 13.3, Flow control, normal disk buffering, no RLTP™ (p. 371) and Section 13.4,Flow control, reliable disk buffering, no RLTP™ (p. 373).

■ When syslog-ng PE is not able to operate normally (for example, when syslog-ng PE crashes dueto some unforeseen event): Log messages that were in the output buffer when the issue occurred getlost because those messages are stored in the memory.

13.3. Flow control, normal disk buffering, no RLTP™

How it works:

1. syslog-ng PE puts messages into the disk buffer (set via disk-buf-size()) when the destinationbecomes unavailable or when it is not able to process logs as fast as they arrive through the sources.

2. When the disk buffer is full, syslog-ng PE puts messages into the memory buffer (set viamem-buf-length()). When the memory buffer gets full too, then syslog-ng PE stops the source(flow-control mechanism).

Figure 13.2. Flow control, normal disk buffering, no RLTP™

How to set key parameters:

Set flags(flow-control) in the log path.

371syslog-ng.com

Flow control, normal disk buffering, no RLTP™

The memory buffer must be large enough to store the incoming messages of every source:

mem-buf-length() > sum of log-iw-size() of sources connected to this destination

Configure disk buffering. For details, see Example 13.1, Example configuration of normal disk-basedbuffering (p. 372).

Example 13.1. Example configuration of normal disk-based buffering

disk-buffer(

mem-buf-length(20000) # storing 20000 messages in memory, sum log-iw-size of sources

should be less than 20000 to use flow-control

disk-buf-size(2147483648) # storing 2 GB of messages on disk

reliable(no)

)

Benefits:

This configuration minimizes the loss of log messages in the following situations:

■ Unreachable destination server(s): Only as many incoming log messages are read as can be"delivered". When flow control is used in combination with disk buffering, messages that have beenwritten to the disk buffer and/or the memory buffer are considered delivered. When the memorybuffer becomes full, syslog-ng PE stops reading messages from the configured sources. This meansthat no log messages get lost.

■ Message loss outside of syslog-ng PE: The greatest advantage of this configuration over when nodisk buffering is used at all is that when the log-iw-size() control window is full, the flow-controlmechanism stops reading logs from the sources much later. This is because when it is not possibleto send logs directly to the destinations, they are first written to the disk and then the memory buffer.It is only after both the disk buffer and the memory buffer have been filled to their full capacity thatthe sources are stopped. This enables you to minimize the loss of log messages during peak hoursor when the network is temporarily down.

■ Message loss when syslog-ng PE is stopped or restarted: When syslog-ng is stopped or restarted,the contents of the memory buffer and the disk buffer are flushed to disk, meaning that no log lossoccurs.

NoteIn rare cases, the buffers stored on the disk can become corrupted, in which case syslog-ng PE may notable to process all the logs stored in the disk buffer.

Drawbacks:

One drawback of using disk buffering is that the processing of log messages by syslog-ng PE is slower.

This configuration does not provide protection against the loss of log messages in the following situations:

■ TCP error: In the case of a TCP connection, when messages are sent from the destination drivers tothe destination servers, messages are written to the TCP socket. The TCP socket sends an

372syslog-ng.com

Flow control, normal disk buffering, no RLTP™

acknowledgement to the destination drivers once it has successfully processed messages. A messageis considered "delivered" when no error occurs during the process of writing the data to the socket,and the acknowledgement is received. Note, however, that if something goes wrong after messageshave been successfully written to the TCP socket, log messages can still get lost. Also note that TCPerrors can occur on both the source and the destination side, and both can cause the loss of logmessages.

■ When syslog-ng PE is not able to operate normally (for example, when syslog-ng PE crashes dueto some unforeseen event): Log messages that were in the output buffer when the issue occurred getlost because those messages are stored in the memory.

13.4. Flow control, reliable disk buffering, no RLTP™

How it works:

syslog-ng puts messages into the disk buffer, until disk-buffer size reaches disk-buf-size(). Above thatsize, flow control is triggered. syslog-ng PE completely stops reading incoming messages from the source,making the control window (governed by log_iw_size()) fill up and blocking the sources.

In this configuration, log messages are stored on the disk (and not in the memory), which increases reliability.

Figure 13.3. Flow control, reliable disk buffering, no RLTP™

How to set key parameters:

Set flags(flow-control) in the log path.

Configure disk buffering. For details, see Example 13.2, Example configuration of reliable disk-basedbuffering (p. 373).

Example 13.2. Example configuration of reliable disk-based buffering

disk-buffer(

mem-buf-size(10485760) # storing 10 MB of messages in memory and on disk

disk-buf-size(2147483648) # storing 2 GB of messages only on disk

reliable(yes)

)

373syslog-ng.com

Flow control, reliable disk buffering, no RLTP™

Benefits:

This configuration minimizes the loss of log messages in the following situations:

■ Unreachable destination server(s): Only as many incoming log messages are read as can be"delivered". When flow control is used in combination with disk buffering, those messages areconsidered delivered that have been written to the disk buffer. As soon as the disk buffer is full,syslog-ng PE stops reading messages. This means that no log messages get lost.

■ Message loss outside of syslog-ng PE: One of the advantages of this configuration over when nodisk buffering is used at all is that when the log-iw-size() control window is full, the flow-controlmechanism stops reading logs from the sources much later. This is because when it is not possibleto send logs directly to the destinations, they are written to the disk. It is only after the disk bufferhas been filled to its full capacity that the sources are stopped. This enables you to minimize the lossof log messages during peak hours or when the network is temporarily down.

■ Message loss when syslog-ng PE is stopped or restarted: When syslog-ng is stopped or restarted,the contents of the disk buffer do not get lost, greatly increasing reliability.Also note that the memory buffer is only used as a cache in this configuration. Any data stored inthe memory has already been written to the disk buffer, which, again, results in more reliability.

NoteIn rare scenarios, the buffers stored on the disk can become corrupted, in which case syslog-ng PE maynot able to process all the logs stored in the disk buffer.

■ When syslog-ng PE is not able to operate normally (for example, when syslog-ng PE crashes dueto some unforeseen event): No messages get lost because the disk buffer is persistent and when thedisk buffer is full, syslog-ng PE stops reading messages from the sources. When syslog-ng PE isrestarted after a crash, it automatically recovers any unsent messages from the disk buffer and theoutput buffer. After the restart, syslog-ng PE sends the saved messages to the destination.

Drawbacks:

One drawback of using reliable disk buffering is that the processing of log messages by syslog-ng PE is slowerthan when messages are stored in the output buffer only, or when using normal disk buffering.

This configuration does not provide protection against the loss of log messages in the following situations:

■ TCP error: In the case of a TCP connection, when messages are sent from the destination drivers tothe destination servers, messages are written to the TCP socket. The TCP socket sends anacknowledgement to the destination drivers once it has successfully processed messages. A messageis considered "delivered" when no error occurs during the process of writing the data to the socket,and the acknowledgement is received. Note, however, that if something goes wrong after messageshave been successfully written to the TCP socket, log messages can still get lost. Also note that TCPerrors can occur on both the source and the destination side, and both can cause the loss of logmessages.

374syslog-ng.com

Flow control, reliable disk buffering, no RLTP™

13.5. Flow control, reliable disk buffering, RLTP™

How it works:

NoteThe example presented here is set in a client-relay-server scenario.

1. The sender sends messages in batches (set via flush-lines()).

2. The relay writes messages to the disk buffer.

3. Once messages have been written to the disk buffer, the relay returns an acknowledgement to theclient.

4. The relay sends messages to the server in batches (set via flush-lines()).

5. When the server has successfully received and processed the messages in the batch, it sends anacknowledgement of the processed messages to the relay.It is only at this point that the relay removes log messages from the disk buffer because this is whenlogs are considered "delivered" to the server.

After receiving the acknowledgement, the sender sends another batch of messages.

This configuration gives you the greatest degree of protection against log message loss. RLTP™ providesacknowledgement about the successful processing of log messages at the level of the application layer. Evenif the reception of log messages has been acknowledged by TCP at the transport layer, log messages areconsidered delievered only when the syslog-ng PE application has received an acknowledgement from the othersyslog-ng PE instance about the successful delivery of log messages.

This mechanism guarantees that log messages are not lost between the client and the relay, or between the relayand the server, or on the relay itself. To minimize the risk of message loss on the client or the server, use flowcontrol and reliable disk buffering.

Figure 13.4. Flow control, reliable disk buffering, RLTP™

How to set key parameters:

375syslog-ng.com

Flow control, reliable disk buffering, RLTP™

Set flags(flow-control) in the log path.

Configure disk buffering. For details, see Example 13.3, Example configuration of reliable disk-basedbuffering (p. 376).

Example 13.3. Example configuration of reliable disk-based buffering

disk-buffer(

mem-buf-size(10485760) # storing 10 MB of messages in memory and on disk

disk-buf-size(2147483648) # storing 2 GB of messages only in disk

reliable(yes)

)

Enable RLTP™ by setting transport(rltp). For details, see Section 12.2, RLTP™ options (p. 366).

Benefits:

■ Unreachable destination server(s): Only as many incoming log messages are read as can be"delivered". When flow control is used in combination with reliable disk buffering and RLTP™,those messages are considered delivered by the very first source driver that have been written to thedisk buffer. syslog-ng PE will not read new messages until the previous batch has been written tothe disk buffer.

■ TCP error: In the case of a TCP connection, when messages are sent from the destination drivers tothe destination servers, messages are written to the TCP socket. The TCP socket sends anacknowledgement to the destination drivers once it has successfully processed messages. However,while the acknowledge messages sent by the TCP socket implement flow control at the transportlayer, RLTP™ introduces flow control at the application layer. This means that log messages areonly considered delivered, when the RLTP™ acknowledge message is returned at the level of thesyslog-ng application. That is to say, when a TCP error occurs, messages that have been written tothe disk buffer do not get lost.

■ Message loss outside of syslog-ng PE: One of the advantages of this configuration over when nodisk buffering is used at all is that when the log-iw-size() control window is full, the flow-controlmechanism stops reading logs from the sources much later. This is because when it is not possibleto send logs directly to the destinations, they are written to the disk. It is only after the disk bufferhas been filled to its full capacity that the sources are stopped. This enables you to minimize the lossof log messages during peak hours or when the network is temporarily down.

■ Message loss when syslog-ng PE is stopped or restarted: When syslog-ng is stopped or restarted,the contents of the disk buffer do not get lost, greatly increasing reliability.Also note that the memory buffer is only used as a cache in this configuration. Any data stored inthe memory has already been written to the disk buffer, which, again, results in more reliability.

NoteIn rare scenarios, the buffers stored on the disk can become corrupted, in which case syslog-ng PE maynot able to process all the logs stored in the disk buffer.

376syslog-ng.com

Flow control, reliable disk buffering, RLTP™

■ When syslog-ng PE is not able to operate normally (for example, when syslog-ng PE crashes dueto some unforeseen event): No messages get lost because the disk buffer is persistent and when thedisk buffer is full, syslog-ng PE stops reading messages from the sources. When syslog-ng PE isrestarted after a crash, it automatically recovers any unsent messages from the disk buffer and theoutput buffer. After the restart, syslog-ng PE sends the saved messages to the destination.

Drawbacks:

This configuration results in the slowest processing time out of all the options described in this chapter.

13.6. Deciding which loss prevention mechanism to apply

Choosing the ideal configuration for your environment may not always be a starightforward decision. Dependingon your use case, it is worth considering which outcome is more desirable (with the following points representingthe two opposite ends of the spectrum):

■ an application that does not slow down or stop — at the price of losing logs

■ no log messages get lost — at the price of a slower application or an application that stops(temporarily)

TipIf your application sends its logs through a blocking I/O socket and you prefer not to slow down or stop the applicationwhen log messages are arriving in volumes greater than syslog-ng PE is able to process, then consider turning flow controloff on the client side. This way, you will not be using the whole application-client-server chain at full capacity, and yetstill be able to spot the loss of application log messages at the beginning of the chain already, in the internal logs of theclient.

377syslog-ng.com

Deciding which loss prevention mechanism to apply

Chapter 14. Manipulating messages

This chapter explains the methods that you can use to customize, reformat, and modify log messages usingsyslog-ng Premium Edition.

■ Section 14.1, Customizing message format (p. 378) explains how to use templates and macros tochange the format of log messages, or the names of logfiles and database tables.

■ Section 14.2, Modifying messages (p. 398) describes how to use rewrite rules to search and replacecertain parts of the message content.

■ Section 14.3, Regular expressions (p. 403) lists the different types of regular expressions that can beused in various syslog-ng PE objects like filters and rewrite rules.

14.1. Customizing message format

The following sections describe how to customize the names of logfiles, and also how to use templates, macros,and template functions.

■ Section 14.1.1, Formatting messages, filenames, directories, and tablenames (p. 378) explains howmacros work.

■ Section 14.2, Modifying messages (p. 398) describes how to use macros and templates to format logmessages or change the names of logfiles and database tables.

■ Section 14.1.5, Macros of syslog-ng PE (p. 383) lists the different types of macros available insyslog-ng PE.

■ Section 14.1.6, Using template functions (p. 391) explains what template functions are and how touse them.

■ Section 14.1.7, Template functions of syslog-ng PE (p. 392) lists the template functions available insyslog-ng PE.

14.1.1. Formatting messages, filenames, directories, and tablenames

The syslog-ng PE application can dynamically create filenames, directories, or names of database tables usingmacros that help you organize your log messages. Macros refer to a property or a part of the log message, forexample, the ${HOST} macro refers to the name or IP address of the client that sent the log message, while${DAY} is the day of the month when syslog-ng has received the message. Using these macros in the path ofthe destination log files allows you for example to collect the logs of every host into separate files for everyday.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example, to addelements of the message header to the message text.

378syslog-ng.com

Customizing message format

NoteIf a message uses the IETF-syslog format (RFC5424), only the text of the message can be customized (that is, the$MESSAGE part of the log), the structure of the header is fixed.

■ For details on using templates and macros, see Section 14.1.2, Templates and macros (p. 379).

■ For a list and description of the macros available in syslog-ng PE, see Section 14.1.5, Macros ofsyslog-ng PE (p. 383).

■ For details on using custom macros created with CSV parsers and pattern databases, see Chapter15, Parsing and segmenting structured messages (p. 407) and Section 16.2.1, Using parser resultsin filters and templates (p. 422), respectively.

14.1.2. Templates and macros

The syslog-ng PE application allows you to define message templates, and reference them from every objectthat can use a template. Templates can include strings, macros (for example date, the hostname, and so on),and template functions. For example, you can use templates to create standard message formats or filenames.For a list of macros available in syslog-ng Premium Edition, see Section 14.1.5, Macros of syslog-ng PE (p. 383).For the macros of the syslog-ng Agent for Windows application, seeChapter 8, Customizing the message formatin The syslog-ng Agent for Windows 6 LTS Administrator Guide. Fields from the structured data (SD) part ofmessages using the new IETF-syslog standard can also be used as macros.

Declaration:

template <template-name> {

template("<template-expression>") <template-escape(yes)>;

};

Template objects have a single option called template-escape(), which is disabled by default(template-escape(no)). This behavior is useful when the messages are passed to an application that cannothandle escaped characters properly. Enabling template escaping (template-escape(yes)) causes syslog-ngto escape the ', ", and backslash characters from the messages.

NoteIn versions 2.1 and earlier, the template-escape() option was enabled by default.

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells.Although using braces around macro names is not mandatory, and the "$MSG" and "${MSG}" formats areequivalent, using the "${MSG}" format is recommended for clarity.

To use a literal $ character in a template, you have to escape it. In syslog-ng PE versions 4.0-4.2, use a backslash(\$). In version 5.0 and later, use $$.

379syslog-ng.com

Templates and macros

NoteTo use a literal @ character in a template, use @@.

Default values for macros can also be specified by appending the :- characters and the default value of themacro. If a message does not contain the field referred to by the macro, or it is empty, the default value will beused when expanding the macro. For example, if a message does not contain a hostname, the following macrocan specify a default hostname.

${HOST:-default_hostname}

NoteFor the macros of the syslog-ng Agent for Windows application, see Chapter 8, Customizing the message format in Thesyslog-ng Agent for Windows 6 LTS Administrator Guide.

By default, syslog-ng sends messages using the following template: ${ISODATE} ${HOST}

${MSGHDR}${MSG}\n. (The ${MSGHDR}${MSG} part is written together because the ${MSGHDR} macroincludes a trailing whitespace.)

NoteEarlier versions of syslog-ng used templates and scripts to send log messages into SQL databases. Starting from version2.1, syslog-ng natively supports direct database access using the sql() destination. For details, see Section 7.13.7, sql()destination options (p. 280).

Example 14.1. Using templates and macrosThe following template (t_demo_filetemplate) adds the date of the message and the name of the host sending themessage to the beginning of the message text. The template is then used in a file destination: messages sent to thisdestination (d_file) will use the message format defined in the template.

template t_demo_filetemplate {

template("${ISODATE} ${HOST} ${MSG}\n"); template-escape(no); };

destination d_file {

file("/var/log/messages" template(t_demo_filetemplate)); };

Templates can also be used inline, if they are used only at a single location. The following destination is equivalent withthe previous example:

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} ${MSG}\n") template-escape(no) );

};

The following file destination uses macros to daily create separate logfiles for every client host.

destination d_file {

file("/var/log/${YEAR}.${MONTH}.${DAY}/${HOST}.log");

};

NoteMacros can be used to format messages, and also in the name of destination files or database tables. However, they cannotbe used in sources as wildcards, for example, to read messages from files or directories that include a date in their name.

380syslog-ng.com

Templates and macros

14.1.3. Date-related macros

The macros related to the date of the message (for example: ${ISODATE}, ${HOUR}, and so on) have twofurther variants each:

■ S_ prefix, for example, ${S_DATE}: The ${S_DATE} macro represents the date found in the logmessage, that is, when the message was sent by the original application.

WarningTo use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior ofsyslog-ng PE).

■ R_ prefix, for example, ${R_DATE}: ${R_DATE} is the date when syslog-ng PE has received themessage.

The ${DATE} macro equals the ${S_DATE} macro.

The values of the date-related macros are calculated using the original timezone information of the message.To convert it to a different timezone, use the time-zone() option. You can set the time-zone() option asa global option, or per destination. For sources, it applies only if the original message does not contain timezoneinformation. Converting the timezone changes the values of the following date-related macros (macros MSECand USEC are not changed):

■ AMPM

■ DATE

■ DAY

■ FULLDATE

■ HOUR

■ HOUR12

■ ISODATE

■ MIN

■ MONTH

■ MONTH_ABBREV

■ MONTH_NAME

381syslog-ng.com

Date-related macros

■ MONTH_WEEK

■ SEC

■ STAMP

■ TZ

■ TZOFFSET

■ UNIXTIME

■ WEEK

■ WEEK_DAY

■ WEEK_DAY_ABBREV

■ WEEK_DAY_NAME

■ YEAR

■ YEAR_DAY

14.1.4. Hard vs. soft macros

Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macroderives its value from the timestamp. Hard macros are read-only. Soft macros (sometimes also called name-valuepairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), orcustom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. In contrast tohard macros, soft macros are writable and can be modified within syslog-ng PE, for example, using rewriterules.

Hard and soft macros are rather similar and often treated as equivalent. Macros are most commonly used infilters and templates, which does not modify the value of the macro, so both soft and hard macros can be used.However, it is not possible to change the values of hard macros in rewrite rules or via any other means.

The following macros in syslog-ng PE are hard macros and cannot be modified: BSDTAG, CONTEXT_ID, DATE,DAY, FACILITY_NUM, FACILITY, FULLDATE, HOUR, ISODATE, LEVEL_NUM, LEVEL, MIN, MONTH_ABBREV,MONTH_NAME, MONTH, MONTH_WEEK, PRIORITY, PRI, RCPTID, SDATA, SEC, SEQNUM, SOURCEIP, STAMP, TAG,TAGS, TZOFFSET, TZ, UNIXTIME, WEEK_DAY_ABBREV, WEEK_DAY_NAME, WEEK_DAY, WEEK, YEAR_DAY, YEAR.

The following macros can be modified:FULLHOST_FROM, FULLHOST, HOST_FROM, HOST, LEGACY_MSGHDR,MESSAGE, MSG,MSGID, MSGONLY, PID, PROGRAM, SOURCE. Custom values created using rewrite rules or parserscan be modified as well, just like stored matches of regular expressions ($0 ... $255).

382syslog-ng.com

Hard vs. soft macros

14.1.5. Macros of syslog-ng PE

The following macros are available in syslog-ng PE.

WarningThese macros are available when syslog-ng PE successfully parses the incoming message as a syslog message, or youuse some other parsing method and map the parsed values to these macros.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entireincoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PE generates a newsyslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing,it interferes with other flags, for example, disables flags(no-multi-line).

AMPMDescription: Typically used together with the ${HOUR12} macro, ${AMPM} returns the period of the day: AMfor hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in syslog-ng PE 3.2 and later.

APP_NAMEDescription: An alias for the APPLICATION_NAME macro.

APPLICATION_NAMEDescription:

■ At event container: Name of the application the message came from

■ At file as the name of creator (default value): syslog-ng-agent

BSDDATE, R_BSDDATE, S_BSDDATEDescription: Date of the message in BSD timestamp format (month/day/hour/minute/second, each expressedin two digits). This is the original syslog time stamp without year information, for example Jun 13 15:58:00.If possible, it is recommended to use ISODATE for timestamping.

BSDTAGDescription: Facility/priority information in the format used by the FreeBSD syslogd: a priority numberfollowed by a letter that indicates the facility. The priority number can range from 0 to 7. The facility letter canrange from A to Y, where A corresponds to facility number zero (LOG_KERN), B corresponds to facility 1(LOG_USER), and so on.

Custom macrosDescription: CSV parsers and pattern databases can also define macros from the content of the messages, forexample, a pattern database rule can extract the username from a login message and create a macro that referencesthe username. For details on using custom macros created with CSV parsers and pattern databases, see Chapter15, Parsing and segmenting structured messages (p. 407) and Section 16.2.1, Using parser results in filters andtemplates (p. 422), respectively.

383syslog-ng.com

Macros of syslog-ng PE

DATE, R_DATE, S_DATEDescription:Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second,each expressed in two digits). This is the original syslog time stamp without year information, for example:Jun 13 15:58:00.

DAY, R_DAY, S_DAYDescription: The day the message was sent.

FACILITYDescription: The name of the facility (for example, kern) that sent the message.

FACILITY_NUMDescription: The numerical code of the facility (for example, 0) that sent the message.

FILE_FACILITYDescription: The facility that sent the message.

FILE_LEVELDescription: Importance level of the message represented as a number: 6 - Success, 5 - Informational, 4-Warning, or 3 - Error).

FILE_MESSAGEDescription: The content of the message.

FILE_MSGDescription: The content of the message. This is an alias of the FILE_MESSAGE macro.

FILE_NAMEDescription: Name of the log file (including its path) from where the syslog-ng PE received the message.

FULLDATE, R_FULLDATE, S_FULLDATEDescription:A nonstandard format for the date of the message using the same format as ${DATE}, but includingthe year as well, for example: 2006 Jun 13 15:58:00.

FULLHOSTDescription: The name of the source host where the message originates from.

■ If the message traverses several hosts and the chain-hostnames() option is on, the first host inthe chain is used.

■ If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $FULLHOSTmacro will be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNShostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will havethe same value.

384syslog-ng.com

Macros of syslog-ng PE

■ If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $FULLHOSTmacro will be the hostname retrieved from the log message. That way the name of the original senderhost can be used, even if there are log relays between the sender and the server.

NoteThe use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have noeffect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains ahostname.

For details on using name resolution in syslog-ng PE, see Section 20.3, Using name resolution insyslog-ng (p. 461).

FULLHOST_FROMDescription: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS.If the message traverses several hosts, this is the last host in the chain.

The syslog-ng PE application uses the following procedure to determine the value of the $FULLHOST_FROM

macro:

1. The syslog-ng PE application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname.If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM macro. This valuewill be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname ifuse-fqdn() is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${FULLHOST_FROM}macrowill return the IP address of the sender host.

For details on using name resolution in syslog-ng PE, see Section 20.3, Using name resolution insyslog-ng (p. 461).

HOUR, R_HOUR, S_HOURDescription: The hour of day the message was sent.

HOUR12, R_HOUR12, S_HOUR12Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM} macro.12AM is midnight. Available in syslog-ng PE 3.2 and later.

HOSTDescription: The name of the source host where the message originates from.

■ If the message traverses several hosts and the chain-hostnames() option is on, the first host inthe chain is used.

385syslog-ng.com

Macros of syslog-ng PE

■ If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $HOST macrowill be the DNS hostname of the host that sent the message to syslog-ng PE (that is, the DNS hostnameof the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.

■ If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $HOSTmacro will be the hostname retrieved from the log message. That way the name of the original senderhost can be used, even if there are log relays between the sender and the server.

NoteThe use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have noeffect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains ahostname.

For details on using name resolution in syslog-ng PE, see Section 20.3, Using name resolution insyslog-ng (p. 461).

HOST_FROMDescription: The FQDN of the host that sent the message to syslog-ng as resolved by syslog-ng using DNS.If the message traverses several hosts, this is the last host in the chain.

The syslog-ng PE application uses the following procedure to determine the value of the $HOST_FROM macro:

1. The syslog-ng PE application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, syslog-ng PE attempts to resolve the IP address to a hostname.If it succeeds, the returned hostname will be the value of the $HOST_FROM macro. This value willbe the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn()is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${HOST_FROM} macro willreturn the IP address of the sender host.

For details on using name resolution in syslog-ng PE, see Section 20.3, Using name resolution insyslog-ng (p. 461).

ISODATE, R_ISODATE, S_ISODATEDescription: Date of the message in the ISO 8601 compatible standard timestamp format(yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it isrecommended to use ${ISODATE} for timestamping. Note that syslog-ng can produce fractions of a second(for example milliseconds) in the timestamp by using the frac-digits() global or per-destination option.

LEVEL_NUMDescription: The priority (also called severity) of the message, represented as a numeric value, for example,3. For the textual representation of this value, use the ${LEVEL}macro. See Section PRIORITY or LEVEL (p. 388)for details.

386syslog-ng.com

Macros of syslog-ng PE

MIN, R_MIN, S_MINDescription: The minute the message was sent.

MONTH, R_MONTH, S_MONTHDescription: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.

MONTH_ABBREV, R_MONTH_ABBREV, S_MONTH_ABBREVDescription: The English abbreviation of the month name (3 letters).

MONTH_NAME, R_MONTH_NAME, S_MONTH_NAMEDescription: The English name of the month name.

MONTH_WEEK, R_MONTH_WEEK, S_MONTH_WEEKDescription: The number of the week in the given month (0-5). The week with numerical value 1 is the firstweek containing a Monday. The days of month before the first Monday are considered week 0. For example,if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30thand 31st) is week 5.

MONTHNAME, R_MONTHNAME, S_MONTHNAMEDescription: The English name of the month the message was sent, abbreviated to three characters (for exampleJan, Feb, and so on).

MSEC, R_MSEC, S_MSECDescription: The millisecond the message was sent.

Available in syslog-ng PE version 4 F2 and later.

MSG or MESSAGEDescription: Text contents of the log message without the program name and pid. Note that this has changedin syslog-ng version 3.0: in earlier versions this macro included the program name and the pid. In syslog-ng3.0, the ${MSG}macro became equivalent with the ${MSGONLY}macro. The program name and the pid togetherare available in the ${MSGHDR} macro.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and theentire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng PEgenerates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse)disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line).

MSGHDRDescription: The name and the PID of the program that sent the log message in PROGRAM[PID]: format.Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PIDfields of the message are empty.

MSGIDDescription: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. Forexample, a firewall might use the ${MSGID} "TCPIN" for incoming TCP traffic and the ${MSGID} "TCPOUT"

387syslog-ng.com

Macros of syslog-ng PE

for outgoing TCP traffic. By default, syslog-ng PE does not specify this value, but uses a dash (-) characterinstead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.

MSGONLYDescription: Message contents without the program name or pid.

OSUPTIMEDescription: The time elapsed since the computer running syslog-ng PE has booted. The value of this macrois an integer containing the time in 1/100th of the second. Note that currently syslog-ng PE can access this dataonly on Linux platforms. On other platforms the macro contains the time elapsed since syslog-ng PE was started.

Available in syslog-ng PE version 6.0.5 and later.

PIDDescription: The PID of the program sending the message.

PRIDescription: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslogmessages.

PRIORITY or LEVELDescription:The priority (also called severity) of the message, for example, error. For the textual representationof this value, use the ${LEVEL} macro. See Section PRIORITY or LEVEL (p. 388) for details.

PROCESS_IDDescription: PID of the application the message came from.

PROGRAMDescription: The name of the program sending the message. Note that the content of the ${PROGRAM}variable may not be completely trusted as it is provided by the client program that constructed the message.

RCPTIDDescription: This is disabled by default due to performance issues. To enable it, add the next option to theglobal options: use-uniqid(yes);. For details, see also Section use-uniqid() (p. 349). A unique ID for messagesgenerated at reception time on the receiving host. It facilitates defining relationships between messages thatare potentially distributed to different files on the same host, or different hosts.

Example 14.2. Using ${RCPTID} macroUsing the following template statement in the configuration: template demo_template { template("${DATE}

${HOST} ${PROG}: ${MSG} ID:${RCPTID}\n"); }; the outgoing message will be the following: <133>Feb 25

14:09:07 webserver syslogd: restart. ID:1.

SDATA, .SDATA.SDID.SDNAME, STRUCTURED_DATADescription: The syslog-ng application automatically parses the STRUCTURED-DATA part of IETF-syslogmessages, which can be referenced in macros. The ${SDATA}macro references the entire STRUCTURED-DATA

388syslog-ng.com

Macros of syslog-ng PE

part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME}

macro.Available only in syslog-ng Premium Edition 4.0 and later.

NoteWhen using STRUCTURED-DATA macros, consider the following:

When referencing an element of the structured data, the macro must begin with the dot (.) character. Forexample, ${.SDATA.timeQuality.isSynced}.

■

■ The SDID and SDNAME parts of the macro names are case sensitive:${.SDATA.timeQuality.isSynced} is not the same as ${.SDATA.TIMEQUALITY.ISSYNCED}.

Example 14.3. Using SDATA macrosFor example, if a log message contains the following structured data: [exampleSDID@0 iut="3"

eventSource="Application" eventID="1011"][examplePriority@0 class="high"] you can use macroslike: ${.SDATA.exampleSDID@0.eventSource} — this would return the Application string in this case.

SEC, R_SEC, S_SECDescription: The second the message was sent.

SEQNUMDescription: The ${SEQNUM} macro contains a sequence number for the log message. The value of the macrodepends on the scenario, and can be one of the following:

■ If syslog-ng PE receives a message via the IETF-syslog protocol that includes a sequence ID, thisID is automatically available in the ${SEQNUM} macro.

■ If the message is a Cisco IOS log message using the extended timestamp format, then syslog-ng PEstores the sequence number from the message in this macro. If you forward this message theIETF-syslog protocol, syslog-ng PE includes the sequence number received from the Cisco devicein the ${.SDATA.meta.sequenceId} part of the message.

NoteTo enable sequence numbering of log messages on Cisco devices, use the following command on thedevice (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual ofyour Cisco device.

■ For locally generated messages (that is, for messages that are received from a local source, and notfrom the network), syslog-ng PE calculates a sequence number when sending the message to adestination (it is not calculated for relayed messages).

• The sequence number is not global, but per-destination. Essentially, it counts the number ofmessages sent to the destination.

• This sequence number increases by one for every message sent to the destination, and is not losteven when syslog-ng PE is reloaded or restarted.

389syslog-ng.com

Macros of syslog-ng PE

• This sequence number is added to every message that uses the IETF-syslog protocol(${.SDATA.meta.sequenceId}), and can be added to BSD-syslog messages using the${SEQNUM} macro.

NoteIf you need a sequence number for every log message that syslog-ng PE receives, use the RCPTID macro.

SOURCEDescription: The identifier of the source statement in the syslog-ng PE configuration file that received themessage. For example, if syslog-ng PE received the log message from the source s_local { internal();

}; source statement, the value of the ${SOURCE} macro is s_local. This macro is mainly useful for debuggingand troubleshooting purposes.

SOURCEIPDescription: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host inthe ${FULLHOST_FROM} macro.) Please note that when a message traverses several relays, this macro containsthe IP of the last relay.

STAMP, R_STAMP, S_STAMPDescription: A timestamp formatted according to the ts-format() global or per-destination option.

SYSUPTIMEDescription: The time elapsed since the syslog-ng PE instance was started (that is, the uptime of the syslog-ngPE process). The value of this macro is an integer containing the time in 1/100th of the second. For the uptimeof the host running syslog-ng PE see Section OSUPTIME (p. 388).

Available in syslog-ng PE version 4 F1 and later.

TAGDescription: The priority and facility encoded as a 2 digit hexadecimal number.

TAGSDescription:A comma-separated list of the tags assigned to the message. Available only in syslog-ng PremiumEdition 3.2 and later.

NoteNote that the tags are not part of the log message and are not automatically transferred from a client to the server. Forexample, if a client uses a pattern database to tag the messages, the tags are not transferred to the server. A way oftransferring the tags is to explicitly add them to the log messages using a template and the ${TAGS} macro, or to addthem to the structured metadata part of messages when using the IETF-syslog message format.

When sent as structured metadata, it is possible to reference to the list of tags on the central server, and for example, toadd them to a database column.

390syslog-ng.com

Macros of syslog-ng PE

TZ, R_TZ, S_TZDescription: An alias of the ${TZOFFSET} macro.

TZOFFSET, R_TZOFFSET, S_TZOFFSETDescription: The time-zone as hour offset from GMT, for example: -07:00. In syslog-ng 1.6.x this used tobe -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.

UNIXTIME, R_UNIXTIME, S_UNIXTIMEDescription: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.

UNIQIDDescription: A globally unique ID generated from the HOSTID and the RCPTID in the format ofHOSTID@RCPTID. For details, see Section use-uniqid() (p. 349) and Section RCPTID (p. 388).

Available in syslog-ng PE version 5 F2 and later.

USEC, R_USEC, S_USECDescription: The microsecond the message was sent.

Available in syslog-ng PE version 4 F2 and later.

YEAR, R_YEAR, S_YEARDescription: The year the message was sent.

WEEK, R_WEEK, S_WEEKDescription: The week number of the year, prefixed with a zero for the first nine week of the year. (The firstMonday in the year marks the first week.)

WEEK_ABBREV, R_WEEK_ABBREV, S_WEEK_ABBREVDescription: The 3-letter English abbreviation of the name of the day the message was sent, for example Thu.

WEEK_DAY, R_WEEK_DAY, S_WEEK_DAYDescription: The day of the week as a numerical value (1-7).

WEEKDAY, R_WEEKDAY, S_WEEKDAYDescription: These macros are deprecated, use ${WEEK_ABBREV}, ${R_WEEK_ABBREV},${S_WEEK_ABBREV} instead. The 3-letter name of the day of week the message was sent, for example Thu.

WEEK_DAY_NAME, R_WEEK_DAY_NAME, S_WEEK_DAY_NAMEDescription: The English name of the day.

14.1.6. Using template functions

A template function is a transformation: it modifies the way macros or name-value pairs are expanded. Templatefunctions can be used in template definitions, or when macros are used in the configuration of syslog-ng PE.Template functions use the following syntax:

391syslog-ng.com

Using template functions

$(function-name parameter1 parameter2 parameter3 ...)

For example, the $(echo) template function simply returns the value of the macro it receives as a parameter,thus $(echo ${HOST}) is equivalent to ${HOST}.

The parameters of template functions are separated by a whitespace character. If you want to use a longer stringor multiple macros as a single parameter, enclose the parameter in double-quotes or apostrophes. For example:

$(echo "${HOST} ${PROGRAM} ${PID}")

Template functions can be nested into each other, so the parameter of a template function can be another templatefunction, like:

$(echo $(echo ${HOST}))

For details on using template functions, see the descriptions of the individual template functions in Section14.1.7, Template functions of syslog-ng PE (p. 392).

14.1.7. Template functions of syslog-ng PE

The following template functions are available in syslog-ng PE.

echoSyntax:

$(echo argument)

Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

format-cef-extensionsyslog-ng PE version 5 F6 includes a new template function (format-cef-extension) to format name-valuepairs as ArcSight Common Event Format extensions. Note that the template function only formats the selectedname-value pairs, it does not provide any mapping. There is no special support for creating the prefix part ofa Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEFextension escaping rules format, see the ArcSight Common Event Format.

You can use the value-pairs that syslog-ng PE stores about the log message as CEF fields. Using value-pairs,you can:

■ select which value-pairs to use as CEF fields,

■ add custom value-pairs as CEF fields,

■ rename value-pairs, and so on.

For details, see Section 2.12, Structuring macros, metadata, and other value-pairs (p. 25). Note that the syntaxof format-* template functions is different from the syntax of value-pairs(): these template functions usea syntax similar to command lines.

Using the format-cef-extension template function, has the following prerequisites:

■ Load the the cef module in your configuration:

392syslog-ng.com

Template functions of syslog-ng PE

@module cef

■ Set the on-error global option to drop-property, otherwise if the name of a name-value pairincludes an invalid character, syslog-ng PE drops the entire message. (Key name in CEF extensionscan contain only the A-Z, a-z and 0-9 characters.)

options {

on-error("drop-property");

};

■ The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8flag in the message source.

Example 14.4. Using the format-cef-extension template functionThe following example selects every available information about the log message, except for the date-related macros(R_* and S_*), selects the .SDATA.meta.sequenceIdmacro, and defines a new value-pair called MSGHDR that containsthe program name and PID of the application that sent the log message (since you will use the template-function in atemplate, you must escape the double-quotes).

$(format-cef-extension --scope syslog,all_macros,selected_macros \

--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \

--pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefixfrom the key names.

template("$(format-cef-extension --subkeys .cef.)\n")

The following example shows how to use this template function to store log messages in CEF format:

destination d_cef_extension {

file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope

selected_macros --scope nv_pairs)\n"));

};

format-jsonSyntax:

$(format-json parameters)

Description: The format-json template function receives value-pairs as parameters and converts them intoJavaScript Object Notation (JSON) format. Including the template function in a message template allows youto store selected information about a log message (that is, its content, macros, or other metadata) in JSONformat. Note that the input log message does not have to be in JSON format to use format-json, you canreformat any incoming message as JSON.

You can use the value-pairs that syslog-ng PE stores about the log message as JSON fields. Using value-pairs,you can:

■ select which value-pairs to use as JSON fields,

■ add custom value-pairs as JSON fields,

■ rename value-pairs, and so on.

393syslog-ng.com

Template functions of syslog-ng PE

For details, see Section 2.12, Structuring macros, metadata, and other value-pairs (p. 25). Note that the syntaxof format-json is different from the syntax of value-pairs(): format-json uses a syntax similar tocommand lines.

NoteBy default, syslog-ng PE handles every message field as a string. For details on how to send selected fields as other typesof data (for example, handle the PID as a number), see Section 2.12.1, Specifying data types in value-pairs (p. 26).

Example 14.5. Using the format-json template functionThe following example selects every available information about the log message, except for the date-related macros(R_* and S_*), selects the .SDATA.meta.sequenceIdmacro, and defines a new value-pair called MSGHDR that containsthe program name and PID of the application that sent the log message (since you will use the template-function in atemplate, you must escape the double-quotes).

$(format-json --scope syslog,all_macros,selected_macros \

--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \

--pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in JSON format:

destination d_json {

file("/var/log/messages.json" template("$(format_json --scope selected_macros --scope

nv_pairs)\n"));

};

NoteIn case of syslog-ng macros starting with a dot (for example ".SDATA.meta.sequenceID") an empty key name is addedat the top level of the JSON structure. You can work around this by adding --shift 1 as a parameter to the templatefunction. For example in case of ".SDATA.meta.sequenceID", an empty key name is added at the top level of theJSON structure:

{"":

{"SDATA" :

{"meta" :

{"sequenceID": "123"}

}

}

}

format-welfThis template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELFformat is a comma-separated list of name=value elements. Note that the order of the elements is random. Ifthe value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on theWELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For detailson selecting value-pairs, see Section value-pairs() (p. 27).

Example 14.6. Using the format-welf() template functionThe following example selects every available information about the log message, except for the date-related macros(R_* and S_*), selects the .SDATA.meta.sequenceIdmacro, and defines a new value-pair called MSGHDR that containsthe program name and PID of the application that sent the log message (since you will use the template-function in atemplate, you must escape the double-quotes).

394syslog-ng.com

Template functions of syslog-ng PE

$(format-welf --scope syslog,all_macros,selected_macros \

--exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \

--pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in WELF format:

destination d_welf {

file("/var/log/messages.welf" template("$(format-welf --scope selected_macros --scope

nv_pairs)\n"));

};

grepSyntax:

$(grep condition value-to-select)

Description: The grep template function is useful when using a pattern database to correlate related logmessages. The grep template function can be used to filter the messages of the same context when the indexof the particular message is not known.

Example 14.7. Using the grep template functionThe following example selects the message of the context that has a username name-value pair with the root value,and returns the value of the auth_method name-value pair.

$(grep ("${username}" == "root") ${auth_method})

It is possible to specify multiple name-value pairs as parameters, separated with commas. If multiple messagesmatch the condition of grep, these will be returned also separated by commas. This can be used for exampleto collect the e-mail recipients from postfix messages.

hashSyntax:

$(<method> [opts] $arg1 $arg2 $arg3...)

Options:

--length N, -l N

Truncate the hash to the first N characters.

Description:Calculates a hash of the string or macro received as argument using the specified hashing method.If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequentarguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to sha256. Macrosare expected as arguments, and they are concatenated without the use of additional characters.

This template function can be used for anonymizing sensitive parts of the log message (for example username)that were parsed out using PatternDB before storing or forwarding the message. This way, the ability ofcorrelating messages along this value is retained.

395syslog-ng.com

Template functions of syslog-ng PE

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way,IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as anon-truncated hash.

Example 14.8. Using the $(hash) template functionThe following example calculates the SHA1 hash of the hostname of the message:

$(sha1 $HOST)

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

$(sha1 $HOST salted)

To use shorter hashes, set the --length:

$(sha1 --length 6 $HOST)

To replace the hostname with its hash, use a rewrite rule:

rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};

Example 14.9. Anonymizing IP addressesThe following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

rewrite pseudonymize_ip_addresses_in_message {

subst (

"((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))",

"$(sha1 $0)",

value("MSG"),

flags(global)

);

};

ifSyntax:

$(if (<condition>) <true template> <false template>)

Description: Returns the value of the <true template> parameter if the <condition> is true. If the<condition> is false, the value of <false template> is returned.

Example 14.10. Using pattern databases and the if template functionThe following example returns violation if the username name-value pair of a message processed with pattern databaseis root, and system otherwise.

$(if ('${username}' == 'root') 'violation' 'system')

This can be used to set the class of a message in pattern database rules based on the condition.

<value name="username">$(if ("${username}" == "root") "violation" "system")</value>

Since template functions can be embedded into each other, it is possible to use another template function as the templateof the first one. For example, the following expression returns root if the username is root, admin if the username isjoe, and normal user otherwise.

<value name="username">

$(if ("${username}" == "root")

"root"

$(if ("${username}" == "joe") "admin" "normal user")

"normal user")</value>

396syslog-ng.com

Template functions of syslog-ng PE

indent-multi-lineSyntax:

$(indent-multi-line parameter)

Description: This template function makes it possible to write multi-line log messages into a file. The first lineis written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

Example 14.11. Using the indent-multi-line template functionThe following example writes multi-line messages into a text file.

destination d_file {

file ("/var/log/messages"

template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );

};

ipv4-to-intSyntax:

$(ipv4-to-int parameter)

Description:Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

NoteThis template function is available only if the convertfuncs module has been loaded. By default, syslog-ng PE loadsevery available module.

By default, syslog-ng PE loads every available module. For details, see Section 5.6.1, Loading modules (p. 80)

Numeric operationsSyntax:

$(<operator> <first_operand> <second_operand>)

Description:

This template function performs simple numerical operations (like addition or multiplication) on integer numbersor macros containing integer numbers (for example, ${LEVEL_NUM} or ${YEAR}), and returns the result of theoperation. The values used in the operation are not modified, that is, macros retain their original values. If oneof the operands is not an integer, syslog-ng PE will not execute the template function, and return the NaN

(Not-a-Number) value.

Available only in syslog-ng PE 4 F1 and later.

397syslog-ng.com

Template functions of syslog-ng PE

Operation:OperatorA d d i t i o n : r e t u r n s t h e v a l u e o f<first_operand>+<second_operand>

+

Subtract ion: re turns the value of<first_operand>-<second_operand>

-

Multiplication: returns the value of<first_operand>*<second_operand>

*

D i v i s i o n : r e t u r n s t h e v a l u e o f<first_operand>/<second_operand>

/

Modulus (remainder): returns the remainder of<first_operand>/<second_operand>

%

WarningThe output of this template function is always an integer. If the return value would be a floating-point number, the floatingpart is simply omitted, for example, 5.2 becomes 5, 5.8 becomes 5, -5.8 becomes -5.

It is also possible to nest numerical operations.

Example 14.12. Using numerical template functionsThe following template function returns the facility of the log message multiplied by 8:

$(* ${FACILITY_NUM} 8)

Template functions can be nested into each other: the following template function returns the facility of the log messagemultiplied by 8, then adds this value to the severity of the message (the result is actually the priority of the message):

$(+ ${LEVEL_NUM} $(* ${FACILITY_NUM} 8))

stripSyntax:

$(strip "<macro>")

Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macrosseparated with whitespace in a single template function, for example:

$(strip "${MESSAGE}" "${PROGRAM}")

Available in syslog-ng PE version 6.0.6 and later.

14.2. Modifying messages

The syslog-ng application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objectssimilar to parsers and filters and can be used in log paths. The syslog-ng application has two methods to rewriteparts of the log messages: substituting (setting) a part of the message to a fix value, and a generalsearch-and-replace mode.

398syslog-ng.com

Modifying messages

Substitution completely replaces a specific part of the message that is referenced using a built-in or user-definedmacro.

General rewriting searches for a string in the entire message (or only a part of the message specified by a macro)and replaces it with another string. Optionally, this replacement string can be a template that contains macros.

Rewriting messages is often used in conjunction with message parsing Chapter 15, Parsing and segmentingstructured messages (p. 407).

Rewrite rules are similar to filters: they must be defined in the syslog-ng configuration file and used in the logstatement. You can also define the rewrite rule inline in the log path.

NoteThe order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

14.2.1. Replacing message parts

To replace a part of the log message, you have to:

■ define a string or regular expression to find the text to replace

■ define a string to replace the original text (macros can be used as well)

■ select the field of the message that the rewrite rule should process

Substitution rules can operate on any soft macros, for example MESSAGE, PROGRAM, or any user-definedmacros created using parsers. Hard macros cannot be modified. For details on the hard and soft macros, seeSection 14.1.4, Hard vs. soft macros (p. 382)). You can also rewrite the structured-data fields of messagescomplying to the RFC5424 (IETF-syslog) message format. Substitution rules use the following syntax:

Declaration:

rewrite <name_of_the_rule> {

subst("<string or regular expression to find>",

"<replacement string>", value(<field name>), flags() );

};

The type() and flags() options are optional. The type() specifies the type of regular expression to use,while the flags() are the flags of the regular expressions. For details on regular expressions, see Section 14.3,Regular expressions (p. 403).

A single substitution rule can include multiple substitutions that are applied sequentially to the message. Notethat rewriting rules must be included in the log statement to have any effect.

TipFor case-insensitive searches, add the flags(ignore-case) option. To replace every occurrence of the string, addflags(global) option.

399syslog-ng.com

Replacing message parts

Example 14.13. Using substitution rulesThe following example replaces the IP in the text of the message with the string IP-Address.

rewrite r_rewrite_subst{subst("IP", "IP-Address", value("MESSAGE"));};

To replace every occurrence, use:

rewrite r_rewrite_subst{

subst("IP", "IP-Address", value("MESSAGE"), flags("global"));

};

Multiple substitution rules are applied sequentially. The following rules replace the first occurrence of the string IP withthe string IP-Addresses.

rewrite r_rewrite_subst{

subst("IP", "IP-Address", value("MESSAGE"));

subst("Address", "Addresses", value("MESSAGE"));

};

Example 14.14. Anonymizing IP addressesThe following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

rewrite pseudonymize_ip_addresses_in_message {

subst (

"((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))",

"$(sha1 $0)",

value("MSG"),

flags(global)

);

};

14.2.2. Setting message fields to specific values

To set a field of the message to a specific value, you have to:

■ define the string to include in the message, and

■ select the field where it should be included.

You can set the value of available macros, for example HOST, MESSAGE, PROGRAM, or any user-definedmacros created using parsers (for details, seeChapter 15, Parsing and segmenting structured messages (p. 407)andChapter 16, Processing message content with a pattern database (p. 418)). Hard macros cannot be modified.For details on the hard and soft macros, see Section 14.1.4, Hard vs. soft macros (p. 382)). Note that the rewriteoperation completely replaces any previous value of that field. Use the following syntax:

Declaration:

rewrite <name_of_the_rule> {

set("<string to include>", value(<field name>));

};

Example 14.15. Setting message fields to a particular valueThe following example sets the HOST field of the message to myhost.

rewrite r_rewrite_set{set("myhost", value("HOST"));};

The following example appends the "suffix" string to the MESSAGE field:

rewrite r_rewrite_set{set("$MESSAGE suffix", value("MESSAGE"));};

400syslog-ng.com

Setting message fields to specific values

For details on rewriting SDATA fields, see Section 14.2.3, Creating custom SDATA fields (p. 401).

14.2.3. Creating custom SDATA fields

If you use RFC5424-formatted (IETF-syslog) messages, you can also create custom fields in the SDATA partof the message (For details on the SDATA message part, see Section 2.10.2.3, The STRUCTURED-DATAmessage part (p. 23)). According to RFC5424, the name of the field (its SD-ID) must not contain the @ characterfor reserved SD-IDs. Custom SDATA fields must be in the following format: .SDATA.group-name@<privatee n t e r p r i s e n u m b e r > . f i e l d - n a m e , f o r e x a m p l e ,.SDATA.mySDATA-field-group@18372.4.mySDATA-field. (18372.4 is the private enterprise number ofOne Identity LLC, the developer of syslog-ng PE.)

Example 14.16. Rewriting custom SDATA fieldsThe following example sets the sequence ID field of the RFC5424-formatted (IETF-syslog) messages to a fixed value.This field is a predefined SDATA field with a reserved SD-ID, therefore its name does not contain the @ character.

rewrite r_sd {

set("55555" value(".SDATA.meta.sequenceId"));

};

The next example creates a new SDATA field-group and field called custom and sourceip, respectively:

rewrite r_rewrite_set {

set("${SOURCEIP}" value(".SDATA.custom@18372.4.sourceip"));

};

If you use the ${.SDATA.custom@18372.4.sourceip} macro in a template or SQL table, its value will be that of theSOURCEIP macro (as seen on the machine where the SDATA field was created) for every message that was processedwith this rewrite rule, and empty for every other message.

You can verify whether or not the format is correct by looking at the actual network traffic. The SDATA field-group willbe called custom@18372.4, and sourceip will become a field within that group. If you decide to set up several fields,they will be listed in consecutive order within the field-group's SDATA block.

NoteWhen using STRUCTURED-DATA macros, consider the following:

When referencing an element of the structured data, the macro must begin with the dot (.) character. Forexample, ${.SDATA.timeQuality.isSynced}.

■

■ The SDID and SDNAME parts of the macro names are case sensitive:${.SDATA.timeQuality.isSynced} is not the same as ${.SDATA.TIMEQUALITY.ISSYNCED}.

14.2.4. Setting multiple message fields to specific values

The groupset() rewrite rule allows you to modify the value of multiple message fields at once, for example,to change the value of sensitive fields extracted using patterndb, or received in a JSON format.

■ The first parameter is the new value of the modified fields. This can be a simple string, a macro, ora template (which can include template functions as well).

■ The second parameter (values()) specifies the fields to modify. You can explicitly list the macrosor fields (a space-separated list with the values enclosed in double-quotes), or use wildcards andglob expressions to select multiple fields.

401syslog-ng.com

Creating custom SDATA fields

■ Note that groupset() does not create new fields, it only modifies existing fields.

■ You can refer to the old value of the field using the $_ macro. This is resolved to the value of thecurrent field, and is available only in groupset() rules.

Declaration:

rewrite <name_of_the_rule> {

groupset("<new-value-of-the-fields>", values("<field-name-or-glob>"

["<another-field-name-or-glob>"]));

};

Example 14.17. Using groupset rewrite rulesThe following examples show how to change the values of multiple fields at the same time.

Change the value of the HOST field to myhost.

groupset ("myhost" values("HOST"))

■

■ Change the value of the HOST and FULLHOST fields to myhost.

groupset ("myhost" values("HOST" "FULLHOST"))

■ Change the value of the HOST FULLHOST and fields to lowercase.

groupset ("$(lowercase "$_")" values("HOST" "FULLHOST"))

■ Change the value of each field and macro that begins with .USER to nobody.

groupset ("nobody" values(".USER.*"))

■ Change the value of each field and macro that begins with .USER to its SHA-1 hash (truncated to 6characters).

groupset ("$(sha1 --length 6 $_)" values(".USER.*"))

14.2.5. Conditional rewrites

Starting with 4 F1, it is possible to apply a rewrite rule to a message only if certain conditions are met. Thecondition() option effectively embeds a filter expression into the rewrite rule: the message is modified onlyif the message passes the filter. If the condition is not met, the message is passed to the next element of the logpath (that is, the element following the rewrite rule in the log statement, for example, the destination). Anyfilter expression normally used in filters can be used as a rewrite condition. Existing filter statements can bereferenced using the filter() function within the condition. For details on filters, see Section 8.5,Filters (p. 324).

TipUsing conditions in rewrite rules can simplify your syslog-ng PE configuration file, as you do not need to create separatelog paths to modify certain messages.

402syslog-ng.com

Conditional rewrites

14.2.5.1. Procedure – How conditional rewriting works

Purpose:

The following procedure summarizes how conditional rewrite rules (rewrite rules that have the condition()parameter set) work. The following configuration snippet is used to illustrate the procedure:

rewrite r_rewrite_set{set("myhost", value("HOST")

condition(program("myapplication")));};

log {

source(s1);

rewrite(r_rewrite_set);

destination(d1);};

Steps:

Step 1. The log path receives a message from the source (s1).

Step 2. The rewrite rule (r_rewrite_set) evaluates the condition. If the message matches the condition (thePROGRAM field of the message is "myapplication"), syslog-ng PE rewrites the log message (sets thevalue of the HOST field to "myhost"), otherwise it is not modified.

Step 3. The next element of the log path processes the message (d1).

Example 14.18. Using conditional rewritingThe following example sets the HOST field of the message to myhost only if the message was sent by the myapplicationprogram.

rewrite r_rewrite_set{set("myhost", value("HOST") condition(program("myapplication")));};

The following example is identical to the previous one, except that the condition references an existing filter template.

filter f_rewritefilter {program("myapplication");};

rewrite r_rewrite_set{set("myhost", value("HOST") condition(filter(f_rewritefilter)));};

14.3. Regular expressions

Filters and substitution rewrite rules can use regular expressions. In regular expressions, the characters()[].*?+^$|\ are used as special symbols. Depending on how you want to use these characters and whichquotation mark you use, these characters must be used differently, as summarized below.

■ Strings between single quotes ('string') are treated literally and are not interpreted at all, you donot have to escape special characters. For example the output of '\x41' is \x41 (characters asfollows: backslash, x(letter), 4(number), 1(number)). This makes writing and reading regularexpressions much more simple: it is recommended to use single quotes when writing regularexpressions.

■ When enclosing strings between double-quotes ("string"), the string is interpreted and you haveto escape special characters, that is, to precede them with a backslash (\) character if they are meantliterally. For example the output of the "\x41" is simply the letter a. Therefore special characterslike \(backslash) or "(quotation mark) must be escaped (\\ and \"). The following expressions areinterpreted: \a, \n, \r, \t, \v. For example, the \$40 expression matches the $40 string. Backslasheshave to be escaped as well if they are meant literally, for example, the \\d expression matches the\d string.

403syslog-ng.com

Regular expressions

TipIf you use single quotes, you do not need to escape the backslash, for example match("\\.") is equivalentto match('\.').

■ Enclosing alphanumeric strings between double-quotes ("string") is not necessary, you can justomit the double-quotes. For example when writing filters, match("sometext") andmatch(sometext) will both match for the sometext string.

NoteOnly strings containing alphanumerical characters can be used without quotes or double quotes. If thestring contains whitespace or any special characters (()[].*?+^$|\ or ;:#), you must use quotes ordouble quotes.

When using the ;:# characters, you must use quotes or double quotes, but escaping them is not required.

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add theflags(ignore-case) option to the regular expression.

filter demo_regexp_insensitive { host("system" flags(ignore-case)); };

The regular expressions can use up to 255 regexp matches (${1} ... ${255}), but only from the last filterand only if the flags("store-matches") flag was set for the filter. For case-insensitive searches, use theflags("ignore-case") option.

14.3.1. Types and options of regular expressions

By default, syslog-ng uses POSIX-style regular expressions. To use other expression types, add the type()

option after the regular expression.

The syslog-ng PE application supports the following expression types:

■ POSIX regular expressions

■ Perl Compatible Regular Expressions (PCRE)

■ Literal string searches

■ Glob patterns without regular expression support

posixDescription: Use POSIX regular expressions. If the type() parameter is not specified, syslog-ng uses POSIXregular expressions by default.

Posix regular expressions have the following flag options:

global: Usable only in rewrite rules: match for every occurrence of the expression, not only the first one.

ignore-case: Disable case-sensitivity.

404syslog-ng.com

Types and options of regular expressions

store-matches: Store the matches of the regular expression into the $0, ... $255 variables. The $0 storesthe entire match, $1 is the first group of the match (parentheses), and so on. Matches from the last filter expressioncan be referenced in regular expressions.

utf8: Use UTF-8 matching.

Example 14.19. Using Posix regular expressions

filter f_message { message("keyword" flags("utf8" "ignore-case") ); };

pcreDescription:Use Perl Compatible Regular Expressions (PCRE). Starting with syslog-ng PE version 3.1, PCREexpressions are supported on every platform.

PCRE regular expressions have the following flag options:

global: Usable only in rewrite rules: match for every occurrence of the expression, not only the first one.

ignore-case: Disable case-sensitivity.

store-matches: Store the matches of the regular expression into the $0, ... $255 variables. The $0 storesthe entire match, $1 is the first group of the match (parentheses), and so on. Named matches (also called namedsubpatterns), for example (?<name>...), are stored as well. Matches from the last filter expression can bereferenced in regular expressions.

unicode: Use Unicode support for UTF-8 matches: UTF-8 character sequences are handled as single characters.

utf8: An alias for the unicode flag.

Example 14.20. Using PCRE regular expressions

rewrite r_rewrite_subst

{subst("a*", "?", value("MESSAGE") type("pcre") flags("utf8" "global")); };

stringDescription: Match the strings literally, without regular expression support. By default, only identical stringsare matched. For partial matches, use the flags("prefix") or the flags("substring") flags.

globDescription: Match the strings against a pattern containing '*' and '?' wildcards, without regular expressionand character range support. The advantage of glob patterns to regular expressions is that globs can be processedmuch faster.

* matches an arbitrary string, including an empty string

? matches an arbitrary character

405syslog-ng.com

Types and options of regular expressions

NoteThe wildcards can match the / character.■

■ You cannot use the * and ? literally in the pattern.

14.3.2. Optimizing regular expressions

The host(), match(), and program() filter functions and some other syslog-ng objects accept regularexpressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which cancause problems when the message traffic is very high. Often the regular expression can be replaced with simplefilter functions and logical operators. Using simple filters and logical operators, the same effect can be achievedat a much lower CPU load.

Example 14.21. Optimizing regular expressions in filtersSuppose you need a filter that matches the following error message logged by the xntpd NTP daemon:

xntpd[1567]: time error -1159.777379 is too large (set clock manually);

The following filter uses regular expressions and matches every instance and variant of this message.

filter f_demo_regexp {

program("demo_program") and

match("time error .* is too large .* set clock manually"); };

Segmenting the match() part of this filter into separate match() functions greatly improves the performance of thefilter.

filter f_demo_optimized_regexp {

program("demo_program") and

match("time error") and

match("is too large") and

match("set clock manually"); };

406syslog-ng.com

Optimizing regular expressions

Chapter 15. Parsing and segmenting structuredmessages

The filters and default macros of syslog-ng work well on the headers and metainformation of the log messages,but are rather limited when processing the content of the messages. Parsers can segment the content of themessages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering orother type of processing of the message can use these custom macros to refer to parts of the message. Parsersare global objects most often used together with filters and rewrite rules.

syslog-ng PE provides the following possibilities to parse the messages, or parts of the messages:

■ To segment a message into columns using a CSV-parser, see Section 15.1, Parsing messages withcomma-separated and similar values (p. 407).

■ To segment a message consisting of whitespace or comma-separated key=value pairs (for example,Postfix log messages), see Section 15.2, Parsing key=value pairs (p. 412).

■ To parse JSON-formatted messages, see Section 15.3, The JSON parser (p. 414).

■ To identify and parse the messages using a pattern database, see Chapter 16, Processing messagecontent with a pattern database (p. 418).

15.1. Parsing messages with comma-separated and similar values

The syslog-ng PE application can separate parts of log messages (that is, the contents of the ${MSG} macro)at delimiter characters or strings to named fields (columns). One way to achieve this is to use a csv(comma-separated-values) parser (for other methods and possibilities, see the other sections of Chapter 15,Parsing and segmenting structured messages (p. 407). The parsed fields act as user-defined macros that can bereferenced in message templates, file- and tablenames, and so on.

Parsers are similar to filters: they must be defined in the syslog-ng PE configuration file and used in the logstatement. You can also define the parser inline in the log path.

NoteThe order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create a csv-parser(), you have to define the columns of the message, the separator characters or strings(also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used toescape the delimiter characters (quote-pairs()).

Declaration:

parser <parser_name> {

csv-parser(

407syslog-ng.com

Parsing messages with comma-separated and similar values

columns(column1, column2, ...)

delimiters(chars("<delimiter_characters>"), strings("<delimiter_string1>"))

);

};

Column names work like macros.

Names starting with a dot (for example, .example) are reserved for use by syslog-ng PE. If you use such amacro name as the name of a parsed value, it will attempt to replace the original value of the macro (note thatonly soft macros can be overwritten, see Section 14.1.4, Hard vs. soft macros (p. 382) for details).

Example 15.1. Segmenting hostnames separated with a dashThe following example separates hostnames like example-1 and example-2 into two parts.

parser p_hostname_segmentation {

csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID")

delimiters("-")

flags(escape-none)

template("${HOST}"));

};

destination d_file { file("/var/log/messages-${HOSTNAME.NAME:-examplehost}"); };

log { source(s_local); parser(p_hostname_segmentation); destination(d_file);};

Example 15.2. Parsing Apache log filesThe following parser processes the log of Apache web servers and separates them into different fields. Apache logmessages can be formatted like:

"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v"

Here is a sample message:

192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708

"-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5"

2 example.balabit

To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces betweenquotes and brackets are ignored (quote-pairs('""[]')).

parser p_apache {

csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",

"APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",

"APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",

"APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")

flags(escape-double-char,strip-whitespace)

delimiters(" ")

quote-pairs('""[]')

);

};

The results can be used for example to separate log messages into different files based on the APACHE.USER_NAMEfield. If the field is empty, the nouser name is assigned.

log { source(s_local);

parser(p_apache); destination(d_file);};

};

destination d_file { file("/var/log/messages-${APACHE.USER_NAME:-nouser}"); };

Example 15.3. Segmenting a part of a messageMultiple parsers can be used to split a part of an already parsed message into further segments. The following examplesplits the timestamp of a parsed Apache log message into separate fields.

parser p_apache_timestamp {

csv-parser(columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH",

408syslog-ng.com

Parsing messages with comma-separated and similar values

"APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN",

"APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.ZONE")

delimiters("/: ")

flags(escape-none)

template("${APACHE.TIMESTAMP}"));

};

log { source(s_local); parser(p_apache); parser(p_apache_timestamp); destination(d_file);

};

Further examples:

■ For an example on using the greedy option, see Example 15.4, Adding the end of the message tothe last column (p. 411).

15.1.1. Options of CSV parsers

The syslog-ng PE application can separate parts of log messages (that is, the contents of the ${MSG} macro)at delimiter characters or strings to named fields (columns). One way to achieve this is to use a csv(comma-separated-values) parser (for other methods and possibilities, see the other sections of Chapter 15,Parsing and segmenting structured messages (p. 407). The parsed fields act as user-defined macros that can bereferenced in message templates, file- and tablenames, and so on.

Parsers are similar to filters: they must be defined in the syslog-ng PE configuration file and used in the logstatement. You can also define the parser inline in the log path.

NoteThe order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.

To create a csv-parser(), you have to define the columns of the message, the separator characters or strings(also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used toescape the delimiter characters (quote-pairs()).

Declaration:

parser <parser_name> {

csv-parser(

columns(column1, column2, ...)

delimiters(chars("<delimiter_characters>"), strings("<delimiter_string1>"))

);

};

Column names work like macros.

Names starting with a dot (for example, .example) are reserved for use by syslog-ng PE. If you use such amacro name as the name of a parsed value, it will attempt to replace the original value of the macro (note thatonly soft macros can be overwritten, see Section 14.1.4, Hard vs. soft macros (p. 382) for details).

409syslog-ng.com

Options of CSV parsers

columns()columns("PARSER.COLUMN1", "PARSER.COLUMN2", ...)Synopsis:

Description: Specifies the name of the columns to separate messages to. These names will be automaticallyavailable as macros. The values of these macros do not include the delimiters.

delimiters()delimiters(chars("<delimiter_characters>")) or delimiters("<delimiter_characters>")

delimiters(strings("<delimiter_string1>", "<delimiter_string2>", ...)")

delimiters(chars("<delimiter_characters>"), strings("<delimiter_string1>"))

Synopsis:

Description: The delimiter is the character or string that separates the columns in the message. If you specifymultiple characters using the delimiters(chars("<delimiter_characters>")) option, every characterwill be treated as a delimiter. To separate the columns at the tabulator (tab character), specify \t. For example,to separate the text at every hyphen (-) and colon (:) character, use delimiters(chars("-:")), Note thatthe delimiters will not be included in the column values.

If you have to use a string as a delimiter, list your string delimiters in thedelimiters(strings("<delimiter_string1>", "<delimiter_string2>", ...)") format.

If you use more than one delimiter, note the following points:

■ syslog-ng PE will split the message at the nearest possible delimiter. The order of the delimiters inthe configuration file does not matter.

■ You can use both string delimiters and character delimiters in a parser.

■ The string delimiters can include characters that are also used as character delimiters.

■ If a string delimiter and a character delimiter both match at the same position of the message, syslog-ngPE uses the string delimiter.

flags()drop-invalid, escape-none, escape-backslash, escape-double-char, greedy, strip-whitespaceSynopsis:

Description: Specifies various options for parsing the message. The following flags are available:

■ drop-invalid: When the drop-invalid option is set, the parser does not process messages that donot match the parser. For example, a message does not match the parser if it has less columns thanspecified in the parser, or it has more columns but the greedy flag is not enabled. Using thedrop-invalid option practically turns the parser into a special filter, that matches messages thathave the predefined number of columns (using the specified delimiters).

TipMessages dropped as invalid can be processed by a fallback log path. For details on the fallback

option, see Section 8.1.2, Log path flags (p. 314).

410syslog-ng.com

Options of CSV parsers

■ escape-backslash: The parsed message uses the backslash (\) character to escape quote characters.

■ escape-double-char: The parsed message repeats the quote character when the quote character isused literally. For example, to escape a comma (,), the message contains two commas (,,).

■ escape-none: The parsed message does not use any escaping for using the quote character literally.

■ greedy: The greedy option assigns the remainder of the message to the last column, regardless ofthe delimiter characters set. You can use this option to process messages where the number of columnsvaries.

Example 15.4. Adding the end of the message to the last columnIf the greedy option is enabled, the syslog-ng application adds the not-yet-parsed part of the message tothe last column, ignoring any delimiter characters that may appear in this part of the message.

For example, you receive the following comma-separated message: example 1, example2, example3,and you segment it with the following parser:

csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(","));

The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, andexample3, respectively. If the message looks like example 1, example2, example3, some more

information, then any text appearing after the third comma (that is, some more information) is notparsed, and possibly lost if you use only the variables to reconstruct the message (for example, to send itto different columns of an SQL table).

Using the greedy flag will assign the remainder of the message to the last column, so that the COLUMN1,COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some

more information.

csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(",")

flags(greedy));

■ strip-whitespace: The strip-whitespace flag removes leading and trailing whitespaces from allcolumns.

quote-pairs()quote-pairs('<quote_pairs>')Synopsis:

Description: List quote-pairs between single quotes. Delimiter characters or strings enclosed between quotecharacters are ignored. Note that the beginning and ending quote character does not have to be identical, forexample [} can also be a quote-pair. For an example of using quote-pairs() to parse Apache log files, seeExample 15.2, Parsing Apache log files (p. 408).

template()template("${<macroname>}")Synopsis:

Description: The macro that contains the part of the message that the parser will process. It can also be a macrocreated by a previous parser of the log path. By default, this is empty and the parser processes the entire message

411syslog-ng.com

Options of CSV parsers

(${MESSAGE}). For examples, see Example 15.1, Segmenting hostnames separated with a dash (p. 408) andExample 15.3, Segmenting a part of a message (p. 408).

15.2. Parsing key=value pairs

The syslog-ng PE application can separate a message consisting of whitespace or comma-separated key=valuepairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator characterinstead of the equal sign, for example, colon (:) to parse MySQL log messages. For details on using value-pairsin syslog-ng PE see Section 2.12, Structuring macros, metadata, and other value-pairs (p. 25).

You can refer to the separated parts of the message using the key of the value as a macro. For example, if themessage contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.

NoteIf a log message contains the same key multiple times (for example, key1=value1, key2=value2, key1=value3,

key3=value4, key1=value5), then syslog-ng PE stores only the last (rightmost) value for the key. Using the previousexample, syslog-ng PE will store the following pairs: key1=value5, key2=value2, key3=value4..

WarningIf the names of keys in the message is the same as the names of syslog-ng PE soft macros, the value from the parsedmessage will overwrite the value of the macro. For example, the PROGRAM=value1, MESSAGE=value2 content willoverwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.

Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Section 14.1.4, Hardvs. soft macros (p. 382).

The parser discards message sections that are not key=value pairs, even if they appear between key=value pairs thatcan be parsed.

To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional.By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parserinline in the log path.

Declaration:

parser parser_name {

kv-parser(

prefix()

);

};

Example 15.5. Using a key=value parserIn the following example, the source is a log message consisting of comma-separated key=value pairs, for example, aPostfix log message:

Jun 20 12:05:12 mail.example.com <info> postfix/qmgr[35789]: EC2AC1947DA:

from=<me@example.com>, size=807, nrcpt=1 (queue active)

The kv-parser inserts the ".kv." prefix before all extracted name-value pairs. The destination is a file, that uses theformat-json template function. Every name-value pair that begins with a dot (".") character will be written to the file(dot-nv-pairs). The log line connects the source, the destination and the parser.

source s_kv {

network(port(21514));

412syslog-ng.com

Parsing key=value pairs

};

destination d_json {

file("/tmp/test.json"

template("$(format-json --scope dot-nv-pairs)\n"));

};

parser p_kv {

kv-parser (prefix(".kv."));

};

log {

source(s_kv);

parser(p_kv);

destination(d_json);

};

You can also define the parser inline in the log path.

source s_kv {

network(port(21514));

};

destination d_json {

file("/tmp/test.json"

template("$(format-json --scope dot-nv-pairs)\n"));

};

log {

source(s_kv);

parser {

kv-parser (prefix(".kv."));

};

destination(d_json);

};

You can set the separator character between the key and the value to parse for example key:value pairs, like MySQLlogs:

Mar 7 12:39:25 myhost MysqlClient[20824]: SYSTEM_USER:'oscar', MYSQL_USER:'my_oscar',

CONNECTION_ID:23, DB_SERVER:'127.0.0.1', DB:'--', QUERY:'USE test;'

parser p_mysql { kv-parser(value-separator(":") prefix(".mysql."));

15.2.1. Options of key=value parsers

The kv-parser has the following options.

prefix()prefix()Synopsis:

Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing.For example:

■ To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.

■ To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example,${my-parsed-data.name} .

■ If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parseddata into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.

Names starting with a dot (for example, .example) are reserved for use by syslog-ng PE. If you use such amacro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that

413syslog-ng.com

Options of key=value parsers

only soft macros can be overwritten, see Section 14.1.4, Hard vs. soft macros (p. 382) for details). To avoidsuch problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

For example, to insert the postfix prefix when parsing Postfix log messages, use the prefix(.postfix.)option.

template()template("${<macroname>}")Synopsis:

Description: The macro that contains the part of the message that the parser will process. It can also be a macrocreated by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

value-separator()value-separator()Synopsis:

Description: Specifies the character that separates the keys from the values. Default value: =

For example, to parse key:value pairs, use kv-parser(value-separator(":"));

15.3. The JSON parser

JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange.It is used primarily to transmit data between a server and web application, serving as an alternative to XML. Itis described in RFC 4627. The syslog-ng PE application can separate parts of incoming JSON-encoded logmessages to name-value pairs. For details on using value-pairs in syslog-ng PE see Section 2.12, Structuringmacros, metadata, and other value-pairs (p. 25).

You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. Forexample, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as${KEY1} and ${KEY2}. If the JSON content is structured, syslog-ng PE converts it to dot-notation-format. Forexample, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the${KEY1.KEY2} macro.

WarningIf the names of keys in the JSON content are the same as the names of syslog-ng PE soft macros, the value from the JSONcontent will overwrite the value of the macro. For example, the {"PROGRAM":"value1","MESSAGE":"value2"}

JSON content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use theprefix() option.

Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Section 14.1.4, Hardvs. soft macros (p. 382).

NoteThe JSON parser currently supports only integer, double and string values when interpreting JSON structures. As syslog-ngdoes not handle different data types internally, the JSON parser converts all JSON data to string values. In case of booleantypes, the value is converted to 'TRUE' or 'FALSE' as their string representation.

The JSON parser discards messages if it cannot parse them as JSON messages, so it acts as a JSON-filter as well.

414syslog-ng.com

The JSON parser

To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and themarker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To processother parts of a log message with the JSON parser, use the template() option. You can also define the parserinline in the log path.

Declaration:

parser parser_name {

json-parser(

marker()

prefix()

);

};

Example 15.6. Using a JSON parserIn the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that syslog-ng PEdoes not parse the message: flags(no-parse). The json-parser inserts ".json." prefix before all extracted name-valuepairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with adot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and theparser.

source s_json {

network(port(21514) flags(no-parse));

};

destination d_json {

file("/tmp/test.json"

template("$(format-json --scope dot-nv-pairs)\n"));

};

parser p_json {

json-parser (prefix(".json."));

};

log {

source(s_json);

parser(p_json);

destination(d_json);

};

You can also define the parser inline in the log path.

source s_json {

network(port(21514) flags(no-parse));

};

destination d_json {

file("/tmp/test.json"

template("$(format-json --scope dot-nv-pairs)\n"));

};

log {

source(s_json);

parser p_json {

json-parser (prefix(".json."));

};

destination(d_json);

};

15.3.1. Options of JSON parsers

The JSON parser has the following options.

415syslog-ng.com

Options of JSON parsers

extract-prefix()extract-prefix()Synopsis:

Description: Extract only the specified subtree from the JSON message. Use the dot-notation to specify thesubtree. The rest of the message will be ignored. For example, assuming that the incoming object is namedmsg, the json-parser(extract-prefix("foo.bar[5]")); syslog-ng PE parser is equivalent to themsg.foo.bar[5] javascript code. Note that the resulting expression must be a JSON object, so that syslog-ngPE can extract its members into name-value pairs.

This feature also works when the top-level object is an array, because you can use an array index at the firstindirection level, for example: json-parser(extract-prefix("[5]")), which is equivalent to msg[5].

marker()marker()Synopsis:

Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.

Some logging implementations require a marker to be set before the JSON payload. The JSON parser is ableto find these markers and parse the message only if it is present.

Example 15.7. Using the marker option in JSON parserThis json parser parses log messages which use the "@cee:" marker in front of the json payload. It inserts ".cee." infront of the name of name-value pairs, so later on it is easier to find name-value pairs that were parsed using this parser.(For details on selecting name-value pairs, see Section value-pairs() (p. 27).)

parser {

json-parser(

marker("@cee:")

prefix(".cee.")

);

};

prefix()prefix()Synopsis:

Description: Insert a prefix before the name part of the name-value pairs to help further processing. For example,if you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into theSDATA part of the message using the prefix(.SDATA.json.) .

Names starting with a dot (for example, .example) are reserved for use by syslog-ng PE. If you use such amacro name as the name of a parsed value, it will attempt to replace the original value of the macro (note thatonly soft macros can be overwritten, see Section 14.1.4, Hard vs. soft macros (p. 382) for details). To avoidsuch problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)

template()template("${<macroname>}")Synopsis:

416syslog-ng.com

Options of JSON parsers

Description: The macro that contains the part of the message that the parser will process. It can also be a macrocreated by a previous parser of the log path. By default, this is empty and the parser processes the entire message(${MESSAGE}).

417syslog-ng.com

Options of JSON parsers

Chapter 16. Processing message content with apattern database

16.1. Classifying log messages

The syslog-ng application can compare the contents of the received log messages to predefined message patterns.By comparing the messages to the known patterns, syslog-ng is able to identify the exact type of the messages,and sort them into message classes. The message classes can be used to classify the type of the event describedin the log message. The message classes can be customized, and for example can label the messages as userlogin, application crash, file transfer, and so on events.

To find the pattern that matches a particular message, syslog-ng uses a method called longest prefix match radixtree. This means that syslog-ng creates a tree structure of the available patterns, where the different charactersavailable in the patterns for a given position are the branches of the tree.

To classify a message, syslog-ng selects the first character of the message (the text of message, not the header),and selects the patterns starting with this character, other patterns are ignored for the rest of the process. Afterthat, the second character of the message is compared to the second character of the selected patterns. Again,matching patterns are selected, and the others discarded. This process is repeated until a single pattern completelymatches the message, or no match is found. In the latter case, the message is classified as unknown, otherwisethe class of the matching pattern is assigned to the message.

To make the message classification more flexible and robust, the patterns can contain pattern parsers: elementsthat match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimalnumber (for example 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings andIP addresses. For the details of available pattern parsers, see Section 16.5.1, Using pattern parsers (p. 431).

The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to writeand maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is mucheasier to understand syslog-ng pattens than regular expressions.

Pattern matching based on regular expressions is computationally very intensive, especially when the numberof patterns increases. The solution used by syslog-ng can be performed real-time, and is independent from thenumber of patterns, so it scales much better. The following patterns describe the same message: Acceptedpassword for bazsi from 10.50.0.247 port 42156 ssh2

A regular expression matching this message from the logcheck project: Accepted

(gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam)

for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

A syslog-ng database pattern for this message: Accepted @QSTRING:auth_method: @

for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2

For details on using pattern databases to classify log messages, see Section 16.2, Using pattern databases (p. 421).

418syslog-ng.com

Classifying log messages

16.1.1. The structure of the pattern database

The pattern database is organized as follows:

Figure 16.1. The structure of the pattern database

■ The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules:the rules of a ruleset are applied to log messages if the name of the application that sent the messagematches the Program Pattern of the ruleset. The name of the application (the content of the${PROGRAM} macro) is compared to the Program Patterns of the available rulesets, and then therules of the matching rulesets are applied to the message.

■ The Program Pattern can be a string that specifies the name of the appliation or the beginning of itsname (for example, to match for sendmail, the program pattern can be sendmail, or just send), andthe Program Pattern can contain pattern parsers. Note that pattern parsers are completely independentfrom the syslog-ng parsers used to segment messages. Additionally, every rule has a unique identifier:if a message matches a rule, the identifier of the rule is stored together with the message.

■ Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern,but is applied to the message part of the log message (the content of the ${MESSAGE} macro). Ifa message pattern matches the message, the class of the rule is assigned to the message (for example,Security, Violation, and so on).

■ Rules can also contain additional information about the matching messages, such as the descriptionof the rule, an URL, name-value pairs, or free-form tags.

■ Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.

NoteIf the ${PROGRAM} part of a message is empty, rules with an empty Program Pattern are used to classifythe message.

If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and everyrule is used to classify the message. Note that message patterns must be unique within the merged rulesets,but the currently only one ruleset is checked for uniqueness.

419syslog-ng.com

The structure of the pattern database

16.1.2. How pattern matching works

Figure 16.2. Applying patterns

The followings describe how patterns work. This information applies to program patterns and message patternsalike, even though message patterns are used to illustrate the procedure.

Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attemptto parse a sequence of characters according to certain rules.

NoteWildcards and regular expressions cannot be used in patterns. The @ character must be escaped, that is, to match for thischaracter, you have to write @@ in your pattern. This is required because pattern parsers of syslog-ng are enclosed between@ characters.

When a new message arrives, syslog-ng attempts to classify it using the pattern database. The available patternsare organized alphabetically into a tree, and syslog-ng inspects the message character-by-character, startingfrom the beginning. This approach ensures that only a small subset of the rules must be evaluated at any givenstep, resulting in high processing speed. Note that the speed of classifying messages is practically independentfrom the total number of rules.

For example, if the message begins with the Apple string, only patterns beginning with the character A areconsidered. In the next step, syslog-ng selects the patterns that start with Ap, and so on, until there is no morespecific pattern left.

Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matchesthe next character with a literal, and another pattern that would match it with a parser, the pattern with the literalmatch is selected. Using the previous example, if at the third step there is the literal pattern Apport and a patternparser Ap@STRING@, the Apport pattern is matched. If the literal does not match the incoming string (forexample, Apple), syslog-ng attempts to match the pattern with the parser. However, if there are two or moreparsers on the same level, only the first one will be applied, even if it does not perfectly match the message.

If there are two parsers at the same level (for example, Ap@STRING@ and Ap@QSTRING@), it is random whichpattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at leastone character of the message, the other parser is used. But having two different parsers at the same level isextremely rare, so the impact of this limitation is much less than it appears.

16.1.3. Artificial ignorance

Artificial ignorance is a method to detect anomalies. When applied to log analysis, it means that you ignore theregular, common log messages - these are the result of the regular behavior of your system, and therefore arenot too interesting. However, new messages that have not appeared in the logs before can sign important events,and should be therefore investigated. "By definition, something we have never seen before is anomalous"(Marcus J. Ranum).

420syslog-ng.com

How pattern matching works

The syslog-ng application can classify messages using a pattern database: messages that do not match anypattern are classified as unknown. This provides a way to use artificial ignorance to review your log messages.You can periodically review the unknown messages — syslog-ng can send them to a separate destination, andadd patterns for them to the pattern database. By reviewing and manually classifying the unknown messages,you can iteratively classify more and more messages, until only the really anomalous messages show up asunknown.

Obviously, for this to work, a large number of message patterns are required. The radix-tree matching methodused for message classification is very effective, can be performed very fast, and scales very well. Basicallythe time required to perform a pattern matching is independent from the number of patterns in the database.For sample pattern databases, see Section 16.2.2, Downloading sample pattern databases (p. 423).

16.2. Using pattern databases

To classify messages using a pattern database, include a db-parser() statement in your syslog-ng configurationfile using the following syntax:

Declaration:

parser <identifier> {db-parser(file("<database_filename>"));};

Note that using the parser in a log statement only performs the classification, but does not automatically doanything with the results of the classification.

Example 16.1. Defining pattern databasesThe following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.

parser pattern_db {

db-parser(

file("/opt/syslog-ng/var/db/patterndb.xml")

);

};

To apply the patterns on the incoming messages, include the parser in a log statement:

log {

source(s_all);

parser(pattern_db);

destination( di_messages_class);

};

NoteThe default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option ofthe db-parser() statement can be used to specify a different file, thus different db-parser statements can use differentpattern databases. Later versions of syslog-ng will be able to dynamically generate a main database from separate patterndatabase files.

Example 16.2. Using classification resultsThe following destination separates the log messages into different files based on the class assigned to the pattern thatmatches the message (for example Violation and Security type messages are stored in a separate file), and also adds theID of the matching rule to the message:

destination di_messages_class {

file("/var/log/messages-${.classifier.class}"

template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MSG}\n")

421syslog-ng.com

Using pattern databases

template-escape(no)

);

};

For details on how to create your own pattern databases see Section 16.5.2, The syslog-ng pattern databaseformat (p. 433).

16.2.1. Using parser results in filters and templates

The results of message classification and parsing can be used in custom filters and templates, for example, infile and database templates. The following built-in macros allow you to use the results of the classification:

■ The .classifier.classmacro contains the class assigned to the message (for example violation,security, or unknown).

■ The .classifier.rule_id macro contains the identifier of the message pattern that matched themessage.

Example 16.3. Using classification results for filtering messagesTo filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in alog statement.

filter fi_class_violation {

match("violation"

value(".classifier.class")

type("string")

);

};

log {

source(s_all);

parser(pattern_db);

filter(fi_class_violation);

destination(di_class_violation);

};

Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messagesinto a separate file allows you to periodically review new or unknown messages.

To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id

macro. The unique identifier of the rule (for example e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attributeof the rule in the XML database.

filter fi_class_rule {

match("e1e9c0d8-13bb-11de-8293-000c2922ed0a"

value(".classifier.rule_id")

type("string")

);

};

Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using thetags() filter function.

422syslog-ng.com

Using parser results in filters and templates

NoteThe syslog-ng PE application automatically adds the class of the message as a tag using the.classifier.<message-class> format. For example, messages classified as "system" receive the.classifier.system tag. Use the tags() filter function to select messages of a specific class.

filter f_tag_filter {tags(".classifier.system");};

The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this,you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed valueof the message.

Example 16.4. Using pattern parsers as macrosFor example, you want to parse messages of an application that look like "Transaction: <type>.", where <type>is a string that has different values (for example refused, accepted, incomplete, and so on). To parse these messages, youcan use the following pattern:

'Transaction: @ESTRING::.@'

Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or afilename template, include a name in the parser of the pattern, for example:

'Transaction: @ESTRING:TRANSACTIONTYPE:.@'

After that, add a custom template to the log path that uses this template. For example, to select every accepted transaction,use the following custom filter in the log path:

match("accepted" value("TRANSACTIONTYPE"));

NoteThe above macros can be used in database columns and filename templates as well, if you create custom templates forthe destination or logspace.

Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.

16.2.2. Downloading sample pattern databases

To simplify the building of pattern databases, Balabit has released (and will continue to release) sample databases.You can download sample pattern databases from the Balabit GitHub page.

Note that these pattern databases are only samples and experimental databases. They are not officially supported,and may or may not work in your environment.

The syslog-ng pattern databases are available under the Creative Commons Attribution-Share Alike 3.0 (CCby-SA) license. This includes every pattern database written by community contributors or the Balabit staff. Itmeans that:

■ You are free to use and modify the patterns for your needs.

■ If you redistribute the pattern databases, you must distribute your modifications under the samelicense.

■ If you redistribute the pattern databases, you must make it obvious that the source of the originalsyslog-ng pattern databases is the Balabit GitHub page.

For legal details, the full text of the license is available here.

423syslog-ng.com

Downloading sample pattern databases

If you create patterns that are not available in the GitHub repository, consider sharing them with us and thesyslog-ng community, and send them to the syslog-ng mailing list, or to the following e-mailaddress:<patterndb@balabit.com>

16.3. Correlating log messages

The syslog-ng PE application is able to correlate log messages identified using pattern databases.

Log messages are supposed to describe events, but applications often separate information about a single eventinto different log messages. For example, the Postfix e-mail server logs the sender and recipient addresses intoseparate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log messageabout the authentication failure, and the reason of the failure in the next message.

Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages,and so on.

To correlate log messages, syslog-ng PE uses the pattern database to add messages into message-groups calledcontexts. A context consists of a series of log messages that are related to each other in some way, for example,the log messages of an SSH session can belong to the same context. As new messages come in, they may beadded to a context. Also, when an incoming message is identified it can trigger actions to be performed, forexample, generate a new message that contains all the important information that was stored previously in thecontext. (For details on triggering actions and generating messages, see Section 16.4, Triggering actions foridentified messages (p. 426).)

There are two attributes for pattern database rules that determine if a message matching the rule is added to acontext: context-scope and context-id. The context-scope attribute acts as an early filter, selectingmessages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application(${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the contextspecified in the id. The context-id can be a simple string, or can contain macros or values extracted fromthe log messages for further filtering.

NoteMessage contexts are persistent and are not lost when syslog-ng PE is reloaded (SIGHUP), but are lost when syslog-ngPE is restarted.

Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored,that is, how long syslog-ng PE waits for related messages to arrive. Note the following points about timeoutvalues:

■ When a new message is added to a context, syslog-ng PE will restart the timeout using thecontext-timeout set for the new message.

■ When calculating if the timeout has already expired or not, syslog-ng PE uses the timestamps of theincoming messages, not system time elapsed between receiving the two messages (unless the messagesdo not include a timestamp, or the keep-timestamp(no) option is set). That way syslog-ng PEcan be used to process and correlate already existing log messages offline. However, the timestampsof the messages must be in chronological order (that is, a new message cannot be older than the one

424syslog-ng.com

Correlating log messages

already processed), and if a message is newer than the current system time (that is, it seems to becoming from the future), syslog-ng PE will replace its timestamp with the current system time.

Example 16.5. How syslog-ng PE calculates context-timeoutConsider the following two messages:

<38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage

<38>1990-01-01T14:46:25 customhostname program6[1234]: program6 testmessage

If the context-timeout is 10 seconds and syslog-ng PE receives the messages within 1 sec, the timeoutevent will occour immediately, because the difference of the two timestamp (60 sec) is larger than thetimeout value (10 sec).

■ Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts formany messages can require considerable memory. For example, if two related messages usuallyarrive within seconds, it is not needed to set the timeout to several hours.

Example 16.6. Using message correlation

<rule xml:id="..." context-id="ssh-session" context-timeout="86400" context-scope="process">

<patterns>

<pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username:

@from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern>

</patterns>

...

</rule>

For details on configuring message correlation, see the description of the context-id (p. 437),context-timeout (p. 437), and context-scope (p. 437) attributes of pattern database rules.

16.3.1. Referencing earlier messages of the context

When using the <value> element in pattern database rules together with message correlation, you can alsorefer to fields and values of earlier messages of the context by adding the@<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if thereare three log messages in a context, and you are creating a generated message for the third log message, the${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2

expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of thefirst message, and so on. For example, the following message can be created from SSH login/logout messages(for details on generating new messages, see Section 16.4, Triggering actions for identified messages (p. 426)):An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session

lasted from ${DATE}@2 to ${DATE}.

WarningWhen referencing an earlier message of the context, always enclose the field name between braces, for example, ${PID}@3.The reference will not work if you omit the braces.

425syslog-ng.com

Referencing earlier messages of the context

NoteTo use a literal @ character in a template, use @@.

Example 16.7. Referencing values from an earlier messageThe following action can be used to log the length of an SSH session (the time difference between a login and a logoutmessage in the context):

<actions>

<action>

<message>

<values>

<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from

${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>

</values>

</message>

</action>

</actions>

16.4. Triggering actions for identified messages

The syslog-ng PE application is able to generate (trigger) messages automatically if certain events occur, forexample, a specific log message is received, or the correlation timeout of a message expires. Basically, you candefine messages for every pattern database rule that are emitted when a message matching the rule is received.Triggering messages is often used together with message correlation, but can also be used separately.

The generated message is injected into the same place where the db-parser() statement is referenced in thelog path. To post the generated message into the internal() source instead, use the inject-mode() optionin the definition of the parser.

Example 16.8. Sending triggered messages to the internal() sourceTo send the generated messages to the internal source, use the inject-mode(internal) option:

parser p_db {db-parser(

file("mypatterndbfile.xml")

inject-mode(internal)

);};

To inject the generated messages where the pattern database is referenced, use the inject-mode(pass-through)

option:

parser p_db {db-parser(

file("mypatterndbfile.xml")

inject-mode(pass-through)

);};

The generated message must be configured in the pattern database rule. It is possible to create an entire message,use macros and values extracted from the original message with pattern database, and so on.

Example 16.9. Generating messages for pattern database matchesWhen inserted in a pattern database rule, the following example generates a message when a message matching the ruleis received.

<actions>

<action>

<message>

426syslog-ng.com

Triggering actions for identified messages

<values>

<value name="MESSAGE">A log message from ${HOST} matched rule number

$.classifier.rule_id</value>

</values>

</message>

</action>

</actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the<message> element to TRUE. That way the triggering log message is cloned, including name-value pairs andtags. If you set any values for the message in the <action> element, they will override the values of the originalmessage.

Example 16.10. Generating messages with inherited valuesThe following action generates a message that is identical to the original message, but its $PROGRAM field is set tooverriding-original-program-name

<actions>

<action>

<message inherit-properties='TRUE'>

<values>

<value name="PROGRAM">overriding-original-program-name</value>

</values>

</message>

</action>

</actions>

For details on configuring actions, see the description of the pattern database format.

16.4.1. Conditional actions

To limit when a message is triggered, use the condition attribute and specify a filter expression: the actionwill be executed only if the condition is met. For example, the following action is executed only if the messagewas sent by the host called myhost.

<action condition="'${HOST}' == 'example'">

You can use the same operators in the condition that can be used in filters. For details, see Section 8.5.3,Comparing macro values in filters (p. 326).

The following action can be used to log the length of an SSH session (the time difference between a login anda logout message in the context):

<actions>

<action>

<message>

<values>

<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from

${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>

</values>

</message>

</action>

</actions>

427syslog-ng.com

Conditional actions

Example 16.11. Actions based on the number of messagesThe following example triggers different actions based on the number of messages in the context. This way you can checkif the context contains enough messages for the event to be complete, and execute a different action if it does not.

<actions>

<action condition='"$(context-length)" >= "4"'>

<message>

<values>

<value name="PROGRAM">event</value>

<value name="MESSAGE">Event complete</value>

</values>

</message>

</action>

<action condition='"$(context-length)" < "4"'>

<message>

<values>

<value name="PROGRAM">error</value>

<value name="MESSAGE">Error detected</value>

</values>

</message>

</action>

</actions>

16.4.2. External actions

To perform an external action when a message is triggered, for example, to send the message in an e-mail, youhave to route the generated messages to an external application using the program() destination.

Example 16.12. Sending triggered messages to external applicationsThe following sample configuration selects the triggered messages and sends them to an external script.

Set a field in the triggered message that is easy to identify and filter. For example:

<values>

<value name="MESSAGE">A log message from ${HOST} matched rule number

1.

$.classifier.rule_id</value>

<value name="TRIGGER">yes</value>

</values>

2. Create a destination that will process the triggered messages.

destination d_triggers { program("/bin/myscript"; ); };

3. Create a filter that selects the triggered messages from the internal source.

filter f_triggers {match("yes" value ("TRIGGER") type(string));};

4. Create a log path that selects the triggered messages from the internal source and sends them to the script:

log { source(s_local); filter(f_triggers); destination(d_triggers); };

5. Create a script that will actually process the generated messages, for example:

#!/usr/bin/perl

while (<>) {

# body of the script to send emails, snmp traps, and so on

}

428syslog-ng.com

External actions

16.4.3. Actions and message correlation

Certain features of generating messages can be used only if message correlation is used as well. For details oncorrelating messages, see Section 16.3, Correlating log messages (p. 424).

■ The syslog-ng PE application automatically fills the fields for the generated message based on thescope of the context, for example, the HOST and PROGRAM fields if the context-scope isprogram.

■ When used together with message correlation, you can also refer to fields and values of earlierm e s s a g e s o f t h e c o n t e x t b y a d d i n g t h e@<distance-of-referenced-message-from-the-current> suffix to the macro. For details,see Section 16.3.1, Referencing earlier messages of the context (p. 425).

Example 16.13. Referencing values from an earlier messageThe following action can be used to log the length of an SSH session (the time difference between a loginand a logout message in the context):

<actions>

<action>

<message>

<values>

<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from

${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>

</values>

</message>

</action>

</actions>

■ You can use the name-value pairs of other messages of the context. If you set theinherit-properties attribute of the generated message to context, syslog-ng PE collects everyname-value pair from each message stored in the context, and includes them in the generated message.This means that you can refer to a name-value pair without having to know which message of thecontext included it. If a name-value pair appears in multiple messages of the context, the value inthe latest message will be used. To refer to an earlier value, use the@<distance-of-referenced-message-from-the-current> suffix format.

<action>

<message inherit-properties='context'>

Example 16.14. Using the inherit-properties optionFor example, if inherit-properties is set to context, and you have a rule that collects SSH loginand logout messages to the same context, you can use the following value to generate a message collectingthe most important information form both messages, including the beginning and end date.

<value name="MESSAGE">An SSH session for ${SSH_USERNAME} from

${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid:

$PID.</value>

The following is a detailed rule for this purpose.

<patterndb version='4' pub_date='2015-04-13'>

<ruleset name='sshd' id='12345678'>

<pattern>sshd</pattern>

<rules>

429syslog-ng.com

Actions and message correlation

<!-- The pattern database rule for the first log message -->

<rule provider='me' id='12347598' class='system'

context-id="ssh-login-logout" context-timeout="86400"

context-scope="process">

<!-- Note the context-id that groups together the

relevant messages, and the context-timeout value that

determines how long a new message can be added to the

context -->

<patterns>

<pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for

@ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @ESTRING::

@@ANYSTRING:SSH_SERVICE@</pattern>

<!-- This is the actual pattern used to identify

the log message. The segments between the @

characters are parsers that recognize the variable

parts of the message - they can also be used as

macros. -->

</patterns>

</rule>

<!-- The pattern database rule for the fourth log message -->

<rule provider='me' id='12347599' class='system'

context-id="ssh-login-logout" context-scope="process">

<patterns>

<pattern>pam_unix(sshd:session): session closed for

user @ANYSTRING:SSH_USERNAME@</pattern>

</patterns>

<actions>

<action>

<message inherit-properties='context'>

<values>

<value name="MESSAGE">An SSH session for

${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2

to $DATE pid: $PID.</value>

<value name="TRIGGER">yes</value>

<!-- This is the new log message

that is generated when the logout

message is received. The macros ending

with @2 reference values of the

previous message from the context. -->

</values>

</message>

</action>

</actions>

</rule>

</rules>

</ruleset>

</patterndb>

■ It is possible to generate a message when the context-timeout of the original message expiresand no new message is added to the context during this time. To accomplish this, include thetrigger="timeout" attribute in the action element:

<action trigger="timeout">

Example 16.15. Sending alert when a client disappearsThe following example shows how to combine various features of syslog-ng PE to send an e-mail alert ifa client stops sending messages.

• Configure your clients to send MARK messages periodically. It is enough to configure MARK messagesfor the destination that forwards your log messages to your syslog-ng PE server(mark-mode(periodical)).

• On your syslog-ng PE server, create a pattern database rule that matches on the incoming MARKmessages. In the rule, set the context-scope attribute to host, and the context-timeout attributeto a value that is higher than the mark-freq value set on your clients (by default, mark-freq is 1200

430syslog-ng.com

Actions and message correlation

seconds, so set context-timeout at least to 1500 seconds, but you might want to use a higher value,depending on your environment).

• Add an action to this rule that sends you an e-mail alert if the context-timeout expires, and the serverdoes not receive a new MARK message (<action trigger="timeout">).

• On your syslog-ng PE server, use the pattern database in the log path that handles incoming log messages.

16.5. Creating pattern databases

16.5.1. Using pattern parsers

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers areenclosed between @ characters. The syntax of parsers is the following:

■ a beginning @ character,

■ the type of the parser written in capitals,

■ optionally a name,

■ parameters of the parser, if any, and

■ a closing @ character.

Example 16.16. Pattern parser syntaxA simple parser:

@STRING@

A named parser:

@STRING:myparser_name@

A named parser with a parameter:

@STRING:myparser_name:*@

A parser with a parameter, but without a name:

@STRING::*@

Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: stringfollowed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used:Host:@IPv4@.

NoteNote that using parsers is a CPU-intensive operation. Use the ESTRING and QSTRING parsers whenever possible, asthese can be processed much faster than the other parsers.

431syslog-ng.com

Creating pattern databases

Example 16.17. Using the STRING and ESTRING parsersFor example, if the message is user=joe96 group=somegroup, @STRING:mytext:@ parses only to the firstnon-alphanumeric character (=), parsing only user. @STRING:mytext:=@ parses the equation mark as well, and proceedsto the next non-alphanumeric character (the whitespace), resulting in user=joe96 being parsed. @STRING:mytext:=@will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character,resulting in user=joe96 group=somegroup.

Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@

group=@STRING:group@".

If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameterof the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the nextwhitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".

16.5.1.1. Pattern parsers of syslog-ng PE

The following parsers are available in syslog-ng PE.

@ANYSTRING@Parses everything to the end of the message, you can use it to collect everything that is not parsed specificallyto a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

@DOUBLE@An obsolete alias of the @FLOAT@ parser.

@ESTRING@This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it findsthe stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You canuse the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstringinstead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter,nor can line-breaks or tabs.

@FLOAT@A floating-point number that may contain a dot (.) character. (Up to syslog-ng 3.1, the name of this parser was@DOUBLE@.)

@IPv4@Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

@IPv6@Parses any valid IPv6 IP address.

@IPvANY@Parses any IP address.

@NUMBER@A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading

432syslog-ng.com

Using pattern parsers

hyphen (–) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma)are not. To parse floating-point numbers, use the @FLOAT@ parser.

@QSTRING@Parse a string between the quote characters specified as parameter. Note that the quote character can be differentat the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotationmarks ("), while @QSTRING:&lt;&gt;@ parses from an opening bracket to the closing bracket. The @ charactercannot be a quote character, nor can line-breaks or tabs.

@STRING@A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other acceptedcharacters can be listed as parameters (for example, to parse a complete sentence, add the whitespace asparameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.

16.5.2. The syslog-ng pattern database format

Pattern databases are XML files that contain rules describing the message patterns. For sample pattern databases,see Section 16.2.2, Downloading sample pattern databases (p. 423).

The following scheme describes the V4 format of the pattern database. This format is used by syslog-ng PE 4F1 and later, and is backwards-compatible with the earlier V3 format.

For a sample database containing only a single pattern, see Example 16.18, A V4 pattern database containinga single rule (p. 433).

TipUse the pdbtool utility that is bundled with syslog-ng to test message patterns and convert existing databases to thelatest format. For details, see pdbtool(1) (p. 479).

To automatically create an initial pattern database from an existing log file, use the pdbtool patternize command.For details, see the section called “The patternize command” (p. 482).

Example 16.18. A V4 pattern database containing a single ruleThe following pattern database contains a single rule that matches a log message of the ssh application. A sample logmessage looks like:

Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2

The following is a simple pattern database containing a matching rule.

<patterndb version='4' pub_date='2010-10-17'>

<ruleset name='ssh' id='123456678'>

<pattern>ssh</pattern>

<rules>

<rule provider='me' id='182437592347598' class='system'>

<patterns>

<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @

for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@

ssh2</pattern>

</patterns>

</rule>

</rules>

</ruleset>

</patterndb>

433syslog-ng.com

The syslog-ng pattern database format

Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macrorefer to the username used in the connection.

The following is the same example, but with a test message and test values for the parsers.

<patterndb version='4' pub_date='2010-10-17'>

<ruleset name='ssh' id='123456678'>

<pattern>ssh</pattern>

<rules>

<rule provider='me' id='182437592347598' class='system'>

<patterns>

<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @

for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@

ssh2</pattern>

</patterns>

<examples>

<example>

<test_message>Accepted password for sampleuser from 10.50.0.247

port 42156 ssh2</test_message>

<test_values>

<test_value name="SSH.AUTH_METHOD">password</test_value>

<test_value name="SSH_USERNAME">sampleuser</test_value>

<test_value

name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>

<test_value name="SSH_PORT_NUMBER">42156</test_value>

</test_values>

</example>

</examples>

</rule>

</rules>

</ruleset>

</patterndb>

16.5.2.1. Element: patterndb

Location/patterndb

DescriptionThe container element of the pattern database.

Attributes

■ version: The schema version of the pattern database. The current version is 4.

■ pubdate: The publication date of the XML file.

Children

■ ruleset

Example

<patterndb version='4' pub_date='2010-10-25'>

16.5.2.2. Element: ruleset

Location/patterndb/ruleset

434syslog-ng.com

The syslog-ng pattern database format

DescriptionA container element to group log patterns for an application or program. A <patterndb> element may containany number of <ruleset> elements.

Attributes

■ name: The name of the application. Note that the function of this attribute is to make the databasemore readable, syslog-ng uses the <pattern> element to identify the applications sending logmessages.

■ id: A unique ID of the application, for example, the md5 sum of the name attribute.

■ description: OPTIONAL — A description of the ruleset or the application.

■ url: OPTIONAL — An URL referring to further information about the ruleset or the application.

Children

■ patterns

■ rules

■ actions

■ tags

Example

<ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>

16.5.2.3. Element: patterns

Location/patterndb/ruleset/patterns

DescriptionA container element. A <patterns> element may contain any number of <pattern> elements.

AttributesN/A

Children

■ pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} headerof the syslog message to find the rulesets applicable to the syslog message.Specifying multiple patterns is useful if two or more applications have different names (that is,different ${PROGRAM} fields), but otherwise send identical log messages.

It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different,use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix e-mailserver sends messages using different process names, but all of them begin with the postfix string.

435syslog-ng.com

The syslog-ng pattern database format

NoteIf the <pattern> element of a ruleset is not specified, syslog-ng PE will use this ruleset as a fallbackruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of theprogram patterns matched the PROGRAM header of the incoming message.

Example

<patterns>

<pattern>firstapplication</pattern>

<pattern>otherapplication</pattern>

</patterns>

16.5.2.4. Element: rules

Location/patterndb/ruleset/rules

DescriptionA container element for the rules of the ruleset.

AttributesN/A

Children

■ rule

Example

<rules>

<rule provider='me' id='182437592347598' class='system'>

<patterns>

<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME:

@from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>

</patterns>

</rule>

</rules>

16.5.2.5. Element: rule

Location/patterndb/ruleset/rules/rule

DescriptionAn element containing message patterns and how a message that matches these patterns is classified.

436syslog-ng.com

The syslog-ng pattern database format

NoteIf the following characters appear in the message, they must be escaped in the rule as follows:

@: Use @@, for example user@@example.com■

■ <: Use &lt;

■ >: Use &gt;

■ &: Use &amp;

The <rules> element may contain any number of <rule> elements.

Attributes

■ provider: The provider of the rule. This is used to distinguish between who supplied the rule, thatis, if it has been created by One Identity, or added to the XML by a local user.

■ id: The globally unique ID of the rule.

■ class: The class of the rule — syslog-ng assigns this class to the messages matching a pattern of thisrule.

■ context-id: OPTIONAL — An identifier to group related log messages when using the patterndatabase to correlate events. The ID can be a descriptive string describing the events related to thelog message (for example, ssh-sessions for log messages related to SSH traffic), but can alsocontain macros to generate IDs dynamically. When using macros in IDs, see also thecontext-scope (p. 437) attribute. For details on correlating messages, see Section 16.3, Correlatinglog messages (p. 424).

NoteThe syslog-ng PE application determines the context of the message after the pattern matching is completed.This means that macros and name-value pairs created by the matching pattern database rule can be usedas context-id macros.

■ context-timeout: OPTIONAL — The number of seconds the context is stored. Note that forhigh-traffic log servers, storing open contexts for long time can require significant amount of memory.For details on correlating messages, see Section 16.3, Correlating log messages (p. 424).

■ context-scope: OPTIONAL — Specifies which messages belong to the same context. This attributeis used to determine the context of the message if the context-id (p. 437) does not specify anymacros. Usually, context-scope acts a filter for the context, with context-id refining the filteringif needed. The context-scope attribute has the following possible values:

• process: Only messages that are generated by the same process of a client belong to the samecontext, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values. Thisis the default behavior of syslog-ng PE if context-scope is not specified.

• program: Messages that are generated by the same application of a client belong to the samecontext, that is, messages that have identical ${HOST} and ${PROGRAM} values.

• host: Every message generated by a client belongs to the same context, only the ${HOST} valueof the messages must be identical.

437syslog-ng.com

The syslog-ng pattern database format

• global: Every message belongs to the same context.

NoteUsing the context-scope attribute is significantly faster than using macros in the context-id attribute.

For details on correlating messages, see Section 16.3, Correlating log messages (p. 424).

Children

■ patterns

Example

<rule provider='balabit' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>

The following example specifies attributes for correlating messages as well. For details on correlating messages,see Section 16.3, Correlating log messages (p. 424).

<rule provider='balabit' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'

context-id='same-session' context-scope='process' context-timeout='360'>

16.5.2.6. Element: patterns

Location/patterndb/ruleset/rules/rule/patterns

DescriptionAn element containing the patterns of the rule. If a <patterns> element contains multiple <pattern> elements,the class of the <rule> is assigned to every syslog message matching any of the patterns.

AttributesN/A

Children

■ pattern: A pattern describing a log message. This element is also called message pattern. Forexample:

<pattern>+ ??? root-</pattern>

NoteSupport for XML entities is limited, you can use only the following entities: &amp; &lt; &gt; &quot;

&apos;. User-defined entities are not supported.

438syslog-ng.com

The syslog-ng pattern database format

■ description: OPTIONAL — A description of the pattern or the log message matching the pattern.

■ urls

■ values

■ examples

Example

<patterns>

<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\

@QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>

</patterns>

16.5.2.7. Element: urls

Location/patterndb/ruleset/rules/rule/patterns/urls

DescriptionOPTIONAL — An element containing one or more URLs referring to further information about the patternsor the matching log messages.

AttributesN/A

Children

■ url: OPTIONAL — An URL referring to further information about the patterns or the matching logmessages.

ExampleN/A

16.5.2.8. Element: values

Location/patterndb/ruleset/rules/rule/patterns/values

DescriptionOPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, therepresentation of the event in the message according to the Common Event Format (CEF) or Common EventExchange (CEE). The names can be used as macros to reference the assigned values.

AttributesN/A

439syslog-ng.com

The syslog-ng pattern database format

Children

■ value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.The <value> element of name-value pairs can include template functions. For details, see Section14.1.6, Using template functions (p. 391), for examples, see Section if (p. 396).

When used together with message correlation, the <value> element of name-value pairs can includereferences to the values of earlier messages from the same context. For details, see Section 16.3,Correlating log messages (p. 424).

■ name: The name of the name-value pair. It can also be used as a macro to reference the assignedvalue.

Example

<values>

<value name=".classifier.outcome">/Success</value>

</values>

16.5.2.9. Element: examples

Location/patterndb/ruleset/rules/rule/patterns/examples

DescriptionOPTIONAL — A container element for sample log messages that should be recognized by the pattern. Thesemessages can be used also to test the patterns and the parsers.

AttributesN/A

Children

■ example

Example

<examples>

<example>

<test_message>Accepted password for sampleuser from 10.50.0.247 port 42156

ssh2</test_message>

<test_values>

<test_value name="SSH.AUTH_METHOD">password</test_value>

<test_value name="SSH_USERNAME">sampleuser</test_value>

<test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>

<test_value name="SSH_PORT_NUMBER">42156</test_value>

</test_values>

</example>

</examples>

440syslog-ng.com

The syslog-ng pattern database format

16.5.2.10. Element: example

Location/patterndb/ruleset/rules/rule/patterns/examples/example

DescriptionOPTIONAL — A container element for a sample log message.

AttributesN/A

Children

■ test_message: OPTIONAL — A sample log message that should match this pattern. For example:

<test_message program="myapplication">Content filter has been

enabled</test_message>

• program: The program pattern of the test message. For example:

<test_message program="proftpd">ubuntu

(::ffff:192.168.2.179[::ffff:192.168.2.179]) - FTP session

closed.</test_message>

■ test_values: OPTIONAL — A container element to test the results of the parsers used in the pattern.

• test_value: OPTIONAL — The expected value of the parser when matching the pattern to thetest message. For example:

<test_value name=".dict.ContentFilter">enabled</test_value>

• name: The name of the parser to test.

Example

<examples>

<example>

<test_message>Accepted password for sampleuser from 10.50.0.247 port 42156

ssh2</test_message>

<test_values>

<test_value name="SSH.AUTH_METHOD">password</test_value>

<test_value name="SSH_USERNAME">sampleuser</test_value>

<test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>

<test_value name="SSH_PORT_NUMBER">42156</test_value>

</test_values>

</example>

</examples>

441syslog-ng.com

The syslog-ng pattern database format

16.5.2.11. Element: actions

Location/patterndb/ruleset/actions

DescriptionOPTIONAL — A container element for actions that are performed if a message is recognized by the pattern.For details on actions, see Section 16.4, Triggering actions for identified messages (p. 426).

AttributesN/A

Children

■ action

Example

Example 16.19. Generating messages for pattern database matchesWhen inserted in a pattern database rule, the following example generates a message when a message matching the ruleis received.

<actions>

<action>

<message>

<values>

<value name="MESSAGE">A log message from ${HOST} matched rule number

$.classifier.rule_id</value>

</values>

</message>

</action>

</actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the<message> element to TRUE. That way the triggering log message is cloned, including name-value pairs andtags. If you set any values for the message in the <action> element, they will override the values of the originalmessage.

Example 16.20. Generating messages with inherited valuesThe following action generates a message that is identical to the original message, but its $PROGRAM field is set tooverriding-original-program-name

<actions>

<action>

<message inherit-properties='TRUE'>

<values>

<value name="PROGRAM">overriding-original-program-name</value>

</values>

</message>

</action>

</actions>

442syslog-ng.com

The syslog-ng pattern database format

16.5.2.12. Element: action

Location/patterndb/ruleset/actions/action

DescriptionOPTIONAL — A container element describing an action that is performed when a message matching the ruleis received.

Attributes

■ condition: A syslog-ng filter expression. The action is performed only if the message matches thefilter. The filter can include macros and name-value pairs extracted from the message.

■ rate: Specifies maximum how many messages should be generated in the specified time period inthe following format: <number-of-messages>/<period-in-seconds>. For example: 1/60allows 1 message per minute. Rates apply within the scope of the context, that is, ifcontext-scope="host" and rate="1/60", then maximum one message is generated per minutefor every host that sends a log message matching the rule. Excess messages are dropped. Note thatwhen applying the rate to the generated messages, syslog-ng PE uses the timestamps of the logmessages, similarly to calculating the context-timeout. That way rate is applied correctly evenif the log messages are processed offline.

■ trigger: Specifies when the action is executed. The trigger attribute has the following possiblevalues:

• match: Execute the action immediately when a message matching the rule is received.

• timeout: Execute the action when the correlation timer (context-timeout (p. 437)) expires. Thisis available only if actions are used together with correlating messages.

Children

■ message: A container element storing the message to be sent when the action is executed. Currentlysyslog-ng PE sends these messages to the internal() destination.

• inherit-properties: If set to TRUE, the original message that triggered the action is cloned, includingits name-value pairs and tags. For details, see Section 16.4, Triggering actions for identifiedmessages (p. 426).

If set to context, syslog-ng PE collects every name-value pair from each message stored in thecontext, and includes them in the generated message. If a name-value pair appears in multiplemessages of the context, the value in the latest message will be used. Note that tags are not merged,the generated message will inherit the tags assigned to the last message of the context. For detailson the message context, see Section 16.3, Correlating log messages (p. 424) and Section 16.4.3,Actions and message correlation (p. 429).

This option is available in syslog-ng PE 5.3.2 and later.

443syslog-ng.com

The syslog-ng pattern database format

■ values: A container element for values and fields that are used to create the message generated bythe action.

• value: Sets the value of the message field specified in the name attribute of the element. Forexample, to specify the body of the generated message, use the following syntax:

<value name="MESSAGE">A log message matched rule number

$.classifier.rule_id</value>

Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.

When the action is used together with message correlation, the syslog-ng PE applicationautomatically adds fields to the message based on the context-scope parameter. For example,using context-scope="process" automatically fills the HOST, PROGRAM, and PID fieldsof the generated message.

• name: Name of the message field set by the value element.

Example

Example 16.21. Generating messages for pattern database matchesWhen inserted in a pattern database rule, the following example generates a message when a message matching the ruleis received.

<actions>

<action>

<message>

<values>

<value name="MESSAGE">A log message from ${HOST} matched rule number

$.classifier.rule_id</value>

</values>

</message>

</action>

</actions>

To inherit the properties and values of the triggering message, set the inherit-properties attribute of the<message> element to TRUE. That way the triggering log message is cloned, including name-value pairs andtags. If you set any values for the message in the <action> element, they will override the values of the originalmessage.

Example 16.22. Generating messages with inherited valuesThe following action generates a message that is identical to the original message, but its $PROGRAM field is set tooverriding-original-program-name

<actions>

<action>

<message inherit-properties='TRUE'>

<values>

<value name="PROGRAM">overriding-original-program-name</value>

</values>

</message>

</action>

</actions>

444syslog-ng.com

The syslog-ng pattern database format

16.5.2.13. Element: tags

Location/patterndb/ruleset/tags

DescriptionOPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. Thetags can be used to label specific events (for example user logons). It is also possible to filter on these tags later(for details, see Section 8.5.5, Tagging messages (p. 328)). Starting with syslog-ng Premium Edition 3.2, thelist of tags assigned to a message can be referenced with the ${TAGS} macro.

AttributesN/A

Children

■ tag: OPTIONAL — A keyword or tags applied to messages matching the rule.

Example

<tags><tag>UserLogin</tag></tags>

445syslog-ng.com

The syslog-ng pattern database format

Chapter 17. Statistics and metrics of syslog-ng

Periodically, syslog-ng sends a message containing statistics about the received messages, and about any lostmessages since the last such message. It includes a processed entry for every source and destination, listingthe number of messages received or sent, and a dropped entry including the IP address of the server for everydestination where syslog-ng has lost messages. The center(received) entry shows the total number ofmessages received from every configured sources.

The following is a sample log statistics message for a configuration that has a single source (s_local) and anetwork and a local file destination (d_network and d_local, respectively). All incoming messages are sentto both destinations.

Log statistics;

dropped='tcp(AF_INET(192.168.10.1:514))=6439',

processed='center(received)=234413',

processed='destination(d_tcp)=234413',

processed='destination(d_local)=234413',

processed='source(s_local)=234413'

Log statistics can be also retrieved on-demand using one of the following options:

■ Using the syslog-ng-ctl stats command.

■ Using the syslog-ng-query *.stats command. Note that the output of the syslog-ng-querycommand is better structured, and also allows you to select which metrics you want to display. Fordetails, see syslog-ng-query(1) (p. 508).

■ U s e t h e socat a p p l i c a t i o n : echo STATS | socat -vv

UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -

■ If you have an OpenBSD-style netcat application installed, use the echo STATS | nc -U

/opt/syslog-ng/var/run/syslog-ng.ctl command. Note that the netcat included in mostLinux distributions is a GNU-style version that is not suitable to query the statistics of syslog-ng.

The statistics include a list of source groups and destinations, as well as the number of processed messages foreach. The verbosity of the statistics can be set using the stats-level() option. For details, see Section 9.2,Global options (p. 334). An example output is shown below.

src.internal;s_all#0;;a;processed;6445

src.internal;s_all#0;;a;stamp;1268989330

destination;df_auth;;a;processed;404

destination;df_news_dot_notice;;a;processed;0

destination;df_news_dot_err;;a;processed;0

destination;d_ssb;;a;processed;7128

destination;df_uucp;;a;processed;0

source;s_all;;a;processed;7128

destination;df_mail;;a;processed;0

destination;df_user;;a;processed;1

destination;df_daemon;;a;processed;1

destination;df_debug;;a;processed;15

destination;df_messages;;a;processed;54

446syslog-ng.com

destination;dp_xconsole;;a;processed;671

dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080

dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128

dst.tcp;d_network#0;10.50.0.111:514;a;stored;2048

destination;df_syslog;;a;processed;6724

destination;df_facility_dot_warn;;a;processed;0

destination;df_news_dot_crit;;a;processed;0

destination;df_lpr;;a;processed;0

destination;du_all;;a;processed;0

destination;df_facility_dot_info;;a;processed;0

center;;received;a;processed;0

destination;df_kern;;a;processed;70

center;;queued;a;processed;0

destination;df_facility_dot_err;;a;processed;0

The statistics are semicolon separated: every line contains statistics for a particular object (for example source,destination, tag, and so on). The statistics have the following fields:

1. The type of the object (for example dst.file, tag, src.facility)

2. The ID of the object used in the syslog-ng configuration file, for example d_internal orsource.src_tcp. The #0 part means that this is the first destination in the destination group.

3. The instance ID (destination) of the object, for example the filename of a file destination, or thename of the application for a program source or destination.

4. The status of the object. One of the following:

■ a - active. At the time of quering the statistics, the source or the destination was still alive (itcontinuously received statistical data).

■ d - dynamic. Such objects may not be continuously available, for example, like statistics basedon the sender's hostname.

■ o - This object was once active, but stopped receiving messages. (For example a dynamic objectmay disappear and become orphan.)

NoteThe syslog-ng PE application stores the statistics of the objects when syslog-ng PE is reloaded. However,if the configuration of syslog-ng PE was changed since the last reload, the statistics of orphaned objectsare deleted.

5. The type of the statistics:

■ processed: The number of messages that successfully reached their destination driver. Notethat this does not necessarily mean that the destination driver successfully delivered the messages(for example, written to disk or sent to a remote server).

■ dropped: The number of dropped messages — syslog-ng PE could not send the messages to thedestination and the output buffer got full, so messages were dropped by the destination driver.

■ stored: The number of messages stored in the message queue of the destination driver, waitingto be sent to the destination.

447syslog-ng.com

■ suppressed: The number of suppressed messages (if the suppress() feature is enabled).

■ stamp: The UNIX timestamp of the last message sent to the destination.

6. The number of such messages.

NoteCertain statistics are available only if the stats-level() option is set to a higher value.

When receiving messages with non-standard facility values (that is, higher than 23), these messages will be listed asother facility instead of their facility number.

To reset the statistics to zero, use the following command: syslog-ng-ctl stats --reset

448syslog-ng.com

Chapter 18. Multithreading and scaling insyslog-ng PE

Starting with version 4 F1, syslog-ng PE can process sources and destinations in multithreaded mode to scaleto multiple CPUs or cores for increased performance. Starting with version 5 F4, this multithreaded mode isthe default.

18.1. Multithreading concepts of syslog-ng PE

This section is a brief overview on how syslog-ng PE works in multithreaded mode. It is mainly for illustrationpurposes: the concept has been somewhat simplified and may not completely match reality.

NoteThe way syslog-ng PE uses multithreading may change in future releases. The current documentation applies to version6 LTS.

syslog-ng PE always uses multiple threads:

■ A main thread that is always running

■ A number of worker threads that process the messages. You can influence the behavior of workerthreads using the threaded() option and the --worker-threads command-line option.

■ Some other, special threads for internal functionalities. For example, certain destinations run in aseparate thread, independently of the multithreading (threaded()) and --worker-threads

settings of syslog-ng PE.

The maximum number of worker threads syslog-ng PE uses is the number of CPUs or cores in the host runningsyslog-ng PE (up to 64). You can limit this value using the --worker-threads command-line option thatsets the maximum total number of threads syslog-ng PE can use, including the main syslog-ng PE thread.However, the --worker-threads option does not affect the supervisor of syslog-ng PE. The supervisor is aseparate process (see syslog-ng(8) (p. 493)), but certain operating systems might display it as a thread. In addition,certain destinations always run in a separate thread, independently of the multithreading (threaded()) and--worker-threads settings of syslog-ng PE.

When an event requiring a new thread occurs (for example, syslog-ng PE receives new messages, or a destinationbecomes available), syslog-ng PE tries to start a new thread. If there are no free threads, the task waits until athread finishes its task and becomes available. There are two types of worker threads:

■ Reader threads read messages from a source (as many as possible, but limited by thelog-fetch-limit() and log-iw-size() options. The thread then processes these messages,that is, performs filtering, rewriting and other tasks as necessary, and puts the log message into thequeue of the destination. If the destination does not have a queue (for example, usertty), the readerthread sends the message to the destination, without the interaction of a separate writer thread.

449syslog-ng.com

Multithreading concepts of syslog-ng PE

■ Writer threads take the messages from the queue of the destination and send them to the destination,that is, write the messages into a file, or send them to the syslog server over the network. The writerthread starts to process messages from the queue only if the destination is writable, and there areenough messages in the queue, as set in the flush-lines() and the flush-timeout() options.Writer threads stop processing messages when the destination becomes unavailable, or there are nomore messages in the queue.

Sources and destinations affected by multithreading. The following list describes which sources anddestinations can use multiple threads. Changing the --worker-threads command-line option changes thenumber of threads available to these sources and destinations.

■ The tcp and syslog(tcp) sources can process independent connections in separate threads. Thenumber of independent connections is limited by the max-connections() option of the source.Separate sources are processed by separate thread, for example, if you have two separate tcp sourcesdefined that receive messages on different IP addresses or port, syslog-ng PE will use separate threadsfor these sources even if they both have only a single active connection.

■ The udp, file, and pipe sources use a single thread for every source statement.

■ The tcp, syslog, and pipe destinations use a single thread for every destination.

■ The file destination uses a single thread for writing the destination file, but may use a separatethread for each destination file if the filename includes macros.

Sources and destinations not affected by multithreading. The following list describes sources anddestinations that use a separate thread even if you disable multithreading in syslog-ng PE, in addition to thelimit set in the --worker-threads command-line option.

■ The logstore destination uses separate threads for writing the messages from the journal to thelogstore files, and also for timestamping. These threads are independent from the setting of the--worker-threads command-line option.

■ Every sql destination uses its own thread. These threads are independent from the setting of the--worker-threads command-line option.

■ The java destinations use one thread, even if there are multiple Java-based destinations configured.This thread is independent from the setting of the --worker-threads command-line option.

18.2. Configuring multithreading

Starting with version 5 F4, syslog-ng PE runs in multithreaded mode by default. You can enable multithreadingin syslog-ng PE using the following methods:

■ Globally using the threaded(yes) option.

■ Separately for selected sources or destinations using the flags("threaded") option.

450syslog-ng.com

Configuring multithreading

Example 18.1. Enabling multithreadingTo enable multithreading globally, use the threaded option:

options {threaded(yes) ; };

To enable multithreading only for a selected source or destination, use the flags("threaded") option:

source s_tcp_syslog { network(ip(127.0.0.1) port(1999) flags("syslog-protocol", "threaded")

); };

18.3. Optimizing multithreaded performance

Sources:

In syslog-ng, every source has a reader thread. To improve scaling on the source side, use multiple sourcesinstead of one.

For example, if you are using a file source with a wildcard character (such as *.txt), syslog-ng will monitorevery matching file (and keep switching between them), but will use only one thread. It is best to configureseveral single file sources (such as file source1, file source2, and so on) that all monitor only a single file or asmaller range of files. In this case, every file source will use its own thread.

TCP-based network sources form an exception: a TCP-based network source will scale based on the numberof active connections. This means that if there are 10 incoming TCP connections all coming to the same networksource, then that source can use 10 threads, one thread for each connection.

NoteUDP-based network sources do not scale by themselves because they always use a single thread. If you want to handlea large number of UDP connections, it is best to configure a subset of your clients to send the messages to a different portof your syslog-ng server, and use separate source definitions for each port.

451syslog-ng.com

Optimizing multithreaded performance

Figure 18.1. How multithreading works — sources

Message processors:

Message processors — such as filters, rewrite rules, and parsers — are executed by the reader thread in asequential manner.

For example, if you have a log path that defines two sources and a filter, the filter will be performed by thesource1 reader thread when log messages come from source1, and by the source2 reader thread when logmessages come from source2. This means that if log messages come from both source1 and source2, they willboth have a reader thread and that way filtering will be performed simultaneously.

NoteThis is not true for PatternDB because it uses message correlation. When using PatternDB, it runs in only one thread ata time, and this significantly decreases performance.

Destinations:

In syslog-ng, every destination has a writer thread. To improve scaling on the destination side, use multipledestinations instead of one.

For example, when sending messages to a syslog-ng server, you can use multiple connections to the server ifyou configure the syslog-ng server to receive messages on multiple ports, and configure the clients to use bothports.

When writing the log messages to files, use macros in the filename to split the messages to separate files (forexample, using the ${HOST} macro). Files with macros in their filenames are processed in separate writerthreads.

452syslog-ng.com

Optimizing multithreaded performance

Figure 18.2. How multithreading works — destinations

453syslog-ng.com

Optimizing multithreaded performance

Chapter 19. Troubleshooting syslog-ng

This chapter provides tips and guidelines about troubleshooting problems related to syslog-ng. Troubleshootingthe syslog-ng Agent for Windows application is discussed in Chapter 10, Troubleshooting syslog-ng Agent forWindows in The syslog-ng Agent for Windows 6 LTS Administrator Guide.

■ As a general rule, first try to get logging the messages to a local file. Once this is working, you knowthat syslog-ng is running correctly and receiving messages, and you can proceed to forwarding themessages to the server.

■ Always check the configuration files for any syntax errors on both the client and the server usingthe syslog-ng --syntax-only command.

■ If the syslog-ng PE server does not receive the messages, verify that the IP addresses and ports arecorrect in your sources and destinations. Also, check that the client and the server uses the sameprotocol (a common error is to send logs on UDP, but configure the server to receive logs on TCP.If the problem persist, use tcpdump or a similar packet sniffer tool on the client to verify that themessages are sent correctly, and on the server to verify that it receives the messages.

■ To find message-routing problems, run syslog-ng PE with the following command syslog-ng

-Fevd. That way syslog-ng PE will run in the foreground, and display debug messages about themessages that are processed.

■ If syslog-ng is closing the connections for no apparent reason, be sure to check the log messages ofsyslog-ng. You might also want to run syslog-ng with the --verbose or --debug command-lineoptions for more-detailed log messages. You can enable these messages without restarting syslog-ngusing the syslog-ng-ctl verbose --set=on command. For details, see the syslog-ng-ctl manpage at syslog-ng-ctl(1) (p. 503).

■ Build up encrypted connections step-by-step: first create a working unencrypted (for example TCP)connection, then add TLS encryption, and finally client authentication if needed.

■ If you use the same driver and options in the destination of your syslog-ng PE client and the sourceof your syslog-ng PE server, everything should work as expected. Unfortunately there are some othercombinations, that seem to be working, but result in losing parts of the messages. For details on theworking combinations, see Section 2.13, Things to consider when forwarding messages betweensyslog-ng PE hosts (p. 31).

19.1. Possible causes of losing log messages

During the course of a message from the sending application to the final destination of the message, there area number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss.Usually losing messages can be avoided with careful planning and proper configuration of syslog-ng and thehosts running syslog-ng. The following list shows the possible locations where messages may be lost, andprovides methods to minimize the risk of losing messages.

454syslog-ng.com

Possible causes of losing log messages

NoteThe following list covers the main possibilities of losing messages, but does not take into account the possible use offlow-control (for details, see Section 8.2, Managing incoming and outgoing messages with flow-control (p. 316)).

If your syslog-ng PE host uses an NFS partition, see Section 2.14, NFS file system for log files (p. 34).

■ Between the application and the syslog-ng client: Make sure to use an appropriate source to receivethe logs from the application (for example from /dev/log). For example, use unix-stream insteadof unix-dgram whenever possible.

■ When syslog-ng is sending messages: If syslog-ng cannot send messages to the destination and theoutput buffer gets full, syslog-ng will drop messages. Use flags (flow-control) to avoid it (for details,see Section 8.2.2, Configuring flow-control (p. 319)). The number of dropped messages is displayedper destination in the log message statistics of syslog-ng (for details, see Chapter 17, Statistics andmetrics of syslog-ng (p. 446)).

■ On the network: When transferring messages using the UDP protocol, messages may be lost withoutany notice or feedback — such is the nature of the UDP protocol. Always use the TCP protocol totransfer messages over the network whenever possible.For details on minimizing message loss when using UDP, see the following tutorial:Collecting logmessages from UDP sources.

■ In the socket receive buffer: When transferring messages using the UDP protocol, the UDP datagram(that is, the message) that reaches the receiving host placed in a memory area called the socket

receive buffer. If the host receives more messages than it can process, this area overflows, andthe kernel drops messages without letting syslog-ng know about it. Using TCP instead of UDPprevents this issue. If you must use the UDP protocol, increase the size of the receive buffer usingthe so-rcvbuf() option.

■ When syslog-ng is receiving messages:

• The receiving syslog-ng (for example the syslog-ng server or relay) may drop messages if the fifoof the destination file gets full. The number of dropped messages is displayed per destination inthe log message statistics of syslog-ng (for details, see Chapter 17, Statistics and metrics ofsyslog-ng (p. 446)).

• If the number of Log Source Hosts reaches the license limit, the syslog-ng PE server will notaccept connections from additional hosts. The messages sent by additional hosts will be dropped,even if the client uses a reliable transport method (for example, RLTP).

■ When the destination cannot handle large load: When syslog-ng is sending messages at a high rateinto an SQL database, a file, or another destination, it is possible that the destination cannot handlethe load, and processes the messages slowly. As a result, the buffers of syslog-ng fill up, syslog-ngcannot process the incoming messages, and starts to loose messages. For details, see the previousentry. Use the throttle parameter to avoid this problem.

■ As a result of an unclean shutdown of the syslog-ng server: If the host running the syslog-ng serverexperiences an unclean shutdown, it takes time until the clients realize that the connection to thesyslog-ng server is down. Messages that are put into the output TCP buffer of the clients during thisperiod are not sent to the server.

455syslog-ng.com

Possible causes of losing log messages

■ When syslog-ng PE is writing messages into files: If syslog-ng PE receives a signal (SIG) whilewriting log messages to file, the log message that is processed by the write call can be lost if theflush_lines parameter is higher than 1.

■ When using the RLTP™ protocol:In the following cases, it is possible to lose log messages even if you use RLTP™:

• If you use RLTP™ together with non-reliable disk-buffer, it is possible to lose logs.

• When sending logs through a relay that is using a non-reliable disk-buffer, it is possible to loselogs if the relay crashes.

• When sending logs through a relay that is using a non-reliable disk-buffer, it is possible that logsare duplicated if the relay crashes, or it is stopped.

• If the underlying disk system of syslog-ng PE fails to write the log messages to the disk, but itdoes not return a write error, or some other hardware or operating-system error happens.

NoteTo minimize the chances of losing messages, it is recommended to set the flush-size() parameterof the receiver to 1 (however, note that depending on your syslog-ng PE configuration, this can decreasethe performance of the receiver).

19.2. Procedure – Creating syslog-ng core files

Purpose:

When syslog-ng crashes for some reason, it can create a core file that contains important troubleshootinginformation. To enable core files, complete the following procedure:

Steps:

Step 1. Core files are produced only if the maximum core file size ulimit is set to a high value in theinit script of syslog-ng.Add the following line to the init script of syslog-ng:

ulimit -c unlimited

Step 2. Verify that syslog-ng has permissions to write the directory it is started from, for example/opt/syslog-ng/sbin/.

Step 3. If syslog-ng crashes, it will create a core file in the directory syslog-ng was started from.

Step 4. To test that syslog-ng can create a core file, you can create a crash manually. For this, determine thePID of syslog-ng (for example using the ps -All|grep syslog-ng command), then issue thefollowing command: kill -ABRT <syslog-ng pid>

This should create a core file in the current working directory.

456syslog-ng.com

Possible causes of losing log messages

19.3. Collecting debugging information with strace, truss, or tusc

To properly troubleshoot certain situations, it can be useful to trace which system calls syslog-ng PE performs.How this is performed depends on the platform running syslog-ng PE. In general, note the following points:

■ When syslog-ng PE is started, a supervisor process might stay in the foreground, while the actualsyslog-ng daemon goes to the background. Always trace the background process.

■ Apart from the system calls, the time between two system calls can be important as well. Make surethat your tracing tool records the time information as well. For details on how to do that, refer to themanual page of your specific tool (for example, strace on Linux, or truss on Solaris and BSD).

■ Run your tracing tool in verbose mode, and if possible, set it to print long output strings, so themessages are not truncated.

■ When using strace, also record the output of lsof to see which files are accessed.

The following are examples for tracing system calls of syslog-ng on some platforms. The output is saved intothe /tmp/syslog-ng-trace.txt file, sufficed with the PID of the related syslog-ng process.The path of thesyslog-ng binary assumes that you have installed syslog-ng PE from the official syslog-ng PE binaries availableat the BalaBit website — native distribution-specific packages may use different paths.

■ Linux: strace -o /tmp/trace.txt -s256 -ff -ttT /opt/syslog-ng/sbin/syslog-ng

-f /opt/syslog-ng/etc/syslog-ng.conf -Fdv

■ HP-UX: tusc -f -o /tmp/syslog-ng-trace.txt -T /opt/syslog-ng/sbin/syslog-ng

■ IBM AIX and Solaris: truss -f -o /tmp/syslog-ng-trace.txt -r all -w all -u

libc:: /opt/syslog-ng/sbin/syslog-ng -d -d -d

TipTo execute these commands on an already running syslog-ng PE process, use the -p <pid_of_syslog-ng> parameter.

19.4. Procedure – Running a failure script

Purpose:

You can create a failure script that is executed when syslog-ng PE terminates abnormally, that is, when it exitswith a non-zero exit code. For example, you can use this script to send an automatic e-mail notification.

Prerequisites:

The failure script must be the following file: /opt/syslog-ng/sbin/syslog-ng-failure, and must beexecutable.

To create a sample failure script, complete the following steps.

Steps:

Step 1. Create a file named /opt/syslog-ng/sbin/syslog-ng-failure with the following content:

457syslog-ng.com

Collecting debugging information with strace, truss, or tusc

#!/usr/bin/env bash

cat >>/tmp/test.txt <<EOF

$(date)

Name............$1

Chroot dir......$2

Pid file dir....$3

Pid file........$4

Cwd.............$5

Caps............$6

Reason..........$7

Argbuf..........$8

Restarting......$9

EOF

Step 2. Make the file executable: chmod +x /opt/syslog-ng/sbin/syslog-ng-failure

Step 3. Run the following command in the /opt/syslog-ng/sbin directory: ./syslog-ng

--process-mode=safe-background; sleep 0.5; ps aux | grep './syslog-ng' |

grep -v grep | awk '{print $2}' | xargs kill -KILL; sleep 0.5; cat

/tmp/test.txt

The command starts syslog-ng PE in safe-background mode (which is needed to use the failure script)and then kills it. You should see that the relevant information is written into the /tmp/test.txt file,for example:

Thu May 18 12:08:58 UTC 2017

Name............syslog-ng

Chroot dir......NULL

Pid file dir....NULL

Pid file........NULL

Cwd.............NULL

Caps............NULL

Reason..........signalled

Argbuf..........9

Restarting......not-restarting

Step 4. You should also see messages similar to the following in system syslog. The exact message dependson the signal (or the reason why syslog-ng PE stopped):

May 18 13:56:09 myhost supervise/syslog-ng[10820]: Daemon exited gracefully,

not restarting; exitcode='0'

May 18 13:57:01 myhost supervise/syslog-ng[10996]: Daemon exited due to a

deadlock/signal/failure, restarting; exitcode='131'

May 18 13:57:37 myhost supervise/syslog-ng[11480]: Daemon was killed, not

restarting; exitcode='9'

The failure script should run on every non-zero exit event.

458syslog-ng.com

Collecting debugging information with strace, truss, or tusc

19.5. Stopping syslog-ng

To avoid problems, always use the init scripts to stop syslog-ng (/etc/init.d/syslog-ng stop), insteadof using the kill command. This is especially true on Solaris and HP-UX systems, here use/etc/init.d/syslog stop.

19.6. Reporting bugs and finding help

If you need help, want to open a support ticket, or report a bug, we recommend using the syslog-debun toolto collect information about your environment and syslog-ng PE version. For details, see syslog-debun(1) (p. 490).For support contacts, see Section 5.2, Support contact (p. xiii).

19.7. Recover data from orphaned diskbuffer files

When you change the configuration of a syslog-ng PE host that uses disk-based buffering (also called diskqueue), syslog-ng PE may start new disk buffer files for the destinations that you have changed. In such case,syslog-ng PE abandons the old disk queue files. If there were unsent log messages in the disk queue files, thesemessages remain in the disk queue files, and will not be sent to the destinations.

To find, examine, and flush the log messages from such orphaned disk queue files, see the Processing messagesstuck in the disk queue files of syslog-ng Premium Edition tutorial.

459syslog-ng.com

Stopping syslog-ng

Chapter 20. Best practices and examples

This chapter discusses some special examples and recommendations.

20.1. General recommendations

This section provides general tips and recommendations on using syslog-ng. Some of the recommendationsare detailed in the subsequent sections.

■ Do not base the separation of log messages into different files on the facility parameter. As severalapplications and processes can use the same facility, the facility does not identify the applicationthat sent the message. By default, the facility parameter is not even included in the log messageitself. In general, sorting the log messages into several different files can make finding specific logmessages difficult. If you must create separate log files, use the application name.

■ Standard log messages include the local time of the sending host, without any time zone information.It is recommended to replace this timestamp with an ISODATE timestamp, because the ISODATEformat includes the year and timezone as well. To convert all timestamps to the ISODATE format,include the following line in the syslog-ng configuration file:

options {ts-format(iso) ; };

■ Resolving the IP addresses of the clients to domain names can decrease the performance of syslog-ng.For details, see Section 20.3, Using name resolution in syslog-ng (p. 461).

20.2. Handling large message load

This section provides tips on optimizing the performance of syslog-ng. Optimizing the performance is importantfor syslog-ng hosts that handle large traffic.

■ Disable DNS resolution, or resolve hostnames locally. For details, see Section 20.3, Using nameresolution in syslog-ng (p. 461).

■ Enable flow-control for the TCP sources. For details, see Section 8.2, Managing incoming andoutgoing messages with flow-control (p. 316).

■ Do not use the usertty() destination driver. Under heavy load, the users are not be able to readthe messages from the console, and it slows down syslog-ng.

■ Do not use regular expressions in our filters. Evaluating general regular expressions puts a high loadon the CPU. Use simple filter functions and logical operators instead. For details, see Section 14.3,Regular expressions (p. 403).

■ WarningWhen receiving messages using the UDP protocol, increase the size of the UDP receive buffer on thereceiver host (that is, the syslog-ng PE server or relay receiving the messages). Note that on certain platforms,for example, on Red Hat Enterprise Linux 5, even low message load (~200 messages per second) can resultin message loss, unless the so-rcvbuf() option of the source is increased. In such cases, you will need

460syslog-ng.com

General recommendations

to increase the net.core.rmem_max parameter of the host (for example, to 1024000), but do not modifynet.core.rmem_default parameter.

As a general rule, increase the so-rcvbuf() so that the buffer size in kilobytes is higher than the rate ofincoming messages per second. For example, to receive 2000 messages per second, set the so-rcvbuf()at least to 2 097 152 bytes.

■ Increase the value of the flush-lines() parameter. Increasing flush-lines() from 0 to 100

can increase the performance of syslog-ng PE by 100%.

20.3. Using name resolution in syslog-ng

The syslog-ng application can resolve the hostnames of the clients and include them in the log messages.However, the performance of syslog-ng is severely degraded if the domain name server is unaccessible or slow.Therefore, it is not recommended to resolve hostnames in syslog-ng. If you must use name resolution fromsyslog-ng, consider the following:

■ Use DNS caching. Verify that the DNS cache is large enough to store all important hostnames. (Bydefault, the syslog-ng DNS cache stores 1007 entries.)

options { dns-cache-size(2000); };

■ If the IP addresses of the clients change only rarely, set the expiry of the DNS cache large.

options { dns-cache-expire(87600); };

■ If possible, resolve the hostnames locally. For details, see Procedure 20.3.1, Resolving hostnameslocally (p. 461).

NoteDomain name resolution is important mainly in relay and server mode.

20.3.1. Procedure – Resolving hostnames locally

Purpose:

Resolving hostnames locally enables you to display hostnames in the log files for frequently used hosts, withouthaving to rely on a DNS server. The known IP address – hostname pairs are stored locally in a file. In the logmessages, syslog-ng will replace the IP addresses of known hosts with their hostnames. To configure localname resolution, complete the following steps:

Steps:

Step 1. Add the hostnames and the respective IP addresses to the file used for local name resolution. On Linuxand UNIX systems, this is the /etc/hosts file. Consult the documentation of your operating systemfor details.

461syslog-ng.com

Using name resolution in syslog-ng

Step 2. Instruct syslog-ng to resolve hostnames locally. Set the use-dns() option of syslog-ng topersist_only.

Step 3. Set the dns-cache-hosts() option to point to the file storing the hostnames.

options {

use-dns(persist_only);

dns-cache-hosts(/etc/hosts); };

20.4. Procedure – Collecting logs from chroot

Purpose:

To collect logs from a chroot using a syslog-ng client running on the host, complete the following steps:

Figure 20.1. Collecting logs from chroot

Steps:

Step 1. Create a /dev directory within the chroot. The applications running in the chroot send their log messageshere.

462syslog-ng.com

Using name resolution in syslog-ng

Step 2. Create a local source in the configuration file of the syslog-ng application running outside the chroot.This source should point to the /dev/log file within the chroot (for example to the /chroot/dev/logdirectory).

Step 3. Include the source in a log statement.

NoteYou need to set up timezone information within your chroot as well. This usually means creating a symlinkto /etc/localtime.

20.5. Configuring log rotation

The syslog-ng PE application does not rotate logs by itself. To use syslog-ng PE for log rotation, consider thefollowing approaches:

Use logrotate together with syslog-ng PE:

■ Ideal for workstations or when processing fewer logs.

■ It is included in most distributions by default.

■ Less scripting is required, only logrotate has to be configured correctly.

■ Requires frequent restart (syslog-ng PE must be reloaded/restarted when the files are rotated). Afterrotating the log files, reload syslog-ng PE using the syslog-ng-ctl reload command, or useanother method to send a SIGHUP to syslog-ng PE.

■ The statistics collected by syslog-ng PE, and the correlation information gathered with PatternDatabase is lost with each restart.

Separate incoming logs based on time, host or other information:

■ Ideal for central log servers, where regular restart of syslog-ng PE is unfavorable.

■ Requires shell scripts or cron jobs to remove old logs.

■ It can be done by using macros in the destination name (in the filename, directory name, or thedatabase table name).

Example 20.1. File destination for log rotationThis sample file destination configuration stores incoming logs in files that are named based on the current year, monthand day, and places these files in directories that are named based on the hostname:

destination d_sorted { file("/var/log/remote/${HOST}/${YEAR}_${MONTH}_${DAY}.log"

create-dirs(yes)); };

Example 20.2. Logstore destination for log rotationThis sample logstore destination configuration stores incoming logs in logstores that are named based on the current year,month and day, and places these logstores in directories that are named based on the hostname:

destination d_logstore { logstore("/var/log/remote/${HOST}/${YEAR}_${MONTH}_${DAY}.lgs"

compress(9) create-dirs(yes)); };

463syslog-ng.com

Configuring log rotation

Example 20.3. Command for cron for log rotationThis sample command for cron removes files older than two weeks from the /var/log/remote directory:

find /var/log/remote/ -daystart -mtime +14 -type f -exec rm {} \;

464syslog-ng.com

Configuring log rotation

Appendix A. The syslog-ng manual pages

465syslog-ng.com

dqtool

dqtool — Display the contents of a disk-buffer file created with syslog-ng Premium Edition

Synopsis

dqtool [command] [options]

Description

NOTE: The dqtool application is distributed with the syslog-ng Premium Edition system logging application,and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at thesyslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The dqtool application is a utility that can be used to display and format the messages stored in a disk-bufferfile.

The cat command

cat [options] [file]

Use the cat command to display the log messages stored in the disk-buffer (also called disk-queue) file, andalso information from the header of the disk queue file. The messages are printed to the standard output (stdout),so it is possible to use grep and other tools to find particular log messages, e.g., dqtool cat

/var/log/messages.qf |grep 192.168.1.1.

The cat command has the following options:

--debug or -d Print diagnostic and debugging messages to stderr.

--help or -h Display a brief help message.

--template=<template> or -t Format the messages using the specified template.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

Example:

./dqtool cat ../var/syslog-ng-00000.qf

The output looks like:

Disk-buffer state loaded; filename='../var/syslog-ng-00000.qf', qout_length='65',

qbacklog_length='0', qoverflow_length='9205', qdisk_length='0'

Mar 3 10:52:05 tristram localprg[1234]: seq: 0000011630, runid: 1267609923, stamp:

2010-03-03T10:52:05

PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD

Mar 3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, stamp:

466syslog-ng.com

2010-03-03T10:52:05

PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD

Files

/opt/syslog-ng/bin/dqtool

See also

syslog-ng.conf(5)

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

467syslog-ng.com

loggen

loggen — Generate syslog messages at a specified rate

Synopsis

loggen [options ]target [port]

Description

NOTE: The loggen application is distributed with the syslog-ng system logging application, and is usually partof the syslog-ng package. The latest version of the syslog-ng application is available the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The loggen application is tool to test and stress-test your syslog server and the connection to the server. It cansend syslog messages to the server at a specified rate, using a number of connection types and protocols,including TCP, UDP, and unix domain sockets. The messages can be generated automatically (repeating thePADDstring over and over), or read from a file or the standard input.

When loggen finishes sending the messages, it displays the following statistics:

■ average rate: Average rate the messages were sent in messages/second.

■ count: The total number of messages sent.

■ time: The time required to send the messages in seconds.

■ average message size: The average size of the sent messages in bytes.

■ bandwidth: The average bandwidth used for sending the messages in kilobytes/second.

Options

--active-connections

<number-of-connections>

Number of connections loggen will use to send messages tothe destination. This option is usable only when using TCP orTLS connections to the destination. Default value: 1

The loggen utility waits until every connection is establishedbefore starting to send messages. See also the--idle-connections option.

--csv or -C Send the statistics of the sent messages to stdout as CSV. Thiscan be used for plotting the message rate.

--dgram or -D Use datagram socket (UDP or unix-dgram) to send the messagesto the target. Requires the --inet option as well.

--dont-parse or -d Do not parse the lines read from the input files, send them asreceived.

--help or -h Display a brief help message.

468syslog-ng.com

--idle-connections

<number-of-connections>

Number of idle connections loggen will establish to thedestination. Note that loggen will not send any messages onidle connections, but the connection is kept open using keep-alivemessages. This option is usable only when using TCP or TLSconnections to the destination. See also the--active-connections option. Default value: 0

--inet or -i Use the TCP (by default) or UDP (when used together with the--dgram option) protocol to send the messages to the target.

--interval <seconds> or -I<seconds>

The number of seconds loggen will run. Default value: 10

NoteNote that when the --interval and --number are usedtogether, loggenwill send messages until the period setin --interval expires or the amount of messages setin --number is reached, whichever happens first.

--ipv6 or -6 Specify the destination using its IPv6 address. Note that thedestination must have a real IPv6 address.

--loop-reading or -l Read the file specified in --read-file option in loop: loggenwill start reading from the beginning of the file when it reachesthe end of the file.

--number

<number-of-messages> or -n<number-of-messages>

Number of messages to generate.

NoteNote that when the --interval and --number are usedtogether, loggenwill send messages until the period setin --interval expires or the amount of messages setin --number is reached, whichever happens first.

--no-framing or -F Do not use the framing of the IETF-syslog protocol style, evenif the --syslog-proto option is set.

--quiet or -Q Output statistics only when the execution of loggen is finished.If not set, the statistics are displayed every second.

--rate <message/second> or -r<message/second>

The number of messages generated per second for every activeconnection. Default value: 1000

--read-file <filename> or -R<filename>

Read the messages from a file and send them to the target. Seealso the --skip-tokens option.

Specify - as the input file to read messages from the standardinput (stdio). Note that when reading messages from the standardinput, loggen can only use a single thread. The -R - parameters

469syslog-ng.com

must be placed at end of command, like: loggen 127.0.0.1

1061 --read-file -

--sdata <data-to-send> or -p<data-to-send>

Send the argument of the --sdata option as the SDATA partof IETF-syslog (RFC5424 formatted) messages. Use it togetherwith the --syslog-proto option. For example: --sdata

"[test name=\"value\"]

--size <message-size> or -s<message-size>

The size of a syslog message in bytes. Default value: 256.Minimum value: 127 bytes, maximum value: 8192 bytes.

--skip-tokens <number> Skip the specified number of space-separated tokens (words) atthe beginning of every line. For example, if the messages in thefile look like foo bar message, --skip-tokens 2 skipsthe foo bar part of the line, and sends only the message part.Works only when used together with the --read-file

parameter. Default value: 3

--stream or -S Use a stream socket (TCP or unix-stream) to send the messagesto the target.

--syslog-proto or -P Use the new IETF-syslog message format as specified inRFC5424. By default, loggen uses the legacy BSD-syslogmessage format (as described in RFC3164). See also the--no-framing option.

--unix </path/to/socket> or-x </path/to/socket>

Use a UNIX domain socket to send the messages to the target.

--use-ssl or -U Use an SSL-encrypted channel to send the messages to the target.Note that it is not possible to check the certificate of the target,or to perform mutual authentication.

--version or -V Display version number of syslog-ng.

Examples

The following command generates 100 messages per second for ten minutes, and sends them to port 2010 ofthe localhost via TCP. Each message is 300 bytes long.

loggen --size 300 --rate 100 --interval 600 127.0.0.1 2010

The following command is similar to the one above, but uses the UDP protocol.

loggen --inet --dgram --size 300 --rate 100 --interval 600 127.0.0.1 2010

Send a single message on TCP6 to the ::1 IPv6 address, port 1061:

loggen --ipv6 --number 1 ::1 1061

Send a single message on UDP6 to the ::1 IPv6 address, port 1061:

470syslog-ng.com

loggen --ipv6 --dgram --number 1 ::1 1061

Send a single message using a unix domain-socket:

loggen --unix --stream --number 1 </path/to/socket>

Read messages from the standard input (stdio) and send them to the localhost:

loggen 127.0.0.1 1061 --read-file -

Files

/opt/syslog-ng/bin/loggen

See also

syslog-ng.conf(5)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

471syslog-ng.com

lgstool

lgstool — Inspect and validate the binary log files (logstores) created with syslog-ng Premium Edition

Synopsis

lgstool [command] [options]

Description

NOTE: The lgstool application is distributed with the syslog-ng Premium Edition system logging application,and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at thesyslog-ng page. The lgstool utility is available for Microsoft Windows operating systems at the syslog-ngDownloads page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The lgstool application is a utility that can be used to:

■ Display and format the messages stored in logstore files

■ Display the record structure of logstore files

■ Process log messages from orphaned journal files and write them into logstore files

■ Follow (tail) messages arriving to a logstore file real-time

■ Validate the digital signature and timestamp of encrypted logstore files

Note that in the Windows-version of lgstool the recover option is not available and the functionality of thetail option is limited.

The cat command

cat [options] [file]

Use the cat command to display the log messages stored in the logstore file. Log messages available in thejournal file of the logstore (but not yet written to the logstore file itself) are displayed as well. The messagesare printed to the standard output (stdout), so it is possible to use grep and other tools to find particular logmessages, e.g., lgstool cat /var/log/messages.lgs |grep 192.168.1.1. Note that can also followlogstore files — for details on this feature, see the section called “The tail command” (p. 475).

The cat command has the following options:

--debug or -d Print diagnostic and debugging messages to stderr.

--filter<expression> or -i Only print messages matching the specified syslog-ng PE filter.All possible macros, regular expressions and logical expressionscan be specified in a filter.

472syslog-ng.com

Example A.1. lgstool cat filter

lgstool cat -t 'host: ${HOST} program:

${PROGRAM} msg: ${MSG}\n'

--filter='program("prg0000[0]")'

/tmp/logstore-serialized.lgs

--help or -h Display a brief help message.

--key=<keyfile> or -k Use the specified private key to decrypt encrypted logstore files.

--seek=<ID> or -s Display only messages newer than the message specified.

--template=<template> or -t Format the messages using the specified template.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

Example:

lgstool cat --key=mykey.pem mylogstore.lgs

The inspect command

inspect [options] [file]

Use the inspect command to display structure of the logstore file. The following information is displayed:

■ cipher: The cipher algorithm used to encrypt the logstore file.

■ digest: The digest (hash) algorithm used.

■ encrypt: TRUE if the logstore file is encrypted.

■ compress: TRUE if the logstore file is compressed.

■ hmac: TRUE if the logstore file includes HMAC (Hash-based Message Authentication Code)information for the chunks.

■ chunk_mac: The MAC (Message Authentication Code) of the chunk.

■ file_mac: The MAC (Message Authentication Code) of the chunk.

For timestamped logstore files, the following information is also displayed:

■ chunk_id: The ID of the chunk.

■ Version: The version of the logstore file format used.

■ Policy OID: The OID of the timestamping policy used in the timestamping request.

■ Hash Algorithm: The digest (hash) algorithm used to create the hash of the chunk.

■ Serial number: The serial number of the timestamp.

■ Timestamp: The date when the Timestamping Authority timestamped the chunk.

■ Accuracy: The accuracy of the timestamp.

■ Ordering: Indicates the status of the ordering field in the timestamping request.

473syslog-ng.com

■ Nonce: The nonce (a large random number with a high probability that it is generated by the clientonly once) included in the timestamping request (if any).

■ TSA: The Distinguished Name (DN) of the Timestamping Authority.

The inspect command has the following options:

--debug or -d Print diagnostic and debugging messages to stderr.

--help or -h Display a brief help message.

--key=<keyfile> or -k Use the specified private key to decrypt encrypted logstore files.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

Example:

lgstool inspect --key=mykey.pem mylogstore.lgs

A sample output looks like this:

XFRM_INFO @941

cipher: aes-128-cbc

digest: sha1

CHUNK 0@1079: [1 - 1000]:

encrypt: TRUE

compress: TRUE

hmac: TRUE

chunk_mac: e4d5d813979cf865d5ae4624f7aa98047123cd52

file_mac: 6600600ca5befb002a73b15be8f0ac04973d5936

TIMESTAMP @36481:

chunk_id: 0

Status info:

Status: Granted.

Status description: unspecified

Failure info: unspecified

TST info:

Version: 1

Policy OID: 1.2.3.4

Hash Algorithm: sha1

Message data:

0000 - 66 00 60 0c a5 be fb 00-2a 73 b1 5b e8 f0 ac 04 f.`.....*s.[....

0010 - 97 3d 59 36 .=Y6

Serial number: 0x029A

Time stamp: Mar 19 13:48:57 2010 GMT

Accuracy: 0x01 seconds, 0x01F4 millis, 0x64 micros

Ordering: no

Nonce: 0xB613F55AEFFA6DC0

TSA: unspecified

Extensions:

The recover command

recover [options] [file]

474syslog-ng.com

WarningDo NOT use the lgstool recover command on logstore files that are actively used by syslog-ng PE. It might lead todata loss. Always stop syslog-ng PE first.

Use the recover command can process and correct broken logstore files. It can also process orphaned journalfiles and move their contents to the respective logstore file. Encrypted, compressed, and timestamped logstorefiles can be recovered as well — the private key of the logstore is not needed to recover encrypted logstore files(recovering the encrypted file does not give access to its contents). Note that the recover option is not availablein the Windows-version of lgstool.

WarningThe lgstool application cannot fetch timestamps to the chunks (message blocks), so chunks recovered with lgstool

are not timestamped (the internal timestamp of the syslog messages is included in the messages).

The recover command is available in syslog-ng Premium Edition 3.2 and later, and has the following options:

--compress-level or -c Set the level of compression when processing a journal file intoa compressed logstore. Default value: 3

--debug or -d Print diagnostic and debugging messages to stderr.

--help or -h Display a brief help message.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

Example:

lgstool recover mylogstore.lgs

The tail command

tail [options] [file]

Use the tail -f command to follow the contents of a logstore file like the traditional tail command doeson Linux/UNIX systems. The messages are printed to the standard output (stdout). Contents of the journal filerelated to the logstore file are displayed as well.

The tail command is available in syslog-ng Premium Edition 3.2 and later, and has the following options.Note that in the Windows-version of lgstool the tail -f option is not available.

--debug or -d Print diagnostic and debugging messages to stderr.

--help or -h Display a brief help message.

--filter=<expression> or -i Only print messages matching the specified syslog-ng PE filter.All possible macros, regular expressions and logical expressionscan be specified in a filter.

475syslog-ng.com

Example A.2. lgstool tail filter

lgstool tail -t 'host: ${HOST} program:

${PROGRAM} msg: ${MSG}\n'

--filter='program("prg0000[0]")'

/tmp/logstore-serialized.lgs

--follow or -f Follow mode: display messages as they arrive into the logstore.

--key=<keyfile> or -k Use the specified private key to decrypt encrypted logstore files.

--lines=<N> or -n Display the last N lines of the logstore file instead of the last 10.Alternatively, use +N to display lines starting with the Nth.

--sleep_interval=<seconds>

or -sNumber of seconds to wait before displaying new messages infollow mode.

--template=<template> or -t Format the messages using the specified template.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

Example:

lgstool tail -f -n=20 --key=mykey.pem mylogstore.lgs

The validate command

validate [options] [file]

Use the validate command to validate the signatures and timestamps of a logstore file. The validate

command has the following options:

--ca-dir=<directory> or -C The directory that stores the certificates of the trusted CertificateAuthorities. Use this option if the timestamps of your logstorefiles were signed with certificates belonging to differentCertificate Authorities.

--ca-dir-layout=<md5|sha1> The type of the hash used for the CA certificates. The defaultvalue (md5) is expected to change to sha1 in subsequent releasesof syslog-ng PE.

--ca-file=<file> or -P A file that stores the certificate of the trusted CertificateAuthority. Use this option if the timestamps of your logstorefiles were signed with a single certificate, or if every suchcertificate belongs to the same Certificate Authority.

--crl-dir=<directory> or -R The directory that stores the Certificate Revocation Lists of thetrusted Certificate Authorities.

--debug or -d Print diagnostic and debugging messages to stderr.

--help or -h Display a brief help message.

--key=<keyfile> or -k Use the specified private key to decrypt encrypted logstore files.

476syslog-ng.com

--require-ts or -T Consider the logstore file invalid unless the entire file is protectedby a valid timestamp.

--seed or -S Use the ~/.rnd file or the file specified in the $RANDFILE

environmental variable as seed. This is needed only on platformsthat do not have a /dev/random device (for example, Solaris)and the entropy gathering daemon egd application is not installedon the system.

--ts-name=<name> or -D Consider the logstore file invalid unless the timestamps aresigned by the specified Timestamping Authority. Specify theDistinguished Name (DN) of the Timestamping Authority.

--verbose or -v Print verbose messages to stderr.

--version or -V Display version information.

By default, the lgstool validate command checks only the checksum of the file. Use the --require-tsoption to validate the timestamps as well. THe digital signature of the timestamps is checked only if the--ca-dir or the --ca-file parameter is set.

Example:

lgstool validate --key=mykey.pem --ca-file=mycacert.pem --ts-name=MYTSA

mylogstore.lgs

The reindex command

reindex [options] [file]

The reindex command is an experimental, currently unsupported tool. Do not attempt to use it unless yoursyslog-ng PE support team explicitly instructs you to do so.

Files

/opt/syslog-ng/bin/lgstool

See also

syslog-ng.conf(5)

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

477syslog-ng.com

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

478syslog-ng.com

pdbtool

pdbtool — An application to test and convert syslog-ng pattern database rules

Synopsis

pdbtool [command] [options]

Description

This manual page is only an abstract, for the complete documentation of syslog-ng and pdbtool, the syslog-ngDocumentation page.

The syslog-ng application can match the contents of the log messages to a database of predefined messagepatterns (also called patterndb). By comparing the messages to the known patterns, syslog-ng is able to identifythe exact type of the messages, tag the messages, and sort them into message classes. The message classes canbe used to classify the type of the event described in the log message. The functionality of the pattern databaseis similar to that of the logcheck project, but the syslog-ng approach is faster, scales better, and is much easierto maintain compared to the regular expressions of logcheck.

The pdbtool application is a utility that can be used to:

■ test messages, or specific rules

■ convert an older pattern database to the latest database format

■ merge pattern databases into a single file

■ automatically create pattern databases from a large amount of log messages

■ dump the RADIX tree built from the pattern database (or a part of it) to explore how the patternmatching works.

The dictionary command

dictionary [options]

Lists every name-value pair that can be set by the rules of the pattern database.

--dump-tags or -T List the tags instead of the names of the name-value pairs.

--pdb <path-to-file> or -p<path-to-file>

Name of the pattern database file to use.

--program <programname> or -P<programname>

List only the name-value pairs that can be set for the messagesof the specified $PROGRAM application.

The dump command

dump [options]

479syslog-ng.com

Display the RADIX tree built from the patterns. This shows how are the patterns represented in syslog-ng andit might also help to track down pattern-matching problems. The dump utility can dump the tree used formatching the PROGRAM or the MSG parts.

--debug or -d Enable debug/diagnostic messages on stderr.

--pdb or -p Name of the pattern database file to use.

--program or -P Displays the RADIX tree built from the patterns belonging tothe ${PROGRAM} application.

--program-tree or -T Display the ${PROGRAM} tree.

--verbose or -v Enable verbose messages on stderr.

Example and sample output:

pdbtool dump -p patterndb.xml -P 'sshd'

'p'

'assword for'

@QSTRING:@

'from'

@QSTRING:@

'port '

@NUMBER:@ rule_id='fc49054e-75fd-11dd-9bba-001e6806451b'

' ssh' rule_id='fc55cf86-75fd-11dd-9bba-001e6806451b'

'2' rule_id='fc4b7982-75fd-11dd-9bba-001e6806451b'

'ublickey for'

@QSTRING:@

'from'

@QSTRING:@

'port '

@NUMBER:@ rule_id='fc4d377c-75fd-11dd-9bba-001e6806451b'

' ssh' rule_id='fc5441ac-75fd-11dd-9bba-001e6806451b'

'2' rule_id='fc44a9fe-75fd-11dd-9bba-001e6806451b'

The match command

match [options]

Use the match command to test the rules in a pattern database. The command tries to match the specifiedmessage against the patterns of the database, evaluates the parsers of the pattern, and also displays which partof the message was parsed successfully. The command returns with a 0 (success) or 1 (no match) return codeand displays the following information:

■ the class assigned to the message (that is, system, violation, and so on),

■ the ID of the rule that matched the message, and

■ the values of the parsers (if there were parsers in the matching pattern).

The match command has the following options:

480syslog-ng.com

--color-out or -c Color the terminal output to highlight the part of the messagethat was successfully parsed.

--debug or -d Enable debug/diagnostic messages on stderr.

--debug-csv or -C Print the debugging information returned by the--debug-pattern option as comma-separated values.

--debug-pattern or -D Print debugging information about the pattern matching. Seealso the --debug-csv option.

--file=<filename-with-path>

or -fProcess the messages of the specified log file with the patterndatabase. This option allows to classify messages offline, andto apply the pattern database to already existing logfiles. To readthe messages from the standard input (stdin), specify a hyphen(-) character instead of a filename.

--filter=<filter-expression>

or -FPrint only messages matching the specified syslog-ng filterexpression.

--message or -M The text of the log message to match (only the ${MESSAGE}

part without the syslog headers).

--pdb or -p Name of the pattern database file to use.

--program or -P Name of the program to use, as contained in the ${PROGRAM}part of the syslog message.

--template=<template-expression>

or -TA syslog-ng template expression that is used to format the outputmessages.

--verbose or -v Enable verbose messages on stderr.

Example: The following command checks if the patterndb.xml file recognizes the Accepted publickey

for myuser from 127.0.0.1 port 59357 ssh2 message:

pdbtool match -p patterndb.xml -P sshd -M "Accepted publickey for myuser from

127.0.0.1 port 59357 ssh2"

The following example applies the sshd.pdb pattern database file to the log messages stored in the/var/log/messages file, and displays only the messages that received a useracct tag.

pdbtool match -p sshd.pdb \

–file /var/log/messages \

–filter ‘tags(“usracct”);’

The merge command

merge [options]

Use the merge command to combine separate pattern database files into a single file (pattern databases areusually stored in separate files per applications to simplify maintenance). If a file uses an older database format,

481syslog-ng.com

it is automatically updated to the latest format (V3). See the syslog-ng Documentation page for details on thedifferent pattern database versions.

--debug or -d Enable debug/diagnostic messages on stderr.

--directory or -D The directory that contains the pattern database XML files to bemerged.

--glob or -G Specify filenames to be merged using a glob pattern, for example,using wildcards. For details on glob patterns, see man glob.This pattern is applied only to the filenames, and not on directorynames.

--pdb or -p Name of the output pattern database file.

--recursive or -r Merge files from subdirectories as well.

--verbose or -v Enable verbose messages on stderr.

Example:

pdbtool merge --recursive --directory /home/me/mypatterns/ --pdb

/var/lib/syslog-ng/patterndb.xml

Currently it is not possible to convert a file without merging, so if you only want to convert an older patterndatabase file to the latest format, you have to copy it into an empty directory.

The patternize command

patternize [options]

Automatically create a pattern database from a log file containing a large number of log messages. The resultingpattern database is printed to the standard output (stdout). The pdbtool patternize command uses a dataclustering technique to find similar log messages and replacing the differing parts with @ESTRING:: @ parsers.For details on pattern databases and message parsers, see the syslog-ng Documentation page. The patternizecommand is available only in syslog-ng PE version 3.2 and later.

--debug or -d Enable debug/diagnostic messages on stderr.

--file=<path> or -f The logfile containing the log messages to create patterns from.To receive the log messages from the standard input (stdin), use-.

--iterate-outliers or -o Recursively iterate on the log lines to cover as many logmessages with patterns as possible.

--named-parsers or -n The number of example log messages to include in the patterndatabase for every pattern. Default value: 1

--no-parse or -p Do not parse the input file, treat every line as the message partof a log message.

--samples=<number-of-samples> Include a generated name in the parsers, for example,.dict.string1, .dict.string2, and so on.

--support=<number> or -S A pattern is added to the output pattern database if at least thespecified percentage of log messages from the input logfile match

482syslog-ng.com

the pattern. For example, if the input logfile contains 1000 logmessages and the --support=3.0 option is used, a pattern iscreated only if the pattern matches at least 3 percent of the logmessages (that is, 30 log messages). If patternize does not createenough patterns, try to decrease the support value.Default value: 4.0

--verbose or -v Enable verbose messages on stderr.

Example:

pdbtool patternize --support=2.5 --file=/var/log/messages

The test command

test [options]

Use the test command to validate a pattern database XML file. Note that you must have the xmllint applicationinstalled. The test command is available only in syslog-ng PE version 3.2 and later.

--color-out or -c Enable coloring in terminal output.

--debug or -d Enable debug/diagnostic messages on stderr.

--debug or -D Print debugging information on non-matching patterns.

--rule-id or -r Test only the patterndb rule (specified by its rule id) against itsexample.

--validate Validate a pattern database XML file.

--verbose or -v Enable verbose messages on stderr.

Example:

pdbtool test --validate /home/me/mypatterndb.pdb

Files

/opt/syslog-ng/

/opt/syslog-ng/etc/syslog-ng.conf

See also

The syslog-ng Documentation page

syslog-ng.conf(5)

syslog-ng(8)

483syslog-ng.com

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

484syslog-ng.com

persist-tool

persist-tool — Display the content of the persist file

Synopsis

persist-tool [command] [options]

Description

NOTE: The persist-tool application is distributed with the syslog-ng Premium Edition system logging application,and is usually part of the syslog-ng package. The latest version of the syslog-ng application is available at thesyslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The persist-tool application is a utility that can be used to dump the content of the persist file, and manipulateits content.

WarningPersist-tool is a special tool for syslog-ng experts. Do use the tool unless you know exactly what you are doing.Misconfiguring it will result in irrecoverable damage to the persist file, without any warning.

NoteLimitations:

The persist-state functions can be used only with syslog-ng PE 5 LTS style persist file (SLP4). Older persistfiles are not supported.

■

■ Wildcard characters are not supported in file/directory names.

The dump command

dump [options] [persist_file]

Use the dump command to print the current content of the persist file in JSON format to the console.

The dump command has the following options:

--help or -? Display a brief help message.

Example:

persist-tool dump /opt/syslog-ng/var/syslog-ng.persist

The output looks like:

evtxlog_reader_curpos(Application) = { "version": 0, "big_endian": false,

"bookmark_xml": "<BookmarkList Direction='backward'>\r\n <Bookmark

485syslog-ng.com

Channel='Application' RecordId='2610170' IsCurrent='true'\

/>\r\n<\/BookmarkList>" }

affile_sd_curpos(/tmp/foo.txt) = { "version": 1, "big_endian": false,

"raw_buffer_leftover_size": 0, "buffer_pos": 0, "pending_buffer_end": 0, "buffer_size

": 0, "buffer_cached_eol": 0, "pending_buffer_pos": 0, "raw_stream_pos": 64,

"pending_raw_stream_pos": 0, "raw_buffer_size": 0, "pending_raw_buffer_size": 0,

"file_size": 64, "file_inode": 0, "run_id"

: 0 }

The generate command

generate [options]

Use the generate command to generate a persist file based on the content of the given configuration file. Itwill find all files and event sources that match the sources configured in the configuration file, and generate astate entry into the persist file for all of them. The position for the state entries will be set to the end of thesource, therefore syslog-ng will continue them from the last position instead of sending messages from thebeginning. Any other type of source will be skipped from persist generation. If the persist file already exists,persist-tool will display an error message and exit to prevent overwriting an already existing persist file.

To use the generate command: stop syslog-ng, remove the persist file, then execute persist-tool

generate.

The generate command has the following options:

--help or -? Display a brief help message.

--config-file=<config_file>

or -cRequired parameter. The syslog-ng configuration file that willbe the base of the generated persist file.

--force or -f Overwrite the persist file if it already exists.

WarningUse this option with care, persist-tool will not ask forconfirmation.

--output-dir=<directory> or-o

Required parameter. The directory where the persist file will begenerated to. The name of the persist file will besyslog-ng.persist.

Example:

/opt/syslog-ng/bin/persist-tool generate -c /opt/syslog-ng/etc/syslog-ng.conf -o

.

The output looks like:

New persist file generated: ./syslog-ng.persist

486syslog-ng.com

The generate-agent command

generate-agent [options]

NoteThis command is only available on Windows.

Use the generate-agent command to generate a persist file based on the registry by default, or use the contentof the given XML configuration file. It will find all files and event sources that match the sources configuredin the registry or configuration file, and generate a state entry into the persist file for all of them. The positionfor the state entries will be set to the end of the source, therefore syslog-ng will continue them from the lastposition instead of sending messages from the beginning. Any other type of source will be skipped from persistgeneration. If the persist file already exists, persist-tool will display an error message and exit to preventoverwriting an already existing persist file.

To use the generate command: stop syslog-ng, remove the persist file, then execute persist-tool

generate.

The generate-agent command has the following options:

--help or -? Display a brief help message.

--force or -f Overwrite the persist file if it already exists.

WarningUse this option with care, persist-tool will not ask forconfirmation.

--output-dir=<directory> or-o

The directory where the persist file will be generated to. Thename of the persist file will be syslog-ng.persist.

--xml-config or -x The XML configuration file of the syslog-ng Agent for Windowsapplication. If this parameter is set, the persist file will begenerated from the XML configuration file. If this parameter ismissing, the configuration available in the system registry willbe used.

Example:

persist-tool.exe generate-agent -f -x c:\eee.xml -o "c:\program files\syslog-ng

agent\var"

The add command

add [options] [input_file]

487syslog-ng.com

Use the add command to add or modify a specified state-entry in the persist file. The state-entry should be inthe same format as the dump command displays it. If the given state-entry already exists, it will be updated.Otherwise, a new value will be added. If the given persist state is invalid, it will be skipped.

To use the add command: use persist-tool dump to print the content of the current persist file, and redirectit to a file. Edit the content of this file. Use persist-tool add with this file to modify the persist.

The add command has the following options:

--help or -? Display a brief help message.

--output-dir=<directory> or-o

Required parameter. The directory where the persist file islocated at. The name of the persist file stored in this directorymust be syslog-ng.persist.

Example:

/opt/syslog-ng/bin/persist-tool add dump_persist -o .

The valid output looks like:

log_reader_curpos(Application) OK

affile_sd_curpos(/var/aaa.txt) OK

The invalid output looks like:

log_reader_curpos(Application) OK

wrong

FAILED (error: Invalid entry syntax)

affile_sd_curpos(/var/aaa.txt) OK

The rename-key command

rename-key [--old_key <old-key-in-persist-file>] [--new_key <new-key-in-persist-file>] [persist-file]

Use the rename-key command rename a specified state-entry in the persist file. The key of the state-entry torename should be the same as the dump command displays it. After renaming, the dump command of the persistfile shows both the old entry with the old key and a new entry with new key, but the old entry will be flaggedas unused. When syslog-ng PE is started and loads the modified persist file, it automatically deletes the oldentry.

Example:

persist-tool rename-key -o "java,org.syslog_ng.hdfs.HdfsDestination" -n

"java_dst(hdfs,hdfs://10.140.32.80:8020,var/dest/hadoop.txt)" syslog-ng.persist

Files

/opt/syslog-ng/bin/persist-tool

See also

syslog-ng.conf(5)

488syslog-ng.com

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

489syslog-ng.com

syslog-debun

syslog-debun — syslog-ng DEBUg buNdle generator

Synopsis

syslog-debun [options]

Description

NOTE: The syslog-debun application is distributed with the syslog-ng PE system logging application, andis usually part of the syslog-ng PE package. The latest version of the syslog-ng PE application is available atthe syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The syslog-debun tool collects and saves information about your syslog-ng PE installation, makingtroubleshooting easier, especially if you ask help about your syslog-ng PE related problem.

General Options

-h Display the help page.

-l Do not collect privacy-sensitive data, for example, process tree,fstab, and so on. If you use with -d, then the followingparameters will be used for debug mode:-Fev

-R <directory> The directory where syslog-ng Premium Edition installed insteadof /opt/syslog-ng.

-W <directory> Set the working directory, where the debug bundle will be saved.Default value: /tmp

Debug mode options

-d Start syslog-ng PE in debug mode, using the -Fedv

--enable-core options.Warning! Using this option under high message load mayincrease disk I/O during the debug, and the resulting debugbundle can be huge. To exit debug mode, press Enter.

-D <options> Start syslog-ng PE in debug mode, using the specifiedcommand-line options. To exit debug mode, press Enter. Fordetails on the available options, see syslog-ng(8) (p. 493).

-t <seconds> Run syslog-ng PE in noninteractive debug mode for <seconds>,and automatically exit debug mode after the specified numberof seconds.

-w <seconds> Wait <seconds> seconds before starting debug mode.

490syslog-ng.com

System call tracing

-s Enable syscall tracing (strace -f or truss -f). Note thatusing -s itself does not enable debug mode, only traces thesystem calls of an already running syslog-ng PE process. Totrace system calls in debug mode, use both the -s and -d options.

Packet capture options

-i <interface> Capture packets only on the specified interface, for example,eth0.

-p Capture incoming packets with tcpdump, using the followingfilter: port 514 or port 601 or port 53

-P <options> Capture incoming packets with tcpdump, using the specifiedfilter.

-t <seconds> Run syslog-ng PE in noninteractive debug mode for <seconds>,and automatically exit debug mode after the specified numberof seconds.

Examples

syslog-debun

Create a simple debug bundle, collecting information about your environment, for example, list packagescontaining the word: syslog, ldd of your syslog-binary, and so on.

syslog-debun -l

Similar to syslog-debun, but without privacy-sensitive information. For example, the following is NOTcollected: fstab, df output, mount info, ip / network interface configuration, DNS resolv info, and process tree.

syslog-debun -d

Similar to syslog-debun, but it also stops syslog-ng, then restarts it in debug mode (-Fedv --enable-core).To stop debug mode, press Enter. The output of the debug mode collected into a separate file, and also addedto the debug bundle.

syslog-debun -s

Trace the system calls (using strace or truss) of an already running syslog-ng PE process.

syslog-debun -d -s

Restart syslog-ng PE in debug mode, and also trace the system calls (using strace or truss) of the syslog-ngPE process.

syslog-debun -p

Run packet capture (pcap, using tcpdump) with the filter: port 514 or port 601 or port 53 Also waitsfor pressing Enter, like debug mode.

491syslog-ng.com

syslog-debun -p -t 10

Noninteractive debug mode: Similar to syslog-debun -p, but automatically exit from tcpdump mode after10 seconds.

syslog-debun -P "host 1.2.3.4" -D "-Fev --enable-core"

Change the packet-capturing filter from the default to host 1.2.3.4. Also change debugging parametersfrom the default to -Fev --enable-core. Since a timeout (-t) is not given, waits for pressing Enter.

syslog-debun -p -d -w 5 -t 10

Collect pcap and debug mode output following this scenario:

■ Start packet capture with default parameters (-p)

■ Wait 5 seconds (-w 5)

■ Stop syslog-ng

■ Start syslog-ng in debug mode with default parameters (-d)

■ Wait 10 seconds (-t 10)

■ Stop syslog-ng debuging

■ Start syslog-ng

■ Stop packet capturing

Files

/opt/syslog-ng/bin/loggen

See also

syslog-ng.conf(5)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

492syslog-ng.com

syslog-ng

syslog-ng — syslog-ng system logger application

Synopsis

syslog-ng [options]

Description

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page or the syslog-ng page.

The syslog-ng PE application is a flexible and highly scalable system logging application. Typically, syslog-ngis used to manage log messages and implement centralized logging, where the aim is to collect the log messagesof several devices on a single, central log server. The different devices - called syslog-ng clients - all runsyslog-ng, and collect the log messages from the various applications, files, and other sources. The clients sendall important log messages to the remote syslog-ng server, where the server sorts and stores them.

Options

--caps Run syslog-ng PE process with the specified POSIX capability flags.

■ If the --no-caps option is not set, and the host supports CAP_SYSLOG,syslog-ng PE uses the following capabilities: "cap_net_bind_service,cap_net_broadcast, cap_net_raw, cap_dac_read_search, cap_dac_override,cap_chown, cap_fowner=p cap_syslog=ep"

■ If the --no-caps option is not set, and the host does not supportCAP_SYSLOG, syslog-ng PE uses the following capabilities:"cap_net_bind_service, cap_net_broadcast, cap_net_raw,cap_dac_read_search,cap_dac_override, cap_chown, cap_fowner=p cap_sys_admin=ep"

For example:

/opt/syslog-ng/sbin/syslog-ng -Fv --caps

cap_sys_admin,cap_chown,cap_dac_override,cap_net_bind_service,cap_fowner=pi

Note that the capabilities are not case sensitive, the following command is alsogood: /opt/syslog-ng/sbin/syslog-ng -Fv --caps

CAP_SYS_ADMIN,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_FOWNER=pi

For details on the capability flags, see the following man pages:cap_from_text(3) and capabilities(7)

--cfgfile

<file> or -f<file>

Use the specified configuration file.

493syslog-ng.com

--chroot <dir>

or -C <dir>

Change root to the specified directory. The configuration file is read afterchrooting so, the configuration file must be available within the chroot. Thatway it is also possible to reload the syslog-ng configuration after chrooting.However, note that the --user and --group options are resolved beforechrooting.

--control

<file> or -c<file>

Set the location of the syslog-ng control socket. Default value:/var/run/syslog-ng.ctl

--debug or -d Start syslog-ng in debug mode.

--default-modules A comma-separated list of the modules that are loaded automatically. Modulesnot loaded automatically can be loaded by including the @module

<modulename> statement in the syslog-ng PE configuration file. The followingmodules are loaded by default: affile, afprog, afsocket, afuser, basicfuncs,csvparser, dbparser, syslogformat, aflogstore, diskq, confighash, afsql. Availableonly in syslog-ng Premium Edition 4.1 and later.

--enable-core Enable syslog-ng to write core files in case of a crash to help support anddebugging.

--fd-limit

<number>

Set the minimal number of required file descriptors (fd-s). This sets how manyfiles syslog-ng can keep open simultaneously. Default value: 4096. Note thatthis does not override the global ulimit setting of the host.

--foreground or-F

Do not daemonize, run in the foreground. When running in the foreground,syslog-ng PE starts from the current directory ($CWD) so it can create core files(normally, syslog-ng PE starts from $PREFIX/var).

--group <group>

or -g <group>

Switch to the specified group after initializing the configuration file.

--help or -h Display a brief help message.

--module-registry Display the list and description of the available modules. Note that not all ofthese modules are loaded automatically, only the ones specified in the--default-modules option. Available only in syslog-ng Premium Edition 4F1 and later.

--no-caps Run syslog-ng as root, without capability-support. This is the default behavior.On Linux, it is possible to run syslog-ng as non-root with capability-support ifsyslog-ng was compiled with the --enable-linux-caps option enabled.(Execute syslog-ng --version to display the list of enabled buildparameters.)

To run syslog-ng PE with specific capabilities, use the --caps option.

--persist-file

<persist-file>

Set the path and name of the syslog-ng.persist file where the persistentoptions and data are stored.

or -R<persist-file>

494syslog-ng.com

--pidfile

<pidfile> or -p<pidfile>

Set path to the PID file where the pid of the main process is stored.

--preprocess-into

<output-file>

After processing the configuration file and resolving included files and variables,write the resulting configuration into the specified output file. Available onlyin syslog-ng Premium Edition 4 F1 and later.

--process-mode

<mode>

Sets how to run syslog-ng: in the foreground (mainly used for debugging),in the background as a daemon, or in safe-background mode. By default,syslog-ng runs in safe-background mode. This mode creates a supervisorprocess called supervising syslog-ng , that restarts syslog-ng if it crashes.

--qdisk-dir

<path> or -Q<path>

Specify the location of the file used for disk-based buffering. By default, thisfile is located at /opt/syslog-ng/var/.

--stderr or -e Log internal messages of syslog-ng to stderr. Mainly used for debugging purposesin conjunction with the --foreground option. If not specified, syslog-ng willlog such messages to its internal source.

--syntax-only or-s

Verify that the configuration file is syntactically correct and exit. Note that thesyntax check does not catch duplicate configuration elements, for example, ifyou define two file destinations which point to the same file.

--user <user> or-u <user>

Switch to the specified user after initializing the configuration file (and optionallychrooting). Note that it is not possible to reload the syslog-ng configuration ifthe specified user has no privilege to create the /dev/log file.

--verbose or -v Enable verbose logging used to troubleshoot syslog-ng.

--version or -V Display version number and compilation information, and also the list and shortdescription of the available modules. For detailed description of the availablemodules, see the --module-registry option. Note that not all of these modulesare loaded automatically, only the ones specified in the --default-modulesoption.

--worker-threads Sets the number of worker threads syslog-ng PE can use, including the mainsyslog-ng PE thread. Note that certain operations in syslog-ng PE can use threadsthat are not limited by this option. This setting has effect only when syslog-ngPE is running in multithreaded mode. Available only in syslog-ng PremiumEdition 4 F1 and later. See The syslog-ng Premium Edition 6 LTS

Administrator Guide for details.

Setting default command-line options

You can set default settings for syslog-ng PE — syslog-ng PE will always run with these default command-lineparameters. You can specify your default settings in the following files:

■ /etc/default/syslog-ng

■ /etc/sysconfig/syslog-ng (only for RedHat platforms)

495syslog-ng.com

■ $SYSLOGNG_PREFIX/etc/default/syslog-ng, where $SYSLOGNG_PREFIX is the installationdirectory of syslog-ng PE. For version 4.0, this is /opt/syslog-ng

During startup, syslog-ng PE will automatically use the settings from these files if they exist. You can set thefollowing options:

MAXWAIT The number of seconds the init script will wait for syslog-ng PEto shut down properly. If the syslog-ng PE process does not shutdown during this period, it is terminated with a SIGKILL signal.Increase this value if you have lots of separate disk-buffer files(for example, to 60 seconds).

SYSLOGNG_OPTIONS A string of additional command-line options for the syslog-ngdaemon.

Files

/opt/syslog-ng/

/opt/syslog-ng/etc/syslog-ng.conf

See also

syslog-ng.conf(5)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

496syslog-ng.com

syslog-ng.conf

syslog-ng.conf — syslog-ng configuration file

Synopsis

syslog-ng.conf

Description

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page or the syslog-ng page.

The syslog-ng PE application is a flexible and highly scalable system logging application. Typically, syslog-ngis used to manage log messages and implement centralized logging, where the aim is to collect the log messagesof several devices on a single, central log server. The different devices - called syslog-ng clients - all runsyslog-ng, and collect the log messages from the various applications, files, and other sources. The clients sendall important log messages to the remote syslog-ng server, where the server sorts and stores them.

Basic concepts of syslog-ng PE

The syslog-ng application reads incoming messages and forwards them to the selected destinations. Thesyslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects, log paths define what syslog-ng does with a message,connecting the sources to the destinations. A log path consists of one or more sources and one or moredestinations: messages arriving from a source are sent to every destination listed in the log path. A log pathdefined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selectingonly messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messagessatisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segmentmessages into different fields to help processing the messages, while rewrite rules modify the messages byadding, replacing, or removing parts of the messages.

Configuring syslog-ng

■ The main body of the configuration file consists of object definitions: sources, destinations, log pathsdefine which log message are received and where they are sent. All identifiers, option names andattributes, and any other strings used in the syslog-ng configuration file are case sensitive. Objectsmust be defined before they are referenced in another statement. Object definitions (also calledstatements) have the following syntax:

object_type object_id {<options>};

497syslog-ng.com

• Type of the object: One of source, destination, log, filter, parser, rewrite rule, ortemplate.

• Identifier of the object: A unique name identifying the object. When using a reserved word as anidentifier, enclose the identifier in quotation marks.

TipUse identifiers that refer to the type of the object they identify. For example, prefix source objects withs_, destinations with d_, and so on.

• Parameters: The parameters of the object, enclosed in braces {parameters}.

• Semicolon: Object definitions end with a semicolon (;).For example, the following line defines a source and calls it s_internal.

source s_internal { internal(); };

The object can be later referenced in other statements using its ID, for example, the previous sourceis used as a parameter of the following log statement:

log { source(s_internal); destination(d_file); };

■ The parameters and options within a statement are similar to function calls of the C programminglanguage: the name of the option followed by a list of its parameters enclosed within brackets andterminated with a semicolon.

option(parameter1, parameter2); option2(parameter1, parameter2);

For example, the file() driver in the following source statement has three options: the filename(/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() optionalso has a parameter, while the flags() option has two parameters.

source s_tail { file("/var/log/apache/access.log"

follow-freq(1) flags(no-parse, validate-utf8)); };

Objects may have required and optional parameters. Required parameters are positional, meaningthat they must be specified in a defined order. Optional parameters can be specified in any orderusing the option(value) format. If a parameter (optional or required) is not specified, its defaultvalue is used. The parameters and their default values are listed in the reference section of theparticular object.

Example A.3. Using required and optional parametersThe unix-stream() source driver has a single required argument: the name of the socket to listen on.Optional parameters follow the socket name in any order, so the following source definitions have thesame effect:

source s_demo_stream1 {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

source s_demo_stream2 {

unix-stream("<path-to-socket>" group(log) max-connections(10)); };

498syslog-ng.com

■ Some options are global options, or can be set globally, for example, whether syslog-ng PE shoulduse DNS resolution to resolve IP addresses. Global options are detailed in Chapter 9, Global optionsof syslog-ng PE (p. 334).

options { use-dns(no); };

■ All identifiers, attributes, and any other strings used in the syslog-ng configuration file are casesensitive.

■ Objects can be used before definition.

■ To add comments to the configuration file, start a line with # and write your comments. These linesare ignored by syslog-ng.

# Comment: This is a stream source

source s_demo_stream {

unix-stream("<path-to-socket>" max-connections(10) group(log)); };

The syntax of log statements is as follows:

log {

source(s1); source(s2); ...

optional_element(filter1|parser1|rewrite1);

optional_element(filter2|parser2|rewrite2);

...

destination(d1); destination(d2); ...

flags(flag1[, flag2...]);

};

The following log statement sends all messages arriving to the localhost to a remote server.

source s_localhost { network(ip(127.0.0.1) port(1999)); };

destination d_tcp { network("10.1.2.3" port(1999) localport(999)); };

log { source(s_localhost); destination(d_tcp); };

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used,and other general points. Each option may have parameters, similarly to driver specifications. To set globaloptions, add an option statement to the syslog-ng configuration file using the following syntax:

options { option1(params); option2(params); ... };

Example A.4. Using global optionsTo disable domain name resolving, add the following line to the syslog-ng configuration file:

options { use-dns(no); };

The sources, destinations, and filters available in syslog-ng are listed below. For details, the syslog-ngDocumentation page.

DescriptionNameCollects Windows eventlog messages.eventlog()

499syslog-ng.com

DescriptionNameOpens the specified file and reads messages.file()

Messages generated internally in syslog-ng.internal()

Receives messages from remote hosts using theBSD-syslog protocol over IPv4 and IPv6. Supports theTCP, UDP,RLTP™, and TLS network protocols.

network()

Opens the specified named pipe and reads messages.pipe()

Opens the specified application and reads messagesfrom its standard output.

program()

Collects logs from tables of relational databasesql()

Opens the specified STREAMS device on Solaris systemsand reads incoming messages.

sun-stream(), sun-streams()

Listens for incoming messages using the newIETF-standard syslog protocol.

syslog()

Automatically detects which platform syslog-ng PE isrunning on, and collects the native log messages of thatplatform.

system()

Collects messages directly from the journal of platformsthat use systemd.

systemd-journal()

Collects messages from the journal using a socket onplatforms that use systemd.

systemd-syslog()

Opens the specified unix socket in SOCK_DGRAM modeand listens for incoming messages.

unix-dgram()

Opens the specified unix socket in SOCK_STREAMmodeand listens for incoming messages.

unix-stream()

Table A.1. Source drivers available in syslog-ng

DescriptionNameSends messages to an Elasticsearch server. Theelasticsearch2 driver supports Elasticsearch version2 and newer.

elasticsearch and elasticsearch2

Writes messages to the specified file.file()

Sends messages into a file on a Hadoop DistributedFile System (HDFS) node.

hdfs

Publishes log messages to the Apache Kafka messagebus, where subscribers can access them.

kafka

Writes messages to the specified binary logstore file.logstore()

Sends messages to a MongoDB database.mongodb()

500syslog-ng.com

DescriptionNameSends messages to a remote host using the BSD-syslogprotocol over IPv4 and IPv6. Supports the TCP,UDP,RLTP™, and TLS network protocols.

network()

Writes messages to the specified named pipe.pipe()

Forks and launches the specified program, and sendsmessages to its standard input.

program()

Sends messages send mail to trusted recipients, througha controlled channel using the SMTP protocol.

smtp()

Sends messages to the specified remote host using theSNMP v2c or v3 protocol.

snmp()

Sends messages into an SQL database. In addition tothe standard syslog-ng packages, the sql() destination

sql()

requires database-specific packages to be installed.Refer to the section appropriate for your platform inChapter 3, Installing syslog-ng (p. 36).

Sends messages to the specified remote host using theIETF-syslog protocol. The IETF standard supports

syslog()

message transport using the UDP, TCP, and TLSnetworking protocols.

Sends messages to the specified unix socket inSOCK_DGRAM style (BSD).

unix-dgram()

Sends messages to the specified unix socket inSOCK_STREAM style (Linux).

unix-stream()

Sends messages to the terminal of the specified user,if the user is logged in.

usertty()

Table A.2. Destination drivers available in syslog-ng

DescriptionNameFilter messages based on the sending facility.facility()

Call another filter function.filter()

Filter messages based on the sending host.host()

File-based whitelisting and blacklisting.inlist()

Filter messages based on their priority.level() or priority()

Use a regular expression to filter messages based on aspecified header or content field.

match()

Use a regular expression to filter messages based ontheir content.

message()

Filter messages based on the IP address of the sendinghost.

netmask()

Filter messages based on the sending application.program()

501syslog-ng.com

DescriptionNameSelect messages of the specified syslog-ng PE sourcestatement.

source()

Select messages having the specified tag.tags()Table A.3. Filter functions available in syslog-ng PE

Files

/opt/syslog-ng/

/opt/syslog-ng/etc/syslog-ng.conf

See also

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

502syslog-ng.com

syslog-ng-ctl

syslog-ng-ctl — Display message statistics and enable verbose, debug and trace modes in syslog-ng PremiumEdition

Synopsis

syslog-ng-ctl [command] [options]

Description

NOTE: The syslog-ng-ctl application is distributed with the syslog-ng Premium Edition system loggingapplication, and is usually part of the syslog-ng package. The latest version of the syslog-ng application isavailable at the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The syslog-ng-ctl application is a utility that can be used to:

■ enable/disable various syslog-ng messages for troubleshooting

■ display statistics about the processed messages

■ reload the configuration of syslog-ng PE

■ stop syslog-ng PE.

Enabling troubleshooting messages

command [options]

Use the syslog-ng-ctl <command> --set=on command to display verbose, trace, or debug messages. Ifyou are trying to solve configuration problems, the verbose (and occasionally trace) messages are usuallysufficient. Debug messages are needed mostly for finding software errors. After solving the problem, do notforget to turn these messages off using the syslog-ng-ctl <command> --set=off. Note that enablingdebug messages does not enable verbose and trace messages.

Use syslog-ng-ctl <command> without any parameters to display whether the particular type of messagesare enabled or not.

If you need to use a non-standard control socket to access syslog-ng, use the syslog-ng-ctl <command>

--set=on --control=<socket> command to specify the socket to use.

verbose Print verbose messages. If syslog-ng was started with the--stderr or -e option, the messages will be sent to stderr. Ifnot specified, syslog-ng will log such messages to its internalsource.

trace Print trace messages of how messages are processed. If syslog-ngwas started with the --stderr or -e option, the messages will

503syslog-ng.com

be sent to stderr. If not specified, syslog-ng will log suchmessages to its internal source.

debug Print debug messages. If syslog-ng was started with the--stderr or -e option, the messages will be sent to stderr. Ifnot specified, syslog-ng will log such messages to its internalsource.

Example:

syslog-ng-ctl verbose --set=on

The stats command

stats [options]

Use the stats command to display statistics about the processed messages. For details about the displayedstatistics, see the syslog-ng Documentation page. The stats command has the following options:

--control=<socket> or -c Specify the socket to use to access syslog-ng. Only needed whenusing a non-standard socket.

--reset=<socket> or -r Reset all statistics to zero, except for the stored counters. (Thestored counters show the number of messages stored in themessage queue of the destination driver, waiting to be sent tothe destination.)

Example:

syslog-ng-ctl stats

An example output:

src.internal;s_all#0;;a;processed;6445

src.internal;s_all#0;;a;stamp;1268989330

destination;df_auth;;a;processed;404

destination;df_news_dot_notice;;a;processed;0

destination;df_news_dot_err;;a;processed;0

destination;d_ssb;;a;processed;7128

destination;df_uucp;;a;processed;0

source;s_all;;a;processed;7128

destination;df_mail;;a;processed;0

destination;df_user;;a;processed;1

destination;df_daemon;;a;processed;1

destination;df_debug;;a;processed;15

destination;df_messages;;a;processed;54

destination;dp_xconsole;;a;processed;671

dst.tcp;d_network#0;10.50.0.111:514;a;dropped;5080

dst.tcp;d_network#0;10.50.0.111:514;a;processed;7128

dst.tcp;d_network#0;10.50.0.111:514;a;stored;2048

destination;df_syslog;;a;processed;6724

destination;df_facility_dot_warn;;a;processed;0

destination;df_news_dot_crit;;a;processed;0

504syslog-ng.com

destination;df_lpr;;a;processed;0

destination;du_all;;a;processed;0

destination;df_facility_dot_info;;a;processed;0

center;;received;a;processed;0

destination;df_kern;;a;processed;70

center;;queued;a;processed;0

destination;df_facility_dot_err;;a;processed;0

Displaying license-related information

syslog-ng-ctl show-license-info [options]

The syslog-ng PE application uses a license in server mode to determine the maximum number of hosts thatare allowed to connect. Use the syslog-ng-ctl show-license-info command to display license-relatedinformation the number of hosts currently logging to your server. This helps you to plan your capacity, to checkyour license usage, and to detect client misconfiguration that can result in a license miscount anomaly. Notethat in client or relay mode, syslog-ng PE does not require a license.

The syslog-ng-ctl show-license-info command displays the following information. In case of anunlimited license, or in client or relay mode, only the license type is displayed:

■ License Type: none, limited, unlimited

■ Host Limit: the maximum number of hosts that are allowed to connect.

■ Currently Used Slots: the number of currently used host slots

■ Usage: the percent of used host slots

■ Licensed Clients: the list of hostnames that are stored in the license module

The syslog-ng-ctl show-license-info command has the following options:

--json or -J Print license-related information in JSON format.

Example:

syslog-ng-ctl show-license-info

An example output:

License-Type: limited

Host-Limit: 10

Currently-Used-Slots: 7

Usage: 70%

Licensed-Clients:

192.168.0.1

192.168.0.2

192.168.0.3

192.168.1.4

192.168.1.5

Example:

syslog-ng-ctl show-license-info --json

505syslog-ng.com

An example output:

{

"license_type": "limited",

"host_limit": 10,

"currently_used_slots": 7,

"usage": "70%",

"licensed_clients": [

"xy.testdomain",

"testhost",

"192.168.0.3",

"test_host",

"192.168.1.5"

]

}

Example:

syslog-ng-ctl show-license-info

in case of an unlimited license

An example output:

$ syslog-ng-ctl show-license-info

License-Type: unlimited

Example:

syslog-ng-ctl show-license-info

if syslog-ng PE is in client or relay mode

An example output:

$ syslog-ng-ctl show-license-info

License-Type: none

Reloading the configuration

syslog-ng-ctl reload [options]

Use the syslog-ng-ctl reload command to reload the configuration file of syslog-ng PE without havingto restart the syslog-ng PE application. The syslog-ng-ctl reloadworks like a SIGHUP (-1). On MicrosoftWindows, this is the only way to reload the configuration of syslog-ng PE.

Stopping syslog-ng PE

syslog-ng-ctl stop [options]

Use the syslog-ng-ctl stop command to stop the syslog-ng PE application. The syslog-ng-ctl stop

works like a SIGHUP (-15) on Linux and Unix systems. On Microsoft Windows, this is the only way to gracefullystop syslog-ng PE if it is running in the foreground.

506syslog-ng.com

Files

/opt/syslog-ng/sbin/syslog-ng-ctl

See also

The syslog-ng Documentation page

syslog-ng.conf(5)

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

507syslog-ng.com

syslog-ng-query

syslog-ng-query — Query metrics and statistic data from a running syslog-ng Premium Edition instance

Synopsis

syslog-ng-query [command] [options]

Description

NOTE: The syslog-ng-query application is distributed with the syslog-ng Premium Edition system loggingapplication, and is usually part of the syslog-ng package. The latest version of the syslog-ng application isavailable at the syslog-ng page.

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ngDocumentation page.

The syslog-ng-query application is a utility that can be used to query metrics and statistic data from arunning syslog-ng Premium Edition instance.

The syslog-ng PE application stores various data, metrics, and statistics in a hierarchical data structure tree(syslog-ng is the root node of the tree). The nodes of the tree can contain properties (but not every nodecontains properties). Every property has a name and a value. For example:

[syslog-ng]

|

|_[destinations]-[network]-[tcp]->[stats]->{received:12;dropped:2}

|

|_[sources]-[sql]-[stats]->{reveived:501;dropped:0}

You can query the nodes of this tree, and also use filters to select the information you need. A query is actuallya path in the tree. You can also use the ? and * wildcards. For example:

■ Select every property: *

■ Select all dropped value from every stats node: *.stats.dropped

The nodes and properties available in the tree depend on your syslog-ng PE configuration (that is, the sources,destinations, and other objects you have configured), and also on your stats-level() settings.

The list command

syslog-ng-query list

Use the syslog-ng-query list command to display the list of metrics that syslog-ng PE collects about theprocessed messages. For details about the displayed metrics, see the syslog-ng Documentation page.

An example output:

stats

center.received.stats.processed

center.queued.stats.processed

508syslog-ng.com

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.dropped

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.processed

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.stored

destination.d_elastic.stats.processed

source.s_tcp.stats.processed

source.severity.7.stats.processed

source.severity.0.stats.processed

source.severity.1.stats.processed

source.severity.2.stats.processed

source.severity.3.stats.processed

source.severity.4.stats.processed

source.severity.5.stats.processed

source.severity.6.stats.processed

source.facility.7.stats.processed

source.facility.16.stats.processed

source.facility.8.stats.processed

source.facility.17.stats.processed

source.facility.9.stats.processed

source.facility.18.stats.processed

source.facility.19.stats.processed

source.facility.20.stats.processed

source.facility.0.stats.processed

source.facility.21.stats.processed

source.facility.1.stats.processed

source.facility.10.stats.processed

source.facility.22.stats.processed

source.facility.2.stats.processed

source.facility.11.stats.processed

source.facility.23.stats.processed

source.facility.3.stats.processed

source.facility.12.stats.processed

source.facility.4.stats.processed

source.facility.13.stats.processed

source.facility.5.stats.processed

source.facility.14.stats.processed

source.facility.6.stats.processed

source.facility.15.stats.processed

source.facility.other.stats.processed

global.payload_reallocs.stats.processed

global.msg_clones.stats.processed

global.sdata_updates.stats.processed

tag..source.s_tcp.stats.processed

Displaying metrics and statistics

syslog-ng-query sum [options]

The syslog-ng-query sum <query> command lists the nodes that match the query, and their values. Forexample, the syslog-ng-query sum "destination*" command lists the configured destinations, and themetrics related to each destination. An example output:

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.dropped:

0

509syslog-ng.com

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.processed:

0

destination.java.d_elastic#0.java_dst(ElasticSearch,elasticsearch-syslog-ng-test,t7cde889529c034aea9ec_micek).stats.stored:

0

destination.d_elastic.stats.processed: 0

The syslog-ng-query sum command has the following options:

--aggregate Add up the result of each matching node and return only a singlenumber.

For example, the syslog-ng-query sum --aggregate "destination*.dropped" command displaysthe number of messages dropped by the syslog-ng PE instance.

Files

/opt/syslog-ng/sbin/syslog-ng-query

See also

The syslog-ng Documentation page

syslog-ng.conf(5)

syslog-ng(8)

NoteFor the detailed documentation of syslog-ng PE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng FAQ or the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng Blog.

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

510syslog-ng.com

syslog-windebun.ps1

syslog-windebun.ps1 — syslog-ng WINdows DEBUg buNdle generator PowerShell script

Synopsis

powershell C:\PATH\TO\syslog-debun.ps1

Description

The syslog-windebun application is a powershell script that collects information about its environment intoa file, to help troubleshoot syslog-ng Premium Edition and syslog-ng Agent for Windows installations.

The syslog-windebun application application is distributed with the syslog-ng PE system logging application,and is usually part of the syslog-ng Premium Edition package or the syslog-ng Agent for Windows package.The latest version of the is available at the syslog-ng Downloads page. You can also contact the One IdentitySupport Team and request the latest version of the script.

Using syslog-windebun

The application requires PowerShell 2.0 or later. Administrator privileges are not required.

To use syslog-windebun, run the application in an interactive powershell terminal. On 64-bit Windows, usethe 64-bit PowerShell terminal.

Example:

powershell -NoProfile -ExecutionPolicy RemoteSigned C:\PATH\TO\syslog-debun.ps1

The script collects the information about the syslog-ng PE environment into a file, and displays the location ofthis file. When creating a support ticket, attach this file to the ticket to help our support team troubleshoot yourproblem. The following is an example output.

PS C:\Users\balabit> C:\Users\balabit\Downloads\syslog-windebun.ps1

syslog-ng Agent is installed

Start gathering Agent related information

Finished gathering syslog-ng Agent information

syslog-ng PE is installed

Start gathering syslog-ng PE related information

Finished gathering syslog-ng PE related information

Starting to gather system related information

Finished gathering system related information.

The resulted zip file is found where you started running the program.

Please send the following file to Balabit Support:

C:\Users\balabit\syslog-windebun-CjZ.zip

Data collection policy

The syslog-windebun application collects the following information into a file.

511syslog-ng.com

■ Routing information

■ System information

■ Network statistics

■ Network port information

■ IP configuration

■ The XML configuration file of syslog-ng Agent for Windows

■ Registry information of syslog-ng Agent for Windows

■ The content of the etc folder of syslog-ng Premium Edition

■ PowerShell and .NET version

■ syslog-ng Premium Edition and syslog-ng Agent for Windows version

■ Installation logs of syslog-ng Premium Edition and syslog-ng Agent for Windows version

Files

/opt/syslog-ng/bin/dqtool

Author

This manual page was written by the BalaBit Documentation Team <documentation@balabit.com>.

Copyright

Copyright© 2000-2018One Identity. Published under the Creative Commons Attribution-Noncommercial-NoDerivative Works (by-nc-nd) 3.0 license. For details, see https://creativecommons.org//. The latest version isalways available at the syslog-ng Documentation page.

512syslog-ng.com

Appendix B. END USER LICENSE AGREEMENTFOR BALABIT PRODUCT (EULA)

SUBJECT OF THE LICENSE AGREEMENTThis License Agreement is entered into by and between Licensor (as defined below) and you as an end-user(hereinafter Licensee) and sets out the terms and conditions under which Licensee and/or Licensee's AuthorizedSubsidiaries may use the Balabit Product (as defined below) under this License Agreement.

DEFINITIONSIn this License Agreement, the following words shall have the following meanings:

DescriptionNameAny third party software that is a not a Balabit Productcontained in the install package of the Balabit Product.

Annexed Software

The companies which are affiliates, a subsidiary or aparent company of the Licensor.

Balabit Group

Any software (other than the Annexed Software),hardware, virtual hardware or service licensed, sold,

Balabit Product

or provided by Licensor including any installation,education, support and warranty services, or anyproduct covered by one or more copyrights owned bya company of the Balabit Group.

License Agreement The present Balabit Product LicenseAgreement.

License Agreement

As indicated on the invoice for the Balabit Product,Balabit-Europe Kft., a limited liability company,

Licensor

incorporated and registered with the BudapestMetropolitan Court as Court of Registration undernumber Cg.01-09-186546 whose registered office is atH-1117 Budapest, Aliz u. 2., or Balabit IT SecurityDeutschland GmbH, a limited liability company,incorporated and registered with the AmtsgerichtMünchen under number HRB 167365, whose registeredoffice is at Stefan-George-Ring 29, D-81929 München,or Balabit Corp., a New York corporation, havingoffices at 40 Wall Street, New York, NY 10005.

Any documentation referring to the Balabit Product orany module thereof, including the administration guide,

Product Documentation

the product description, the installation guide and userguides and manuals.

The document signed by Licensor which contains a)identification data of the Licensee; b) the name of the

Certificate of Authenticity

513syslog-ng.com

SUBJECT OF THE LICENSE AGREEMENT

DescriptionNameBalabit Product and the designation of licensed modulesthereof; c) an explicit warning that the validity of thecertificate is subject to the acceptance by the Licenseeof the terms and conditions of the EULA; and d)information with regard to on-line registration, accessto upgrade and support services and Product UsageTerms.

Sets forth the conditions (the usage environment andlimitations) under which the Balabit Product may beused by the Licensee.

Product Usage Terms

A period of twelve (12) months from the date ofdelivery of the Balabit Product to Licensee.

Warranty Period

Table B.1. Words and expressions

LICENSE GRANTS AND RESTRICTIONSA. Subject to payment of the License Fee and the terms and conditions of this License Agreement, the applicableCertificate of Authenticity and the Product Usage Terms, Licensor hereby grants to Licensee, a limited, personal,non-exclusive and non-transferable license to use Balabit Product (“License”) for its own internal businesspurposes. This License does not convey any license or right, express or implied, to manufacture, duplicate orotherwise copy or reproduce the Balabit Product or any part thereof. This License is transferable only with theprior written approval of Licensor, which may be withheld in Licensor's sole discretion.

B. Licensee shall use the Balabit Product in accordance with the conditions set by the Product Usage Termsand the Certificate of Authenticity, especially in the configuration and subject to the quantities specified inthese documents.

C. All modules of the Balabit software will be delivered to Licensee. However, Licensee shall not be entitledto use any module which is not specified in the applicable Certificate of Authenticity. Access rights to modulesand IP connections are controlled by an “electronic key” accompanying the Balabit Product.

D. Licensee shall be entitled to make one back-up copy of the Balabit software that is licensed to it.

E. Licensee shall make the Balabit Product available solely to its own employees and those of the AuthorizedSubsidiaries that are listed in the applicable Certificate of Authenticity or in the related agreement between theLicensor and the Licensee (e.g. Master Purchase Agreement) and shall take special care to protect the BalabitProduct from any unauthorized access.

F. Licensee shall, in five (5) working days properly answer any queries of Licensor regarding the actual usageconditions of the Balabit Product that may differ or allegedly differ from the License conditions set forth in theProduct Usage Terms.

G. Licensee shall install the code permitting the usage of the Balabit Product strictly in accordance and to theprovisions defined for it by Licensor. Licensee shall not modify or cancel the Balabit Product functions thereofthat inspect the usage of the software. Configuration settings of the Balabit Product in accordance with thepossibilities offered by the system shall not be construed as modification of the software.

514syslog-ng.com

LICENSE GRANTS AND RESTRICTIONS

H. Licensee shall not copy, distribute, market, sell, lease, sublicense, assign or otherwise transfer the BalabitProduct to any third party, or use the Balabit Product in a manner that (i) infringes the intellectual propertyrights or otherwise violates the rights of any third party, or (ii) violates applicable law, (iii) provides for orallows timesharing, rental or use of the Balabit Product in a service bureau or as a provider of services utilizingthe Balabit Product, or (iv) allow a competitor of Balabit to use or have access to the Balabit Product. Licenseeshall not remove or modify any program markings or any notice of Balabit's or proprietary rights.

I. Licensee shall not (i) modify, translate, decompile or reverse engineer the Balabit Product, (ii) attempt tocreate the source code from the executable or object code of the Balabit Product by reverse engineering ordisassembling or otherwise adopt, manipulate the executable or object code of the Balabit Product, (iii) createa derivative work based upon the Balabit Product or the Product Documentation or permit a third party to dothe same, or (iv) modify, tamper with, reverse engineer, reverse compile or disassemble the electronic key forthe Balabit Product.

(v) Notwithstanding the foregoing, Licensee shall be entitled to analyze the structure of the Balabit Product(decompilation or reverse- engineering) only if necessary to coordinate operation of the Balabit Product withsoftware developed by a third party, and only if Licensor does not provide such information within 60 (sixty)days from the receipt of such a request. Such analysis of the structure of the Balabit Product is strictly limitedto those parts of the Balabit Product which are necessary for concurrent operation with the third party softwareand is subject to either a) Licensor's prior written consent, or b) the failure of Licensor to provide the aforesaidinformation within the aforesaid 60 (sixty) day period.

Any information obtained by Licensee as a result of applying subsection (v) (a) cannot be used for any purposesother than concurrent operation of the third party software with the Balabit Product, (b) shall not be disclosedto third parties unless it is necessary to disclose it to the owner of the third party software for concurrent operationwith the Balabit Product; (c) shall not be used for the development, production or distribution of software whichis the same as or similar to the Balabit Product in features or in functionality, or (d) for any other act or purposethat violates Licensor's copyrights in the Balabit Product.

(vi) Notwithstanding the foregoing, Licensee shall be entitled to analyze the structure of those components ofthe Balabit Product (decompilation or reverse-engineering) that fall under GNU Lesser General Public License3.0 (https://www.gnu.org/licenses/lgpl-3.0.en.html) according to the Product Documentation. Any activity underthis sub-clause shall be made in full compliance with the license terms referred to herein.

J. Licensee shall comply with all terms and conditions made applicable to all Annexed Software contained inthe same install package with the Balabit Product by the owner of the Annexed Software. Licensor does notgrant any license rights to any Annexed Software by including it with a Balabit Product in the same installpackage. Such rights must be acquired by Licensee directly from the owner of the Annexed Software.

K. Any usage of the Balabit Product exceeding the limits and restrictions defined in the Certificate of Authenticityshall be a material breach of the License Agreement and Licensee shall be fully liable to Licensor for suchbreach, including for monetary damages and/or termination of this License Agreement and the Master PurchaseAgreement and any Order made thereunder.

L. Licensee shall have the right to obtain and use content updates of the Balabit Product only if Licenseeconcludes a support contract that includes such content updates (maintenance of the software), or if Licenseehas otherwise separately acquired the right to obtain and use such content updates. This License Agreementdoes not otherwise permit Licensee to obtain and use content updates.

M. Licensor expressly reserves all rights not expressly granted herein.

515syslog-ng.com

LICENSE GRANTS AND RESTRICTIONS

CONFIDENTIALITYA. “Confidential Information” means any business, marketing, technical, scientific or other information disclosedby the Balabit Group which, at the time of disclosure is designated as confidential (or like designation), isdisclosed in circumstances of confidence, or would be understood by the parties (or their Affiliates), exercisingreasonable business judgment, to be confidential.

B. License acknowledges that the Balabit Product, the Product Documentation and related materials are thetrade secrets and Confidential Information of the Balabit Group. Licensee agrees to keep confidential allconfidential information of the Balabit Group including but not limited to the Balabit Product, the ProductDocumentation and related materials. Licensee agrees to use all confidential information of the Balabit Groupincluding but not limited to the Balabit Product, the Product Documentation and related materials only asexpressly permitted by this Agreement.

C. Licensee shall retain the Confidential Information of the Balabit Group in confidence and shall use anddisclose it solely for the purpose of, and in accordance with, this License Agreement. Licensee shall onlydisclose Confidential Information of the Balabit Group to those of its employees with a need to know suchConfidential Information. Licensee shall use the same degree of care as it uses to protect its own confidentialinformation of a similar nature, but no less than reasonable care, to prevent the unauthorized use or disclosureof the Balabit Group's Confidential Information.

INTELLECTUAL PROPERTY RIGHTSA. All right, title, and interest in and to the Balabit Product, including all patents, trademarks, trade names,inventions, know-how, trade secrets and all other intellectual property rights relating to the design, manufacture,operation or service of the Balabit Product are owned by one or more of the companies of the Balabit Group.No right or interest in any of such intellectual property rights is transferred to Licensee by this License otherthan the right and license to use the Balabit Product modules licensed hereunder in accordance with this LicenseAgreement and the Product Usage Terms.

B. Licensee will advise its Authorized Subsidiaries, if any, of and assure compliance with the restrictionscontained in the License Agreement, including those relating to the Confidential Information and proprietaryproperty of the Balabit Group. Licensee shall implement adequate security measures to protect such trade secretsand confidential information.

C. The use by Licensee of any of the intellectual property rights in the Balabit Product is authorized only forthe purposes set forth herein, and upon termination of this License Agreement for any reason, such authorizationshall cease and Licensee shall immediately cease the use of the Balabit Product.

WARRANTIESA. Licensor warrants that during the Warranty Period, the Balabit provided hardware upon which the BalabitProduct is installed provided to Licensee by Licensor (“Appliance”) will be free of defects of material orworkmanship under normal use. Licensor will replace any defective Appliance returned to it, accompanied bya dated proof of purchase that is within the Warranty Period, at no charge to Licensee. Upon receipt of theallegedly defective Appliance, Licensor will at its option, deliver a replacement Appliance or Licensor's currentequivalent Appliance to Licensee at no additional cost. Licensor will bear all delivery charges to Licensee forthe replacement Appliance.

B. In the event Licensee uses the Balabit Product in conjunction with any third party software, Licensor shallnot be liable for any errors in the operation of the Balabit Product that is due to the third party software.

516syslog-ng.com

CONFIDENTIALITY

C. Licensor warrants that during the Warranty Period, the Balabit Product software without unauthorizedmodification shall perform in substantial compliance with the Product Documentation accompanying the BalabitProduct, when it is used in normal use (i) on that hardware or virtual appliance for which it was installed and(ii) in compliance with the provisions of the Product Documentation and the Product Usage Terms. If the BalabitProduct fails to so operate, Licensee shall promptly notify Licensor (the date of the notification sent to Licensorshall be deemed to be the date of the failure) and Licensee shall do its best to mitigate the consequences of thatfailure until Licensor can address the failure to operate in accordance with the aforesaid documentation. If thefailure is reported by Licensee to Licensor within the Warranty Period, Licensor's sole obligation and liabilityfor breach of this warranty is, at Licensor's sole option, either: (i) to correct such failure, or (ii) to replace thedefective Balabit Product.

D. Where the Balabit Product has not been acquired directly from Licensor, Licensee must contact the entitythat has sold the license to the Balabit Product to Licensee in order to exercise its rights under this warranty.Licensor will not provide to Licensee any after-sale warranty if Licensor has not sold the license to the BalabitProduct directly to Licensee.

E. EXCEPT AS SET FORTH IN THIS LICENSE AGREEMENT, LICENSOR MAKES NO WARRANTIESOF ANY KIND WITH RESPECT TO THE BALABIT PRODUCT. TO THE MAXIMUM EXTENTPERMITTED BY APPLICABLE LAW, LICENSOR DISCLAIMS ANY OTHER WARRANTIES, INCLUDINGBUT NOT LIMITED TO ANY IMPLIED WARRANTIES OF SATISFACTORY QUALITY,MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT OFINTELLECTUAL PROPERTY RIGHTS.

LICENSE FEEA. The Certificate of Authenticity and the Product Usage Term contain the details of the purchased Licenseand usage limitations. This information serves as the calculation base of the License fee. Licensee acknowledgesthat payment of the License fee is a condition of lawful usage.

B. License fees do not include any installation or post sale charges, taxes, duties, etc., all of which are for theaccount of Licensee. Applicable taxes shall be added to all invoices to Licensee for License fees.

C. The license rights to the Balabit Product are transferred to the Licensee only when Licensee pays the Licensefee to Licensor. In case of non-payment Licensor has right to terminate, or rescind the License Agreement withimmediate effect and Licensee shall promptly cease all use of the Balabit Product and return it to Licensor atits own cost and expense and shall be liable for its unlawful usage and the early termination.

LIMITATION OF LIABILITYSOME STATES AND COUNTRIES, INCLUDING MEMBER COUNTRIES OF THE EUROPEAN UNION,DO NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY FOR INCIDENTAL ORCONSEQUENTIAL DAMAGES AND, THEREFORE, THE FOLLOWING LIMITATION OR EXCLUSIONMAY NOT APPLY TO THIS LICENSE AGREEMENT IN THOSE STATES AND COUNTRIES.

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW AND REGARDLESS OF WHETHERANY REMEDY SET OUT IN THIS LICENSE AGREEMENT FAILS OF ITS ESSENTIAL PURPOSE, INNO EVENT SHALL LICENSOR BE LIABLE TO LICENSEE FOR ANY SPECIAL, EXEMPLARY,CONSEQUENTIAL, INDIRECT, PUNITIVE, OR SIMILAR DAMAGES OR LOST PROFITS OR LOSTDATA ARISING OUT OF THE USE OR INABILITY TO USE THE BALABIT PRODUCT EVEN IFLICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

517syslog-ng.com

LICENSE FEE

IN NO CASE SHALL LICENSOR'S TOTAL LIABILITY UNDER THIS LICENSE AGREEMENT EXCEEDTHE FEES RECEIVED BY LICENSOR FOR THE BALABIT PRODUCT LICENSED UNDER THISLICENSE AGREEMENT.

NOTWITHSTANDING ANYTHING SET FORTH IN THIS AGREEMENT TO THE CONTRARY, IN NOEVENT SHALL LICENSOR BE LIABLE FOR ANY DAMAGES CAUSED BY THE USAGE OF THEBALABIT PRODUCT WHICH IS NOT IN ACCORDANCE WITH THE PRODUCT DOCUMENTATIONAND THE PRODUCT USAGE TERMS.

TERM AND TERMINATIONA. This License Agreement shall come into effect on the day when the Licensee declares acceptance of its termsand conditions, provided that the License Fee has been fully paid. Either the signing a copy of the LicenseAgreement by the Licensee's duly authorized representative, or Licensee “clicking” on the "Confirmation"button (“I have read and agree ...”) with regard to this License Agreement at the beginning of the installationprocess of the Balabit Product shall be deemed to be acceptance by the Licensee to the terms and conditionsof the License Agreement. The Buyer represents and warrants that the members of its IT staff working on theinstallation of the Products (either with or without the Supplier's installation personnel) are authorized to bindthe Buyer to this License Agreement by signing a copy of the License Agreement or “clicking” on theConfirmation button.

B. Licensee may terminate the License Agreement at any time by written notice sent to Licensor and bysimultaneously destroying all copies of the Balabit Product licensed under this License Agreement and certifyingto Licensor that it has done so.

C. Licensor may terminate this License Agreement with immediate effect by written notice to Licensee, ifLicensee is in material or persistent breach of the License Agreement and either that breach is incapable ofremedy or Licensee shall have failed to remedy that breach within 30 (thirty) days after receiving written noticerequiring it to remedy that breach. In such a case, Licensee must immediately destroy all copies of the BalabitProduct licensed under this License Agreement, the Product Documentation and all other materials containingthe Confidential Information of Licensor and certify to Licensor that it has done so.

D. The provisions of this Agreement relating to confidentiality, applicable law and jurisdiction, notices,indemnification, disclaimers and limits of liability shall survive the expiration or termination of this Agreementfor any reason.

AMENDMENTSExcept as expressly provided in this License Agreement, no amendment or variation of this License Agreementshall be effective unless in writing and signed by a duly authorized representative of both parties hereto.

WAIVERThe failure of a party to exercise or enforce any right under this License Agreement shall not be deemed to bea waiver of that right nor operate to bar the exercise or enforcement of such right or any other right at any timeor times thereafter.

SEVERABILITYIf any part of this License Agreement becomes invalid, illegal or unenforceable, the parties shall in such anevent negotiate in good faith in order to agree on the terms of a mutually satisfactory provision to be substituted

518syslog-ng.com

TERM AND TERMINATION

for the invalid, illegal or unenforceable provision which as nearly as possible validly gives effect to theirintentions as expressed in this License Agreement.

NOTICESAny notice required to be given pursuant to this License Agreement shall be in writing and shall be given bydelivering the notice by hand, or by sending the same by prepaid first class post (airmail if to an address outsidethe country of posting) or by recognized courier service such as Federal Express to the address of the relevantparty. Any notice given according to the above procedure shall be deemed to have been given at the time ofdelivery (if delivered by hand) and when received (if sent by post or courier service).

APPLICABLE LAW AND JURISDICTIONThis Agreement shall be construed, interpreted and the rights of the parties determined

a) in case of US customers in accordance with the laws of the State of New York without giving effect to anyconflict of law provision thereof which would result in the law of any other jurisdiction applying to theconstruction or interpretation of this Agreement. Any dispute, controversy or claim arising out of, connectedwith, related to or incidental to this Agreement, whether arising in contract, tort, equity or otherwise, shall bebrought in and resolved by a state or federal court located in New York County, New York and each partyhereby consents and submits to the jurisdiction of any such state or federal court and hereby waives any objectionsbased on forum non conveniens or any other objections to the jurisdiction and venue of any such state or federalcourt.

b) in case of other than US customers, in accordance with the laws of the Luxembourg without giving effectto any conflict of law provision thereof which would result in the law of any other jurisdiction applying to theconstruction or interpretation of this Agreement. Any dispute arising from this Agreement, or the breach,termination, validity or interpretation thereof or in relation thereto shall come under the exclusive jurisdictionof the courts of Luxembourg-city. Each party hereby consents and submits to this jurisdiction and hereby waivesany objections based on forum non conveniens or any other objections to Luxembourg jurisdiction and venue.

INDEMNIFICATIONIn addition to the indemnifications by Licensee set forth in the Master Purchase Agreement between the Licensorand Licensee, Licensee shall indemnify, defend and hold Balabit Group harmless from and against all losses(including reasonable attorneys' fees and expenses) arising out of any third party suit or claim alleging that (i)Licensee's unauthorized use of the Balabit Product hereunder has harmed such third party claimant, or (ii)Licensee's use of the Balabit Product not as intended or indicated by applicable Product Documentation is inviolation of any law, rule or regulation applicable to such use, or violates the intellectual property rights of anythird party.

AUDITA third party auditor selected by Licensor may upon reasonable notice to Licensee and during normal businesshours, but not more often than once each year, inspect Licensee's relevant records in order to confirm that usageof the Balabit Product complies with the terms and conditions of this License Agreement. Licensor shall bearthe costs of such audit. All audits shall be subject to the reasonable safety and security policies and proceduresof Licensee. The auditor shall be entitled to examine, inspect, copy and audit the usage of the Balabit Productby Licensee. If the inspection or audit reveals that the usage does not comply with the conditions of the LicenseAgreement the Licensee shall immediately:

519syslog-ng.com

NOTICES

(a) pay to Licensor the amount of any underpayment, together with interest on that amountcalculated at the rate of two per cent (2%) over the Barclay Bank base rate in New York Cityfrom time to time; and

(b) pay the costs of the audit and/or inspection where that audit or inspection reveals anunderpayment in excess of five per cent (5%).

In the event Licensee does not permit the auditor selected by Licensor to inspect, or examine the usage ofBalabit Product, Licensor shall have the right to terminate the License Agreement with immediate effect uponnotice to Licensee. Upon such termination, Licensee shall return the Balabit Product to Licensor at its own costand expense and shall remain liable for any unlawful usage and the early termination of this Agreement.

HEADINGSHeadings are for convenience only and shall be ignored in interpreting this License Agreement.

ENTIRE AGREEMENTThis License Agreement together with the Product Documentation, the Product Usage Terms, the Certificateof Authenticity and the documents referred to therein constitutes the entire agreement between the parties withregard to the subject matter hereof and supersedes all prior and contemporaneous understandings and agreements,both written and oral, with respect thereto.

Licensee hereby accepts the terms and conditions of the above End User License Agreement:

SUBSCRIPTION BASED END USER License Agreement for Balabit Product

(“SB EULA” or “SB License Agreement”)

SUBJECT OF THE SB LICENSE AGREEMENTThis SB License Agreement is entered into by and between Licensor (as defined below) and you as an end-user(hereinafter Licensee) and sets out the terms and conditions under which Licensee and/or Licensee’s AuthorizedSubsidiaries may use the Balabit Product (as defined below) under this SB License Agreement.

1. DEFINITIONSIn this SB EULA, the following words shall have the following meanings:

DescriptionNameAny third party software that is a not a Balabit Productcontained in the install package of the Balabit Product.

Annexed Software

The companies which are affiliates, a subsidiary or aparent company of the Licensor.

Balabit Group

Any software (other than the Annexed Software),hardware, virtual hardware or service licensed, sold,

Balabit Product

or provided by Licensor including any installation,education, support and warranty services, or any

520syslog-ng.com

HEADINGS

DescriptionNameproduct covered by one or more copyrights owned bya company of the Balabit Group.

The present Balabit Product Subscription Based SBLicense Agreement.

SB License Agreement

As indicated on the invoice for the Balabit Product,Balabit-Europe Kft., a limited liability company,

Licensor

incorporated and registered with the BudapestMetropolitan Court as Court of Registration undernumber Cg.01-09-186546 whose registered office is atH-1117 Budapest, Aliz u. 2., or Balabit IT SecurityDeutschland GmbH, a limited liability company,incorporated and registered with the AmtsgerichtMünchen under number HRB 167365, whose registeredoffice is at Stefan-George-Ring 29, D-81929 München,or Balabit Corp., a New York corporation, havingoffices at 40 Wall Street, New York, NY 10005.

Any documentation referring to the Balabit Product orany module thereof, including the administration guide,

Product Documentation

the product description, the installation guide and userguides and manuals.

The document signed by Licensor which contains a)identification data of the Licensee; b) the name of the

Certificate of Authenticity

Balabit Product and the designation of licensed modulesthereof; c) the Subscription Fees and payment terms)d) an explicit warning that the validity of the certificateis subject to the acceptance by the Licensee of the termsand conditions of this SB EULA; and e) informationwith regards to the extension of subscription etc.

Sets forth the conditions (the usage environment andlimitations) under which the Balabit Product may beused by the Licensee.

Product Usage Terms

A period of twelve (12) or thirty six (36) months interms of which Subscription Fees are duly paid by theLicensee.

Subscription Period

The whole Subscription Period.Warranty PeriodTable B.2. Words and expressions

2. LICENSE GRANTS AND RESTRICTIONSA. Subject to payment of the Subscription Fee and the terms and conditions of this SB License Agreement, theapplicable Certificate of Authenticity and the Product Usage Terms, Licensor hereby grants to Licensee, alimited, personal, non-exclusive and non-transferable license to use Balabit Product (“License”) for its owninternal business purposes during the Subscription Period. This License does not convey any license or right,express or implied, to manufacture, duplicate or otherwise copy or reproduce the Balabit Product or any part

521syslog-ng.com

2. LICENSE GRANTS AND RESTRICTIONS

thereof. This License is transferable only with the prior written approval of Licensor, which may be withheldin Licensor’s sole discretion.

B. Licensee shall use the Balabit Product in accordance with the conditions set by the Product Usage Termsand the Certificate of Authenticity, especially in the configuration and subject to the quantities specified inthese documents.

C. All modules of the Balabit software will be delivered to Licensee. However, Licensee shall not be entitledto use any module which is not specified in the applicable Certificate of Authenticity. Access rights to modulesand IP connections are controlled by an “electronic key” accompanying the Balabit Product.

D. Licensee shall be entitled to make one back-up copy of the Balabit software that is licensed to it.

E. Licensee shall make the Balabit Product available solely to its own employees and those of the AuthorizedSubsidiaries that are listed in the applicable Certificate of Authenticity or in the related agreement between theLicensor and the Licensee (e.g. Master Purchase Agreement) and shall take special care to protect the BalabitProduct from any unauthorized access.

F. Licensee shall, in five (5) working days properly answer any queries of Licensor regarding the actual usageconditions of the Balabit Product that may differ or allegedly differ from the License conditions set forth in theProduct Usage Terms.

G. Licensee shall install the code permitting the usage of the Balabit Product strictly in accordance and to theprovisions defined for it by Licensor. Licensee shall not modify or cancel the Balabit Product functions thereofthat inspect the usage of the software. Configuration settings of the Balabit Product in accordance with thepossibilities offered by the system shall not be construed as modification of the software.

H. Licensee shall not copy, distribute, market, sell, lease, sublicense, assign or otherwise transfer the BalabitProduct to any third party, or use the Balabit Product in a manner that (i) infringes the intellectual propertyrights or otherwise violates the rights of any third party, or (ii) violates applicable law, (iii) provides for orallows timesharing, rental or use of the Balabit Product in a service bureau or as a provider of services utilizingthe Balabit Product, or (iv) allow a competitor of Balabit to use or have access to the Balabit Product. Licenseeshall not remove or modify any program markings or any notice of Balabit’s or proprietary rights.

I. Licensee shall not (i) modify, translate, decompile or reverse engineer the Balabit Product, (ii) attempt tocreate the source code from the executable or object code of the Balabit Product by reverse engineering ordisassembling or otherwise adopt, manipulate the executable or object code of the Balabit Product, (iii) createa derivative work based upon the Balabit Product or the Product Documentation or permit a third party to dothe same, or (iv) modify, tamper with, reverse engineer, reverse compile or disassemble the electronic key forthe Balabit Product.

(v) Notwithstanding the foregoing, Licensee shall be entitled to analyze the structure of the Balabit Product(decompilation or reverse- engineering) only if necessary to coordinate operation of the Balabit Product withsoftware developed by a third party, and only if Licensor does not provide such information within 60 (sixty)days from the receipt of such a request. Such analysis of the structure of the Balabit Product is strictly limitedto those parts of the Balabit Product which are necessary for concurrent operation with the third party softwareand is subject to either a) Licensor’s prior written consent, or b) the failure of Licensor to provide the aforesaidinformation within the aforesaid 60 (sixty) day period.

522syslog-ng.com

2. LICENSE GRANTS AND RESTRICTIONS

Any information obtained by Licensee as a result of applying subsection (v) (a) cannot be used for any purposesother than concurrent operation of the third party software with the Balabit Product, (b) shall not be disclosedto third parties unless it is necessary to disclose it to the owner of the third party software for concurrent operationwith the Balabit Product; (c) shall not be used for the development, production or distribution of software whichis the same as or similar to the Balabit Product in features or in functionality, or (d) for any other act or purposethat violates Licensor’s copyrights in the Balabit Product.

(vi) Notwithstanding the foregoing, Licensee shall be entitled to analyze the structure of those components ofthe BalaBit Product (decompilation or reverse-engineering) that fall under GNU Lesser General Public License3.0 (https://www.gnu.org/licenses/lgpl-3.0.en.html) according to the Product Documentation. Any activity underthis sub-clause shall be made in full compliance with the license terms referred to herein.

J. Licensee shall comply with all terms and conditions made applicable to all Annexed Software contained inthe same install package with the Balabit Product by the owner of the Annexed Software. Licensor does notgrant any license rights to any Annexed Software by including it with a Balabit Product in the same installpackage. Such rights must be acquired by Licensee directly from the owner of the Annexed Software.

K. Any usage of the Balabit Product exceeding the limits and restrictions defined in the Certificate of Authenticityshall be a material breach of the SB License Agreement and Licensee shall be fully liable to Licensor for suchbreach, including for monetary damages and/or termination of this SB License Agreement and the MasterPurchase Agreement and any Order made thereunder.

L. During the Subscription Period Licensee shall have the right to obtain and use content updates of the BalabitProduct (maintenance of the software) and shall be provided with support services in accordance with Balabit’sthen current Support General Terms and Conditions (hereinafter Support GTC).

M. Licensor expressly reserves all rights not expressly granted herein.

3. CONFIDENTIALITYA. “Confidential Information” means any business, marketing, technical, scientific or other information disclosedby the Balabit Group which, at the time of disclosure is designated as confidential (or like designation), isdisclosed in circumstances of confidence, or would be understood by the parties (or their Affiliates), exercisingreasonable business judgment, to be confidential.

B. License acknowledges that the Balabit Product, the Product Documentation and related materials are thetrade secrets and Confidential Information of the Balabit Group. Licensee agrees to keep confidential allconfidential information of the Balabit Group including but not limited to the Balabit Product, the ProductDocumentation and related materials. Licensee agrees to use all confidential information of the Balabit Groupincluding but not limited to the Balabit Product, the Product Documentation and related materials only asexpressly permitted by this Agreement.

C. Licensee shall retain the Confidential Information of the Balabit Group in confidence and shall use anddisclose it solely for the purpose of, and in accordance with, this SB License Agreement. Licensee shall onlydisclose Confidential Information of the Balabit Group to those of its employees with a need to know suchConfidential Information. Licensee shall use the same degree of care as it uses to protect its own confidentialinformation of a similar nature, but no less than reasonable care, to prevent the unauthorized use or disclosureof the Balabit Group’s Confidential Information.

523syslog-ng.com

3. CONFIDENTIALITY

4. INTELLECTUAL PROPERTY RIGHTSA. All right, title, and interest in and to the Balabit Product, including all patents, trademarks, trade names,inventions, know-how, trade secrets and all other intellectual property rights relating to the design, manufacture,operation or service of the Balabit Product are owned by one or more of the companies of the Balabit Group.No right or interest in any of such intellectual property rights is transferred to Licensee by this License otherthan the right and license to use the Balabit Product modules licensed hereunder in accordance with this SBLicense Agreement and the Product Usage Terms.

B. Licensee will advise its Authorized Subsidiaries, if any, of and assure compliance with the restrictionscontained in the SB License Agreement, including those relating to the Confidential Information and proprietaryproperty of the Balabit Group. Licensee shall implement adequate security measures to protect such trade secretsand confidential information.

C. The use by Licensee of any of the intellectual property rights in the Balabit Product is authorized only forthe purposes set forth herein, and upon termination of this SB License Agreement, such authorization shallcease and Licensee shall immediately cease the use of the Balabit Product.

5. WARRANTIESA. Licensor warrants that during the Subscription Period, the Balabit provided hardware upon which the BalabitProduct is installed provided to Licensee by Licensor (“Appliance”) will be free of defects of material orworkmanship under normal use. Licensor will replace any defective Appliance returned to it, accompanied bya dated proof of purchase that is within the Subscription Period, at no charge to Licensee. Upon receipt of theallegedly defective Appliance, Licensor will at its option, deliver a replacement Appliance or Licensor’s currentequivalent Appliance to Licensee at no additional cost. Licensor will bear all delivery charges to Licensee forthe replacement Appliance.

B. In the event Licensee uses the Balabit Product in conjunction with any third party software, Licensor shallnot be liable for any errors in the operation of the Balabit Product that is due to the third party software.

C. Licensor warrants that during the Subscription Period, the Balabit Product software without unauthorizedmodification shall perform in substantial compliance with the Product Documentation accompanying the BalabitProduct, when it is used in normal use (i) on that hardware or virtual appliance for which it was installed and(ii) in compliance with the provisions of the Product Documentation and the Product Usage Terms. If the BalabitProduct fails to so operate, Licensee shall promptly notify Licensor (the date of the notification sent to Licensorshall be deemed to be the date of the failure) and Licensee shall do its best to mitigate the consequences of thatfailure until Licensor can address the failure to operate in accordance with the aforesaid documentation. If thefailure is reported by Licensee to Licensor, Licensor’s sole obligation and liability for breach of this warrantyis, at Licensor’s sole option, either: (i) to correct such failure, or (ii) to replace the defective Balabit Product.

D. Where the Balabit Product has not been acquired directly from Licensor, Licensee must contact the entitythat has sold the license to the Balabit Product to Licensee in order to exercise its rights under this warranty.Licensor will not provide to Licensee any after-sale warranty if Licensor has not sold the license to the BalabitProduct directly to Licensee.

E. EXCEPT AS SET FORTH IN THIS SB LICENSE AGREEMENT, LICENSOR MAKES NO WARRANTIESOF ANY KIND WITH RESPECT TO THE BALABIT PRODUCT. TO THE MAXIMUM EXTENTPERMITTED BY APPLICABLE LAW, LICENSOR DISCLAIMS ANY OTHER WARRANTIES, INCLUDINGBUT NOT LIMITED TO ANY IMPLIED WARRANTIES OF SATISFACTORY QUALITY,

524syslog-ng.com

4. INTELLECTUAL PROPERTY RIGHTS

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT OFINTELLECTUAL PROPERTY RIGHTS.

6. SUBSCRIPTION FEEA. The Certificate of Authenticity and the Product Usage Term contain the details of the purchased Licenseand usage limitations. This information serves as the calculation base of the Subscription Fee. Licenseeacknowledges that payment of the Subscription Fee is a condition of lawful usage.

B. Subscription Fees do not include any installation or post sale charges, taxes, duties, etc., all of which are forthe account of Licensee. Applicable taxes shall be added to all invoices to Licensee for Subscription Fees.

C. The license rights to the Balabit Product are transferred to the Licensee only when Licensee pays theSubscription Fees to Licensor. In case of non-payment Licensor has right to terminate, or rescind the SB LicenseAgreement with immediate effect and Licensee shall promptly cease all use of the Balabit Product and returnit to Licensor at its own cost and expense and shall be liable for its unlawful usage and the early termination.

7. LIMITATION OF LIABILITYSOME STATES AND COUNTRIES, INCLUDING MEMBER COUNTRIES OF THE EUROPEAN UNION,DO NOT ALLOW THE LIMITATION OR EXCLUSION OF LIABILITY FOR INCIDENTAL ORCONSEQUENTIAL DAMAGES AND, THEREFORE, THE FOLLOWING LIMITATION OR EXCLUSIONMAY NOT APPLY TO THIS SB LICENSE AGREEMENT IN THOSE STATES AND COUNTRIES.

TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW AND REGARDLESS OF WHETHERANY REMEDY SET OUT IN THIS SB LICENSE AGREEMENT FAILS OF ITS ESSENTIAL PURPOSE,IN NO EVENT SHALL LICENSOR BE LIABLE TO LICENSEE FOR ANY SPECIAL, EXEMPLARY,CONSEQUENTIAL, INDIRECT, PUNITIVE, OR SIMILAR DAMAGES OR LOST PROFITS OR LOSTDATA ARISING OUT OF THE USE OR INABILITY TO USE THE BALABIT PRODUCT EVEN IFLICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

IN NO CASE SHALL LICENSOR’S TOTAL LIABILITY UNDER THIS SB LICENSE AGREEMENTEXCEED THE FEES RECEIVED BY LICENSOR FOR THE BALABIT PRODUCT LICENSED UNDERTHIS SB LICENSE AGREEMENT.

NOTWITHSTANDING ANYTHING SET FORTH IN THIS AGREEMENT TO THE CONTRARY, IN NOEVENT SHALL LICENSOR BE LIABLE FOR ANY DAMAGES CAUSED BY THE USAGE OF THEBALABIT PRODUCT WHICH IS NOT IN ACCORDANCE WITH THE PRODUCT DOCUMENTATIONAND THE PRODUCT USAGE TERMS.

8. TERM AND TERMINATIONA. This SB License Agreement shall come into effect on the day when the Licensee declares acceptance of itsterms and conditions, provided that the Subscription Fee has been fully paid. Either the signing a copy of theSB License Agreement by the Licensee’s duly authorized representative, or Licensee “clicking” on the"Confirmation" button (“I have read and agree ...”) with regard to this SB License Agreement at the beginningof the installation process of the Balabit Product shall be deemed to be acceptance by the Licensee to the termsand conditions of the SB License Agreement. The Buyer represents and warrants that the members of its ITstaff working on the installation of the Products (either with or without the Supplier’s installation personnel)are authorized to bind the Buyer to this SB License Agreement by signing a copy of the SB License Agreementor “clicking” on the Confirmation button.

525syslog-ng.com

6. SUBSCRIPTION FEE

B. Licensee may terminate the SB License Agreement at any time by written notice sent to Licensor and bysimultaneously destroying all copies of the Balabit Product licensed under this SB License Agreement andcertifying to Licensor that it has done so.

C. Licensor may terminate this SB License Agreement with immediate effect by written notice to Licensee, ifLicensee is in material or persistent breach of the SB License Agreement and either that breach is incapable ofremedy or Licensee shall have failed to remedy that breach within 30 (thirty) days after receiving written noticerequiring it to remedy that breach. In such a case, Licensee must immediately destroy all copies of the BalabitProduct licensed under this SB License Agreement, the Product Documentation and all other materials containingthe Confidential Information of Licensor and certify to Licensor that it has done so.

D. The provisions of this Agreement relating to confidentiality, applicable law and jurisdiction, notices,indemnification, disclaimers and limits of liability shall survive the expiration or termination of this Agreementfor any reason.

9. AMENDMENTSExcept as expressly provided in this SB License Agreement, no amendment or variation of this SB LicenseAgreement shall be effective unless in writing and signed by a duly authorized representative of both partieshereto.

10. WAIVERThe failure of a party to exercise or enforce any right under this SB License Agreement shall not be deemed tobe a waiver of that right nor operate to bar the exercise or enforcement of such right or any other right at anytime or times thereafter.

11. SEVERABILITYIf any part of this SB License Agreement becomes invalid, illegal or unenforceable, the parties shall in such anevent negotiate in good faith in order to agree on the terms of a mutually satisfactory provision to be substitutedfor the invalid, illegal or unenforceable provision which as nearly as possible validly gives effect to theirintentions as expressed in this SB License Agreement.

12. NOTICESAny notice required to be given pursuant to this SB License Agreement shall be in writing and shall be givenby delivering the notice by hand, or by sending the same by prepaid first class post (airmail if to an addressoutside the country of posting) or by recognized courier service such as Federal Express to the address of therelevant party. Any notice given according to the above procedure shall be deemed to have been given at thetime of delivery (if delivered by hand) and when received (if sent by post or courier service).

13. APPLICABLE LAW AND JURISDICTIONThis Agreement shall be construed, interpreted and the rights of the parties determined

a) in case of US Licensees (customers) in accordance with the laws of the State of New York without givingeffect to any conflict of law provision thereof which would result in the law of any other jurisdiction applyingto the construction or interpretation of this Agreement. Any dispute, controversy or claim arising out of,connected with, related to or incidental to this Agreement, whether arising in contract, tort, equity or otherwise,shall be brought in and resolved by a state or federal court located in New York County, New York and eachparty hereby consents and submits to the jurisdiction of any such state or federal court and hereby waives any

526syslog-ng.com

9. AMENDMENTS

objections based on forum non conveniens or any other objections to the jurisdiction and venue of any suchstate or federal court.

b) in case of other than US Licensees (customers), in accordance with the laws of the Luxembourg withoutgiving effect to any conflict of law provision thereof which would result in the law of any other jurisdictionapplying to the construction or interpretation of this Agreement. Any dispute arising from this Agreement, orthe breach, termination, validity or interpretation thereof or in relation thereto shall come under the exclusivejurisdiction of the courts of Luxembourg-city. Each party hereby consents and submits to this jurisdiction andhereby waives any objections based on forum non conveniens or any other objections to Luxembourg jurisdictionand venue.

14. INDEMNIFICATIONIn addition to the indemnifications by Licensee set forth in the Master Purchase Agreement between the Licensorand Licensee, Licensee shall indemnify, defend and hold Balabit Group harmless from and against all losses(including reasonable attorneys’ fees and expenses) arising out of any third party suit or claim alleging that (i)Licensee’s unauthorized use of the Balabit Product hereunder has harmed such third party claimant, or (ii)Licensee’s use of the Balabit Product not as intended or indicated by applicable Product Documentation is inviolation of any law, rule or regulation applicable to such use, or violates the intellectual property rights of anythird party.

15. AUDITA third party auditor selected by Licensor may upon reasonable notice to Licensee and during normal businesshours, but not more often than once each year, inspect Licensee’s relevant records in order to confirm that usageof the Balabit Product complies with the terms and conditions of this SB License Agreement. Licensor shallbear the costs of such audit. All audits shall be subject to the reasonable safety and security policies andprocedures of Licensee. The auditor shall be entitled to examine, inspect, copy and audit the usage of the BalabitProduct by Licensee. If the inspection or audit reveals that the usage does not comply with the conditions ofthe SB License Agreement the Licensee shall immediately:

(a) pay to Licensor the amount of any underpayment, together with interest on that amount calculated at therate of two per cent (2%) over the Barclay Bank base rate in New York City from time to time; and

(b) pay the costs of the audit and/or inspection where that audit or inspection reveals an underpayment in excessof five per cent (5%).

In the event Licensee does not permit the auditor selected by Licensor to inspect, or examine the usage ofBalabit Product, Licensor shall have the right to terminate the SB License Agreement with immediate effectupon notice to Licensee. Upon such termination, Licensee shall return the Balabit Product to Licensor at itsown cost and expense and shall remain liable for any unlawful usage and the early termination of this Agreement.

16. HEADINGSHeadings are for convenience only and shall be ignored in interpreting this SB License Agreement.

17. ENTIRE AGREEMENTThis SB License Agreement together with the Product Documentation, the Product Usage Terms, the Certificateof Authenticity and the documents referred to therein constitutes the entire agreement between the parties with

527syslog-ng.com

14. INDEMNIFICATION

regard to the subject matter hereof and supersedes all prior and contemporaneous understandings and agreements,both written and oral, with respect thereto.

Licensee hereby accepts the terms and conditions of the above SB License Agreement.

528syslog-ng.com

17. ENTIRE AGREEMENT

Appendix C. Open source licenses

This appendix includes the open source licenses and attributions applicable to syslog-ng Premium Edition.

C.1. GNU General Public License v2

Version 2, June 1991Copyright © 1989, 1991 Free Software Foundation, Inc.

Free Software Foundation, Inc.51 Franklin Street, Fifth Floor,Boston, MA

02110-1301USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is notallowed.

Version 2, June 1991

C.1.1. Preamble

The licenses for most software are designed to take away your freedom to share and change it. By contrast, theGNU General Public License is intended to guarantee your freedom to share and change free software - to makesure the software is free for all its users. This General Public License applies to most of the Free SoftwareFoundation's software and to any other program whose authors commit to using it. (Some other Free SoftwareFoundation software is covered by the GNU Library General Public License instead.) You can apply it to yourprograms, too.

When we speak of free software, we are referring to freedom, not price. Our General Public Licenses aredesigned to make sure that you have the freedom to distribute copies of free software (and charge for this serviceif you wish), that you receive source code or can get it if you want it, that you can change the software or usepieces of it in new free programs; and that you know you can do these things.

To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask youto surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies ofthe software, or if you modify it.

For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipientsall the rights that you have. You must make sure that they, too, receive or can get the source code. And youmust show them these terms so they know their rights.

We protect your rights with two steps:

1. copyright the software, and

2. offer you this license which gives you legal permission to copy, distribute and/or modify the software.

529syslog-ng.com

GNU General Public License v2

Also, for each author's protection and ours, we want to make certain that everyone understands that there is nowarranty for this free software. If the software is modified by someone else and passed on, we want its recipientsto know that what they have is not the original, so that any problems introduced by others will not reflect onthe original authors' reputations.

Finally, any free program is threatened constantly by software patents. We wish to avoid the danger thatredistributors of a free program will individually obtain patent licenses, in effect making the program proprietary.To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensedat all.

The precise terms and conditions for copying, distribution and modification follow.

C.1.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.1.2.1. Section 0

This License applies to any program or other work which contains a notice placed by the copyright holdersaying it may be distributed under the terms of this General Public License. The“Program”, below, refers toany such program or work, and a“work based on the Program”means either the Program or any derivative workunder copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or withmodifications and/or translated into another language. (Hereinafter, translation is included without limitationin the term“modification”.) Each licensee is addressed as“you”.

Activities other than copying, distribution and modification are not covered by this License; they are outsideits scope. The act of running the Program is not restricted, and the output from the Program is covered only ifits contents constitute a work based on the Program (independent of having been made by running the Program).Whether that is true depends on what the Program does.

C.1.2.2. Section 1

You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium,provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice anddisclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty;and give any other recipients of the Program a copy of this License along with the Program.

You may charge a fee for the physical act of transferring a copy, and you may at your option offer warrantyprotection in exchange for a fee.

C.1.2.3. Section 2

You may modify your copy or copies of the Program or any portion of it, thus forming a work based on theProgram, and copy and distribute such modifications or work under the terms of Section 1 above, provided thatyou also meet all of these conditions:

a. You must cause the modified files to carry prominent notices stating that you changed the files andthe date of any change.

b. You must cause any work that you distribute or publish, that in whole or in part contains or is derivedfrom the Program or any part thereof, to be licensed as a whole at no charge to all third parties underthe terms of this License.

530syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

c. If the modified program normally reads commands interactively when run, you must cause it, whenstarted running for such interactive use in the most ordinary way, to print or display an announcementincluding an appropriate copyright notice and a notice that there is no warranty (or else, saying thatyou provide a warranty) and that users may redistribute the program under these conditions, andtelling the user how to view a copy of this License. (Exception: If the Program itself is interactivebut does not normally print such an announcement, your work based on the Program is not requiredto print an announcement.)

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derivedfrom the Program, and can be reasonably considered independent and separate works in themselves, then thisLicense, and its terms, do not apply to those sections when you distribute them as separate works. But whenyou distribute the same sections as part of a whole which is a work based on the Program, the distribution ofthe whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole,and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you;rather, the intent is to exercise the right to control the distribution of derivative or collective works based onthe Program.

In addition, mere aggregation of another work not based on the Program with the Program (or with a workbased on the Program) on a volume of a storage or distribution medium does not bring the other work underthe scope of this License.

C.1.2.4. Section 3

You may copy and distribute the Program (or a work based on it, under Section 2 in object code or executableform under the terms of Sections 1 and 2 above provided that you also do one of the following:

a. Accompany it with the complete corresponding machine-readable source code, which must bedistributed under the terms of Sections 1 and 2 above on a medium customarily used for softwareinterchange; or,

b. Accompany it with a written offer, valid for at least three years, to give any third party, for a chargeno more than your cost of physically performing source distribution, a complete machine-readablecopy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 aboveon a medium customarily used for software interchange; or,

c. Accompany it with the information you received as to the offer to distribute corresponding sourcecode. (This alternative is allowed only for noncommercial distribution and only if you received theprogram in object code or executable form with such an offer, in accord with Subsection b above.)

The source code for a work means the preferred form of the work for making modifications to it. For anexecutable work, complete source code means all the source code for all modules it contains, plus any associatedinterface definition files, plus the scripts used to control compilation and installation of the executable. However,as a special exception, the source code distributed need not include anything that is normally distributed (ineither source or binary form) with the major components (compiler, kernel, and so on) of the operating systemon which the executable runs, unless that component itself accompanies the executable.

If distribution of executable or object code is made by offering access to copy from a designated place, thenoffering equivalent access to copy the source code from the same place counts as distribution of the sourcecode, even though third parties are not compelled to copy the source along with the object code.

531syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.1.2.5. Section 4

You may not copy, modify, sublicense, or distribute the Program except as expressly provided under thisLicense. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and willautomatically terminate your rights under this License. However, parties who have received copies, or rights,from you under this License will not have their licenses terminated so long as such parties remain in fullcompliance.

C.1.2.6. Section 5

You are not required to accept this License, since you have not signed it. However, nothing else grants youpermission to modify or distribute the Program or its derivative works. These actions are prohibited by law ifyou do not accept this License. Therefore, by modifying or distributing the Program (or any work based on theProgram), you indicate your acceptance of this License to do so, and all its terms and conditions for copying,distributing or modifying the Program or works based on it.

C.1.2.7. Section 6

Each time you redistribute the Program (or any work based on the Program), the recipient automatically receivesa license from the original licensor to copy, distribute or modify the Program subject to these terms andconditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein.You are not responsible for enforcing compliance by third parties to this License.

C.1.2.8. Section 7

If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limitedto patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradictthe conditions of this License, they do not excuse you from the conditions of this License. If you cannot distributeso as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then asa consequence you may not distribute the Program at all. For example, if a patent license would not permitroyalty-free redistribution of the Program by all those who receive copies directly or indirectly through you,then the only way you could satisfy both it and this License would be to refrain entirely from distribution ofthe Program.

If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance ofthe section is intended to apply and the section as a whole is intended to apply in other circumstances.

It is not the purpose of this section to induce you to infringe any patents or other property right claims or tocontest validity of any such claims; this section has the sole purpose of protecting the integrity of the freesoftware distribution system, which is implemented by public license practices. Many people have madegenerous contributions to the wide range of software distributed through that system in reliance on consistentapplication of that system; it is up to the author/donor to decide if he or she is willing to distribute softwarethrough any other system and a licensee cannot impose that choice.

This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

532syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.1.2.9. Section 8

If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrightedinterfaces, the original copyright holder who places the Program under this License may add an explicitgeographical distribution limitation excluding those countries, so that distribution is permitted only in or amongcountries not thus excluded. In such case, this License incorporates the limitation as if written in the body ofthis License.

C.1.2.10. Section 9

The Free Software Foundation may publish revised and/or new versions of the General Public License fromtime to time. Such new versions will be similar in spirit to the present version, but may differ in detail to addressnew problems or concerns.

Each version is given a distinguishing version number. If the Program specifies a version number of this Licensewhich applies to it and“any later version”, you have the option of following the terms and conditions either ofthat version or of any later version published by the Free Software Foundation. If the Program does not specifya version number of this License, you may choose any version ever published by the Free Software Foundation.

C.1.2.11. Section 10

If you wish to incorporate parts of the Program into other free programs whose distribution conditions aredifferent, write to the author to ask for permission. For software which is copyrighted by the Free SoftwareFoundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision willbe guided by the two goals of preserving the free status of all derivatives of our free software and of promotingthe sharing and reuse of software generally.

C.1.2.12. NO WARRANTY Section 11

BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THEPROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISESTATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THEPROGRAM“AS IS”WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY ANDFITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY ANDPERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE,YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

C.1.2.13. Section 12

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANYCOPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THEPROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANYGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USEOR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA ORDATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIESOR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCHHOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

533syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

END OF TERMS AND CONDITIONS

C.1.3. How to Apply These Terms to Your New Programs

If you develop a new program, and you want it to be of the greatest possible use to the public, the best way toachieve this is to make it free software which everyone can redistribute and change under these terms.

To do so, attach the following notices to the program. It is safest to attach them to the start of each source fileto most effectively convey the exclusion of warranty; and each file should have at least the“copyright”line anda pointer to where the full notice is found.

<one line to give the program's name and a brief idea of what it does.> Copyright (C) <year> <name of author>

This program is free software; you can redistribute it and/or modify it under the terms of the GNU GeneralPublic License as published by the Free Software Foundation; either version 2 of the License, or (at your option)any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without eventhe implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNUGeneral Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write tothe Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Also add information on how to contact you by electronic and paper mail.

If the program is interactive, make it output a short notice like this when it starts in an interactive mode:

Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NOWARRANTY; for details type“show w”. This is free software, and you are welcome to redistribute it undercertain conditions; type“show c”for details.

The hypothetical commands“show w”and“show c”should show the appropriate parts of the General PublicLicense. Of course, the commands you use may be called something other than“show w”and“show c”; theycould even be mouse-clicks or menu items--whatever suits your program.

You should also get your employer (if you work as a programmer) or your school, if any, to sign a“copyrightdisclaimer”for the program, if necessary. Here is a sample; alter the names:

Yoyodyne, Inc., hereby disclaims all copyright interest in the program“Gnomovision”(which makes passes atcompilers) written by James Hacker.

<signature of Ty Coon>, 1 April 1989 Ty Coon, President of Vice

This General Public License does not permit incorporating your program into proprietary programs. If yourprogram is a subroutine library, you may consider it more useful to permit linking proprietary applications withthe library. If this is what you want to do, use the GNU Library General Public License instead of this License.

534syslog-ng.com

How to Apply These Terms to Your New Programs

C.2. GNU Lesser General Public License v2.1

This is the first released version of the Lesser GPL. It also counts as the successor of the GNU Library PublicLicense, version 2, hence the version number 2.1.Copyright © 1991, 1999 Free Software Foundation, Inc.

Free Software Foundation, Inc.51 Franklin Street, Fifth Floor,Boston, MA 02110-1301USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is notallowed.

Version 2.1, February 1999

C.2.1. Preamble

The licenses for most software are designed to take away your freedom to share and change it. By contrast, theGNU General Public Licenses are intended to guarantee your freedom to share and change free software--tomake sure the software is free for all its users.

This license, the Lesser General Public License, applies to some specially designated software packages--typicallylibraries--of the Free Software Foundation and other authors who decide to use it. You can use it too, but wesuggest you first think carefully about whether this license or the ordinary General Public License is the betterstrategy to use in any particular case, based on the explanations below.

When we speak of free software, we are referring to freedom of use, not price. Our General Public Licensesare designed to make sure that you have the freedom to distribute copies of free software (and charge for thisservice if you wish); that you receive source code or can get it if you want it; that you can change the softwareand use pieces of it in new free programs; and that you are informed that you can do these things.

To protect your rights, we need to make restrictions that forbid distributors to deny you these rights or to askyou to surrender these rights. These restrictions translate to certain responsibilities for you if you distributecopies of the library or if you modify it.

For example, if you distribute copies of the library, whether gratis or for a fee, you must give the recipients allthe rights that we gave you. You must make sure that they, too, receive or can get the source code. If you linkother code with the library, you must provide complete object files to the recipients, so that they can relinkthem with the library after making changes to the library and recompiling it. And you must show them theseterms so they know their rights.

We protect your rights with a two-step method:

1. we copyright the library, and

2. we offer you this license, which gives you legal permission to copy, distribute and/or modify thelibrary.

535syslog-ng.com

GNU Lesser General Public License v2.1

To protect each distributor, we want to make it very clear that there is no warranty for the free library. Also, ifthe library is modified by someone else and passed on, the recipients should know that what they have is notthe original version, so that the original author's reputation will not be affected by problems that might beintroduced by others.

Finally, software patents pose a constant threat to the existence of any free program. We wish to make sure thata company cannot effectively restrict the users of a free program by obtaining a restrictive license from a patentholder. Therefore, we insist that any patent license obtained for a version of the library must be consistent withthe full freedom of use specified in this license.

Most GNU software, including some libraries, is covered by the ordinary GNU General Public License. Thislicense, the GNU Lesser General Public License, applies to certain designated libraries, and is quite differentfrom the ordinary General Public License. We use this license for certain libraries in order to permit linkingthose libraries into non-free programs.

When a program is linked with a library, whether statically or using a shared library, the combination of thetwo is legally speaking a combined work, a derivative of the original library. The ordinary General PublicLicense therefore permits such linking only if the entire combination fits its criteria of freedom. The LesserGeneral Public License permits more lax criteria for linking other code with the library.

We call this license the Lesser General Public License because it does Less to protect the user's freedom thanthe ordinary General Public License. It also provides other free software developers Less of an advantage overcompeting non-free programs. These disadvantages are the reason we use the ordinary General Public Licensefor many libraries. However, the Lesser license provides advantages in certain special circumstances.

For example, on rare occasions, there may be a special need to encourage the widest possible use of a certainlibrary, so that it becomes a de-facto standard. To achieve this, non-free programs must be allowed to use thelibrary. A more frequent case is that a free library does the same job as widely used non-free libraries. In thiscase, there is little to gain by limiting the free library to free software only, so we use the Lesser General PublicLicense.

In other cases, permission to use a particular library in non-free programs enables a greater number of peopleto use a large body of free software. For example, permission to use the GNU C Library in non-free programsenables many more people to use the whole GNU operating system, as well as its variant, the GNU/Linuxoperating system.

Although the Lesser General Public License is Less protective of the users' freedom, it does ensure that theuser of a program that is linked with the Library has the freedom and the wherewithal to run that program usinga modified version of the Library.

The precise terms and conditions for copying, distribution and modification follow. Pay close attention to thedifference between a “work based on the library” and a “work that uses the library”. The former contains codederived from the library, whereas the latter must be combined with the library in order to run.

536syslog-ng.com

Preamble

C.2.2. TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.2.2.1. Section 0

This License Agreement applies to any software library or other program which contains a notice placed bythe copyright holder or other authorized party saying it may be distributed under the terms of this Lesser GeneralPublic License (also called “this License”). Each licensee is addressed as “you”.

A “library” means a collection of software functions and/or data prepared so as to be conveniently linked withapplication programs (which use some of those functions and data) to form executables.

The “Library”, below, refers to any such software library or work which has been distributed under these terms.A “work based on the Library” means either the Library or any derivative work under copyright law: that is tosay, a work containing the Library or a portion of it, either verbatim or with modifications and/or translatedstraightforwardly into another language. (Hereinafter, translation is included without limitation in the term“modification”.)

“Source code” for a work means the preferred form of the work for making modifications to it. For a library,complete source code means all the source code for all modules it contains, plus any associated interfacedefinition files, plus the scripts used to control compilation and installation of the library.

Activities other than copying, distribution and modification are not covered by this License; they are outsideits scope. The act of running a program using the Library is not restricted, and output from such a program iscovered only if its contents constitute a work based on the Library (independent of the use of the Library in atool for writing it). Whether that is true depends on what the Library does and what the program that uses theLibrary does.

C.2.2.2. Section 1

You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in anymedium, provided that you conspicuously and appropriately publish on each copy an appropriate copyrightnotice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of anywarranty; and distribute a copy of this License along with the Library.

You may charge a fee for the physical act of transferring a copy, and you may at your option offer warrantyprotection in exchange for a fee.

C.2.2.3. Section 2

You may modify your copy or copies of the Library or any portion of it, thus forming a work based on theLibrary, and copy and distribute such modifications or work under the terms of Section 1 above, provided thatyou also meet all of these conditions:

a. The modified work must itself be a software library.

b. You must cause the files modified to carry prominent notices stating that you changed the files andthe date of any change.

c. You must cause the whole of the work to be licensed at no charge to all third parties under the termsof this License.

537syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

d. If a facility in the modified Library refers to a function or a table of data to be supplied by anapplication program that uses the facility, other than as an argument passed when the facility isinvoked, then you must make a good faith effort to ensure that, in the event an application does notsupply such function or table, the facility still operates, and performs whatever part of its purposeremains meaningful.(For example, a function in a library to compute square roots has a purpose that is entirely well-definedindependent of the application. Therefore, Subsection 2d requires that any application-suppliedfunction or table used by this function must be optional: if the application does not supply it, thesquare root function must still compute square roots.)

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derivedfrom the Library, and can be reasonably considered independent and separate works in themselves, then thisLicense, and its terms, do not apply to those sections when you distribute them as separate works. But whenyou distribute the same sections as part of a whole which is a work based on the Library, the distribution of thewhole must be on the terms of this License, whose permissions for other licensees extend to the entire whole,and thus to each and every part regardless of who wrote it.

Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you;rather, the intent is to exercise the right to control the distribution of derivative or collective works based onthe Library.

In addition, mere aggregation of another work not based on the Library with the Library (or with a work basedon the Library) on a volume of a storage or distribution medium does not bring the other work under the scopeof this License.

C.2.2.4. Section 3

You may opt to apply the terms of the ordinary GNU General Public License instead of this License to a givencopy of the Library. To do this, you must alter all the notices that refer to this License, so that they refer to theordinary GNU General Public License, version 2, instead of to this License. (If a newer version than version 2of the ordinary GNU General Public License has appeared, then you can specify that version instead if youwish.) Do not make any other change in these notices.

Once this change is made in a given copy, it is irreversible for that copy, so the ordinary GNU General PublicLicense applies to all subsequent copies and derivative works made from that copy.

This option is useful when you wish to copy part of the code of the Library into a program that is not a library.

C.2.2.5. Section 4

You may copy and distribute the Library (or a portion or derivative of it, under Section 2) in object code orexecutable form under the terms of Sections 1 and 2 above provided that you accompany it with the completecorresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2above on a medium customarily used for software interchange.

If distribution of object code is made by offering access to copy from a designated place, then offering equivalentaccess to copy the source code from the same place satisfies the requirement to distribute the source code, eventhough third parties are not compelled to copy the source along with the object code.

538syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.2.2.6. Section 5

A program that contains no derivative of any portion of the Library, but is designed to work with the Libraryby being compiled or linked with it, is called a “work that uses the Library”. Such a work, in isolation, is nota derivative work of the Library, and therefore falls outside the scope of this License.

However, linking a “work that uses the Library” with the Library creates an executable that is a derivative ofthe Library (because it contains portions of the Library), rather than a “work that uses the library”. The executableis therefore covered by this License. Section 6 states terms for distribution of such executables.

When a “work that uses the Library” uses material from a header file that is part of the Library, the object codefor the work may be a derivative work of the Library even though the source code is not. Whether this is trueis especially significant if the work can be linked without the Library, or if the work is itself a library. Thethreshold for this to be true is not precisely defined by law.

If such an object file uses only numerical parameters, data structure layouts and accessors, and small macrosand small inline functions (ten lines or less in length), then the use of the object file is unrestricted, regardlessof whether it is legally a derivative work. (Executables containing this object code plus portions of the Librarywill still fall under Section 6.)

Otherwise, if the work is a derivative of the Library, you may distribute the object code for the work under theterms of Section 6. Any executables containing that work also fall under Section 6, whether or not they arelinked directly with the Library itself.

C.2.2.7. Section 6

As an exception to the Sections above, you may also combine or link a “work that uses the Library” with theLibrary to produce a work containing portions of the Library, and distribute that work under terms of yourchoice, provided that the terms permit modification of the work for the customer's own use and reverseengineering for debugging such modifications.

You must give prominent notice with each copy of the work that the Library is used in it and that the Libraryand its use are covered by this License. You must supply a copy of this License. If the work during executiondisplays copyright notices, you must include the copyright notice for the Library among them, as well as areference directing the user to the copy of this License. Also, you must do one of these things:

a. Accompany the work with the complete corresponding machine-readable source code for the Libraryincluding whatever changes were used in the work (which must be distributed under Sections 1 and2 above); and, if the work is an executable linked with the Library, with the completemachine-readable “work that uses the Library”, as object code and/or source code, so that the usercan modify the Library and then relink to produce a modified executable containing the modifiedLibrary. (It is understood that the user who changes the contents of definitions files in the Librarywill not necessarily be able to recompile the application to use the modified definitions.)

b. Use a suitable shared library mechanism for linking with the Library. A suitable mechanism is onethat (1) uses at run time a copy of the library already present on the user's computer system, ratherthan copying library functions into the executable, and (2) will operate properly with a modifiedversion of the library, if the user installs one, as long as the modified version is interface-compatiblewith the version that the work was made with.

539syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

c. Accompany the work with a written offer, valid for at least three years, to give the same user thematerials specified in Subsection 6a, above, for a charge no more than the cost of performing thisdistribution.

d. If distribution of the work is made by offering access to copy from a designated place, offer equivalentaccess to copy the above specified materials from the same place.

e. Verify that the user has already received a copy of these materials or that you have already sent thisuser a copy.

For an executable, the required form of the “work that uses the Library” must include any data and utilityprograms needed for reproducing the executable from it. However, as a special exception, the materials to bedistributed need not include anything that is normally distributed (in either source or binary form) with themajor components (compiler, kernel, and so on) of the operating system on which the executable runs, unlessthat component itself accompanies the executable.

It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do notnormally accompany the operating system. Such a contradiction means you cannot use both them and theLibrary together in an executable that you distribute.

C.2.2.8. Section 7

You may place library facilities that are a work based on the Library side-by-side in a single library togetherwith other library facilities not covered by this License, and distribute such a combined library, provided thatthe separate distribution of the work based on the Library and of the other library facilities is otherwise permitted,and provided that you do these two things:

a. Accompany the combined library with a copy of the same work based on the Library, uncombinedwith any other library facilities. This must be distributed under the terms of the Sections above.

b. Give prominent notice with the combined library of the fact that part of it is a work based on theLibrary, and explaining where to find the accompanying uncombined form of the same work.

C.2.2.9. Section 8

You may not copy, modify, sublicense, link with, or distribute the Library except as expressly provided underthis License. Any attempt otherwise to copy, modify, sublicense, link with, or distribute the Library is void,and will automatically terminate your rights under this License. However, parties who have received copies,or rights, from you under this License will not have their licenses terminated so long as such parties remain infull compliance.

C.2.2.10. Section 9

You are not required to accept this License, since you have not signed it. However, nothing else grants youpermission to modify or distribute the Library or its derivative works. These actions are prohibited by law ifyou do not accept this License. Therefore, by modifying or distributing the Library (or any work based on theLibrary), you indicate your acceptance of this License to do so, and all its terms and conditions for copying,distributing or modifying the Library or works based on it.

540syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.2.2.11. Section 10

Each time you redistribute the Library (or any work based on the Library), the recipient automatically receivesa license from the original licensor to copy, distribute, link with or modify the Library subject to these termsand conditions. You may not impose any further restrictions on the recipients' exercise of the rights grantedherein. You are not responsible for enforcing compliance by third parties with this License.

C.2.2.12. Section 11

If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limitedto patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradictthe conditions of this License, they do not excuse you from the conditions of this License. If you cannot distributeso as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then asa consequence you may not distribute the Library at all. For example, if a patent license would not permitroyalty-free redistribution of the Library by all those who receive copies directly or indirectly through you,then the only way you could satisfy both it and this License would be to refrain entirely from distribution ofthe Library.

If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance ofthe section is intended to apply, and the section as a whole is intended to apply in other circumstances.

It is not the purpose of this section to induce you to infringe any patents or other property right claims or tocontest validity of any such claims; this section has the sole purpose of protecting the integrity of the freesoftware distribution system which is implemented by public license practices. Many people have made generouscontributions to the wide range of software distributed through that system in reliance on consistent applicationof that system; it is up to the author/donor to decide if he or she is willing to distribute software through anyother system and a licensee cannot impose that choice.

This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.

C.2.2.13. Section 12

If the distribution and/or use of the Library is restricted in certain countries either by patents or by copyrightedinterfaces, the original copyright holder who places the Library under this License may add an explicitgeographical distribution limitation excluding those countries, so that distribution is permitted only in or amongcountries not thus excluded. In such case, this License incorporates the limitation as if written in the body ofthis License.

C.2.2.14. Section 13

The Free Software Foundation may publish revised and/or new versions of the Lesser General Public Licensefrom time to time. Such new versions will be similar in spirit to the present version, but may differ in detail toaddress new problems or concerns.

Each version is given a distinguishing version number. If the Library specifies a version number of this Licensewhich applies to it and “any later version”, you have the option of following the terms and conditions either ofthat version or of any later version published by the Free Software Foundation. If the Library does not specifya license version number, you may choose any version ever published by the Free Software Foundation.

541syslog-ng.com

TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

C.2.2.15. Section 14

If you wish to incorporate parts of the Library into other free programs whose distribution conditions areincompatible with these, write to the author to ask for permission. For software which is copyrighted by theFree Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Ourdecision will be guided by the two goals of preserving the free status of all derivatives of our free software andof promoting the sharing and reuse of software generally.

C.2.2.16. NO WARRANTY Section 15

BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THELIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISESTATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THELIBRARY “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY ANDFITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY ANDPERFORMANCE OF THE LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE,YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

C.2.2.17. Section 16

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANYCOPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THELIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANYGENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USEOR INABILITY TO USE THE LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA ORDATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIESOR A FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCHHOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

END OF TERMS AND CONDITIONS

C.2.3. How to Apply These Terms to Your New Libraries

If you develop a new library, and you want it to be of the greatest possible use to the public, we recommendmaking it free software that everyone can redistribute and change. You can do so by permitting redistributionunder these terms (or, alternatively, under the terms of the ordinary General Public License).

To apply these terms, attach the following notices to the library. It is safest to attach them to the start of eachsource file to most effectively convey the exclusion of warranty; and each file should have at least the “copyright”line and a pointer to where the full notice is found.

<one line to give the library's name and a brief idea of what it does.> Copyright (C) <year> <name of author>

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser GeneralPublic License as published by the Free Software Foundation; either version 2.1 of the License, or (at youroption) any later version.

542syslog-ng.com

How to Apply These Terms to Your New Libraries

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without eventhe implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNULesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not,write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Also add information on how to contact you by electronic and paper mail.

You should also get your employer (if you work as a programmer) or your school, if any, to sign a “copyrightdisclaimer” for the library, if necessary. Here is a sample; alter the names:

Yoyodyne, Inc., hereby disclaims all copyright interest in the library `Frob' (a library for tweaking knobs)written by James Random Hacker.

<signature of Ty Coon>, 1 April 1990 Ty Coon, President of Vice

That's all there is to it!

C.3. License attributions

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit(https://www.openssl.org/). This product includes cryptographic software written by Eric Young(eay@cryptsoft.com)

543syslog-ng.com

License attributions

Appendix D. Creative Commons AttributionNon-commercial No Derivatives (by-nc-nd)License

THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVECOMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHTAND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZEDUNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED. BY EXERCISING ANY RIGHTS TOTHE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS OF THISLICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED TO BE A CONTRACT, THELICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN CONSIDERATION OF YOURACCEPTANCE OF SUCH TERMS AND CONDITIONS.

1. Definitions

a. "Adaptation" means a work based upon the Work, or upon the Work and other pre-existing works,such as a translation, adaptation, derivative work, arrangement of music or other alterations of aliterary or artistic work, or phonogram or performance and includes cinematographic adaptationsor any other form in which the Work may be recast, transformed, or adapted including in anyform recognizably derived from the original, except that a work that constitutes a Collection willnot be considered an Adaptation for the purpose of this License. For the avoidance of doubt,where the Work is a musical work, performance or phonogram, the synchronization of the Workin timed-relation with a moving image ("synching") will be considered an Adaptation for thepurpose of this License.

b. "Collection" means a collection of literary or artistic works, such as encyclopedias and anthologies,or performances, phonograms or broadcasts, or other works or subject matter other than workslisted in Section 1(f) below, which, by reason of the selection and arrangement of their contents,constitute intellectual creations, in which the Work is included in its entirety in unmodified formalong with one or more other contributions, each constituting separate and independent works inthemselves, which together are assembled into a collective whole. A work that constitutes aCollection will not be considered an Adaptation (as defined above) for the purposes of thisLicense.

c. "Distribute" means to make available to the public the original and copies of the Work throughsale or other transfer of ownership.

d. "Licensor" means the individual, individuals, entity or entities that offer(s) the Work under theterms of this License.

e. "Original Author" means, in the case of a literary or artistic work, the individual, individuals,entity or entities who created the Work or if no individual or entity can be identified, the publisher;and in addition (i) in the case of a performance the actors, singers, musicians, dancers, and otherpersons who act, sing, deliver, declaim, play in, interpret or otherwise perform literary or artisticworks or expressions of folklore; (ii) in the case of a phonogram the producer being the personor legal entity who first fixes the sounds of a performance or other sounds; and, (iii) in the caseof broadcasts, the organization that transmits the broadcast.

544syslog-ng.com

f. "Work" means the literary and/or artistic work offered under the terms of this License includingwithout limitation any production in the literary, scientific and artistic domain, whatever may bethe mode or form of its expression including digital form, such as a book, pamphlet and otherwriting; a lecture, address, sermon or other work of the same nature; a dramatic ordramatico-musical work; a choreographic work or entertainment in dumb show; a musicalcomposition with or without words; a cinematographic work to which are assimilated worksexpressed by a process analogous to cinematography; a work of drawing, painting, architecture,sculpture, engraving or lithography; a photographic work to which are assimilated works expressedby a process analogous to photography; a work of applied art; an illustration, map, plan, sketchor three-dimensional work relative to geography, topography, architecture or science; aperformance; a broadcast; a phonogram; a compilation of data to the extent it is protected as acopyrightable work; or a work performed by a variety or circus performer to the extent it is nototherwise considered a literary or artistic work.

g. "You" means an individual or entity exercising rights under this License who has not previouslyviolated the terms of this License with respect to the Work, or who has received express permissionfrom the Licensor to exercise rights under this License despite a previous violation.

h. "Publicly Perform" means to perform public recitations of the Work and to communicate to thepublic those public recitations, by any means or process, including by wire or wireless means orpublic digital performances; to make available to the public Works in such a way that membersof the public may access these Works from a place and at a place individually chosen by them;to perform the Work to the public by any means or process and the communication to the publicof the performances of the Work, including by public digital performance; to broadcast andrebroadcast the Work by any means including signs, sounds or images.

i. "Reproduce" means to make copies of the Work by any means including without limitation bysound or visual recordings and the right of fixation and reproducing fixations of the Work,including storage of a protected performance or phonogram in digital form or other electronicmedium.

2. Fair Dealing Rights. Nothing in this License is intended to reduce, limit, or restrict any uses freefrom copyright or rights arising from limitations or exceptions that are provided for in connectionwith the copyright protection under copyright law or other applicable laws.

3. License Grant. Subject to the terms and conditions of this License, Licensor hereby grants You aworldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright)license to exercise the rights in the Work as stated below:

a. to Reproduce the Work, to incorporate the Work into one or more Collections, and to Reproducethe Work as incorporated in the Collections; and,

b. to Distribute and Publicly Perform the Work including as incorporated in Collections.The above rights may be exercised in all media and formats whether now known or hereafter devised.The above rights include the right to make such modifications as are technically necessary to exercisethe rights in other media and formats, but otherwise you have no rights to make Adaptations. Subjectto 8(f), all rights not expressly granted by Licensor are hereby reserved, including but not limitedto the rights set forth in Section 4(d).

4. Restrictions. The license granted in Section 3 above is expressly made subject to and limited by thefollowing restrictions:

545syslog-ng.com

a. You may Distribute or Publicly Perform the Work only under the terms of this License. Youmust include a copy of, or the Uniform Resource Identifier (URI) for, this License with everycopy of the Work You Distribute or Publicly Perform. You may not offer or impose any termson the Work that restrict the terms of this License or the ability of the recipient of the Work toexercise the rights granted to that recipient under the terms of the License. You may not sublicensethe Work. You must keep intact all notices that refer to this License and to the disclaimer ofwarranties with every copy of the Work You Distribute or Publicly Perform. When You Distributeor Publicly Perform the Work, You may not impose any effective technological measures on theWork that restrict the ability of a recipient of the Work from You to exercise the rights grantedto that recipient under the terms of the License. This Section 4(a) applies to the Work asincorporated in a Collection, but this does not require the Collection apart from the Work itselfto be made subject to the terms of this License. If You create a Collection, upon notice from anyLicensor You must, to the extent practicable, remove from the Collection any credit as requiredby Section 4(c), as requested.

b. You may not exercise any of the rights granted to You in Section 3 above in any manner that isprimarily intended for or directed toward commercial advantage or private monetary compensation.The exchange of the Work for other copyrighted works by means of digital file-sharing orotherwise shall not be considered to be intended for or directed toward commercial advantageor private monetary compensation, provided there is no payment of any monetary compensationin connection with the exchange of copyrighted works.

c. If You Distribute, or Publicly Perform the Work or Collections, You must, unless a request hasbeen made pursuant to Section 4(a), keep intact all copyright notices for the Work and provide,reasonable to the medium or means You are utilizing: (i) the name of the Original Author (orpseudonym, if applicable) if supplied, and/or if the Original Author and/or Licensor designateanother party or parties (for example a sponsor institute, publishing entity, journal) for attribution("Attribution Parties") in Licensor's copyright notice, terms of service or by other reasonablemeans, the name of such party or parties; (ii) the title of the Work if supplied; (iii) to the extentreasonably practicable, the URI, if any, that Licensor specifies to be associated with the Work,unless such URI does not refer to the copyright notice or licensing information for the Work.The credit required by this Section 4(c) may be implemented in any reasonable manner; provided,however, that in the case of a Collection, at a minimum such credit will appear, if a credit for allcontributing authors of Collection appears, then as part of these credits and in a manner at leastas prominent as the credits for the other contributing authors. For the avoidance of doubt, Youmay only use the credit required by this Section for the purpose of attribution in the manner setout above and, by exercising Your rights under this License, You may not implicitly or explicitlyassert or imply any connection with, sponsorship or endorsement by the Original Author, Licensorand/or Attribution Parties, as appropriate, of You or Your use of the Work, without the separate,express prior written permission of the Original Author, Licensor and/or Attribution Parties.

d. For the avoidance of doubt:

i. Non-waivable Compulsory License Schemes. In those jurisdictions in which the right to collectroyalties through any statutory or compulsory licensing scheme cannot be waived, the Licensorreserves the exclusive right to collect such royalties for any exercise by You of the rightsgranted under this License;

ii. Waivable Compulsory License Schemes. In those jurisdictions in which the right to collectroyalties through any statutory or compulsory licensing scheme can be waived, the Licensorreserves the exclusive right to collect such royalties for any exercise by You of the rights

546syslog-ng.com

granted under this License if Your exercise of such rights is for a purpose or use which isotherwise than noncommercial as permitted under Section 4(b) and otherwise waives the rightto collect royalties through any statutory or compulsory licensing scheme; and,

iii. Voluntary License Schemes. The Licensor reserves the right to collect royalties, whetherindividually or, in the event that the Licensor is a member of a collecting society thatadministers voluntary licensing schemes, via that society, from any exercise by You of therights granted under this License that is for a purpose or use which is otherwise thannoncommercial as permitted under Section 4(b).

e. Except as otherwise agreed in writing by the Licensor or as may be otherwise permitted byapplicable law, if You Reproduce, Distribute or Publicly Perform the Work either by itself or aspart of any Collections, You must not distort, mutilate, modify or take other derogatory actionin relation to the Work which would be prejudicial to the Original Author's honor or reputation.

5. Representations, Warranties and Disclaimer UNLESS OTHERWISE MUTUALLY AGREED BYTHE PARTIES IN WRITING, LICENSOR OFFERS THE WORK AS-IS AND MAKES NOREPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE WORK,EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT LIMITATION,WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR PURPOSE,NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY,OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE.SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES,SO SUCH EXCLUSION MAY NOT APPLY TO YOU.

6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, INNO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANYSPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGESARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HASBEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

7. Termination

a. This License and the rights granted hereunder will terminate automatically upon any breach byYou of the terms of this License. Individuals or entities who have received Collections from Youunder this License, however, will not have their licenses terminated provided such individualsor entities remain in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will surviveany termination of this License.

b. Subject to the above terms and conditions, the license granted here is perpetual (for the durationof the applicable copyright in the Work). Notwithstanding the above, Licensor reserves the rightto release the Work under different license terms or to stop distributing the Work at any time;provided, however that any such election will not serve to withdraw this License (or any otherlicense that has been, or is required to be, granted under the terms of this License), and thisLicense will continue in full force and effect unless terminated as stated above.

8. Miscellaneous

a. Each time You Distribute or Publicly Perform the Work or a Collection, the Licensor offers tothe recipient a license to the Work on the same terms and conditions as the license granted toYou under this License.

b. If any provision of this License is invalid or unenforceable under applicable law, it shall not affectthe validity or enforceability of the remainder of the terms of this License, and without further

547syslog-ng.com

action by the parties to this agreement, such provision shall be reformed to the minimum extentnecessary to make such provision valid and enforceable.

c. No term or provision of this License shall be deemed waived and no breach consented to unlesssuch waiver or consent shall be in writing and signed by the party to be charged with such waiveror consent.

d. This License constitutes the entire agreement between the parties with respect to the Work licensedhere. There are no understandings, agreements or representations with respect to the Work notspecified here. Licensor shall not be bound by any additional provisions that may appear in anycommunication from You. This License may not be modified without the mutual written agreementof the Licensor and You.

e. The rights granted under, and the subject matter referenced, in this License were drafted utilizingthe terminology of the Berne Convention for the Protection of Literary and Artistic Works (asamended on September 28, 1979), the Rome Convention of 1961, the WIPO Copyright Treatyof 1996, the WIPO Performances and Phonograms Treaty of 1996 and the Universal CopyrightConvention (as revised on July 24, 1971). These rights and subject matter take effect in therelevant jurisdiction in which the License terms are sought to be enforced according to thecorresponding provisions of the implementation of those treaty provisions in the applicablenational law. If the standard suite of rights granted under applicable copyright law includesadditional rights not granted under this License, such additional rights are deemed to be includedin the License; this License is not intended to restrict the license of any rights under applicablelaw.

548syslog-ng.com

Glossary

alias IP An additional IP address assigned to an interface that already has an IP address.The normal and alias IP addresses both refer to the same physical interface.

authentication The process of verifying the authenticity of a user or client before allowingaccess to a network system or service.

auditing policy The auditing policy determines which events are logged on host runningMicrosoft Windows operating systems.

BOM The byte order mark (BOM) is a Unicode character used to signal the byte-orderof the message text.

BSD-syslog protocol The old syslog protocol standard described in RFC 3164. Sometimes alsoreferred to as the legacy-syslog protocol.

CA A Certificate Authority (CA) is an institute that issues certificates.

certificate A certificate is a file that uniquely identifies its owner. Certificates containsinformation identifying the owner of the certificate, a public key itself, theexpiration date of the certificate, the name of the CA that signed the certificate,and some other data.

client mode In client mode, syslog-ng collects the local logs generated by the host andforwards them through a network connection to the central syslog-ng serveror to a relay.

destination A named collection of configured destination drivers.

destination driver A communication method used to send log messages.

destination, network A destination that sends log messages to a remote host (that is, a syslog-ngrelay or server) using a network connection.

destination, local A destination that transfers log messages within the host, for example writesthem to a file, or passes them to a log analyzing application.

disk buffer The Premium Edition of syslog-ng can store messages on the local hard diskif the central log server or the network connection to the server becomesunavailable.

disk queue See disk buffer.

domain name The name of a network, for example: balabit.com.

549syslog-ng.com

embedded log statement A log statement that is included in another log statement to create a complexlog path.

filter An expression to select messages.

fully qualified domain name(FQDN)

A domain name that specifies its exact location in the tree hierarchy of theDomain Name System (DNS). For example, given a device with a localhostname myhost and a parent domain name example.com, the fully qualifieddomain name is myhost.example.com.

gateway A device that connects two or more parts of the network, for example: yourlocal intranet and the external network (the Internet). Gateways act as entrancesinto other networks.

high availability High availability uses a second syslog-ng server unit to ensure that the logsare received even if the first unit breaks down.

host A computer connected to the network.

hostname A name that identifies a host on the network.

IETF-syslog protocol The syslog-protocol standard developed by the Internet Engineering TaskForce (IETF), described in RFC 5424-5427.

key pair A private key and its related public key. The private key is known only to theowner, while the public key can be freely distributed. Information encryptedwith the private key can only be decrypted using the public key.

license The syslog-ng license determines the number of distinct hosts (clients andrelays) that can connect to the syslog-ng server.

log path A combination of sources, filters, parsers, rewrite rules, and destinations:syslog-ng examines all messages arriving to the sources of the log path andsends the messages matching all filters to the defined destinations.

logstore A binary logfile format that can encrypt, compress, and timestamp logmessages.

Long Term Supported release Long Term Supported releases are major releases of syslog-ng PE that aresupported for three years after their original release.

LSH See log source host.

LTS See Long Term Supported release.

log source host A host or network device (including syslog-ng clients and relays) that sendslogs to the syslog-ng server. Log source hosts can be servers, routers, desktopcomputers, or other devices capable of sending syslog messages or runningsyslog-ng.

550syslog-ng.com

log statement See log path.

name server A network computer storing the IP addresses corresponding to domain names.

Oracle Instant Client The Oracle Instant Client is a small set of libraries, which allow you to connectto an Oracle Database. A subset of the full Oracle Client, it requires minimalinstallation but has full functionality.

output buffer A part of the memory of the host where syslog-ng stores outgoing log messagesif the destination cannot accept the messages immediately.

output queue Messages from the output queue are sent to the target syslog-ng server. Thesyslog-ng application puts the outgoing messages directly into the outputqueue, unless the output queue is full. The output queue can hold 64 messages,this is a fixed value and cannot be modified.

overflow queue See output buffer.

parser A set of rules to segment messages into named fields or columns.

ping A command that sends a message from a host to another host over a networkto test connectivity and packet loss.

port A number ranging from 1 to 65535 that identifies the destination applicationof the transmitted data. For example: SSH commonly uses port 22, web servers(HTTP) use port 80, and so on.

Public-key authentication An authentication method that uses encryption key pairs to verify the identityof a user or a client.

regular expression A regular expression is a string that describes or matches a set of strings. Thesyslog-ng application supports extended regular expressions (also called POSIXmodern regular expressions).

relay mode In relay mode, syslog-ng receives logs through the network from syslog-ngclients and forwards them to the central syslog-ng server using a networkconnection.

rewrite rule A set of rules to modify selected elements of a log message.

template A user-defined structure that can be used to restructure log messages orautomatically generate file names.

server mode In server mode, syslog-ng acts as a central log-collecting server. It receivesmessages from syslog-ng clients and relays over the network, and stores themlocally in files, or passes them to other applications, for example, log analyzers.

source A named collection of configured source drivers.

551syslog-ng.com

source, network A source that receives log messages from a remote host using a networkconnection, for example, network(), syslog().

source, local A source that receives log messages from within the host, for example, froma file.

source driver A communication method used to receive log messages.

SSL See TLS.

syslog-ng The syslog-ng application is a flexible and highly scalable system loggingapplication, typically used to manage log messages and implement centralizedlogging.

syslog-ng agent The syslog-ng Agent for Windows is a commercial log collector and forwarderapplication for the Microsoft Windows platform. It collects the log messagesof the Windows-based host and forwards them to a syslog-ng server usingregular or SSL-encrypted TCP connections.

syslog-ng client A host running syslog-ng in client mode.

syslog-ng Premium Edition The syslog-ng Premium Edition is the commercial version of the open-sourceapplication. It offers additional features, like encrypted message transfer andan agent for Microsoft Windows platforms.

syslog-ng relay A host running syslog-ng in relay mode.

syslog-ng server A host running syslog-ng in server mode.

TLS Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer(SSL), are cryptographic protocols which provide secure communications onthe Internet. The syslog-ng Premium Edition application can encrypt thecommunication between the clients and the server using TLS to preventunauthorized access to sensitive log messages.

traceroute A command that shows all routing steps (the path of a message) between twohosts.

UNIX domain socket A UNIX domain socket (UDS) or IPC socket (inter-procedure call socket) isa virtual socket, used for inter-process communication.

552syslog-ng.com

List of syslog-ng PEparameters

Symbols$(echo), 391$(indent-multi-line ${MESSAGE}), 99, 109, 147, 163$columns, 135$DATE, 30$FACILITY, 30$fetch_limit, 135$FULLHOST_FROM, 385$HOST, 30$HOST_FROM, 386$last_read_uid, 135$MESSAGE, 30$MSGID, 30$PID, 30$PRIORITY, 30$PROGRAM, 30, 479$RANDFILE, 477$SEQNUM, 30$SOURCEIP, 30$table, 134, 135$TAGS, 30$uid, 135$UNIXTIME, 26$_, 402${.SDATA.SDID.SDNAME}, 388${AMPM}, 383, 385${DATE}, 381, 384${DAY}, 378${EVENT_CATEGORY}, 89${EVENT_LEVEL}, 89, 90${EVENT_MESSAGE_XML}, 89${EVENT_MSG_XML}, 89${EVENT_SOURCE}, 90${EVENT_TASK}, 89${EVENT_TYPE}, 90${EVENT_USERNAME}, 90${FULLHOST_FROM}, 385, 390${HOST_FROM}, 386${HOST}, 12, 70, 191, 218, 219, 378, 385, 391, 392${HOUR12}, 383${HOUR}, 381

${ISODATE}, 381, 386, 391${LEVEL_NUM}, 386, 397${LEVEL}, 386, 388${MESSAGE}, 99, 109, 147, 163, 325, 411, 481${MSGHDR}, 379, 387${MSGONLY}, 387, 388${MSG}, 24, 133, 387${PID}, 326, 388${PRI}, 90, 388${PROGRAM}, 191, 216, 218, 269, 388, 480, 481${R_DATE}, 381${SDATA}, 388${SEQNUM}, 389, 390${SYSLOGNG_OPTIONS}, 37, 46, 48${S_DATE}, 381${TAGS}, 328, 390, 445${TZOFFSET}, 391${WEEKDAY}, 196${YEAR}, 397--active-connections, 469--ca-dir, 476--ca-file, 476--caps, 493, 494--debug, 454, 494--debug-csv, 481--debug-pattern, 481--dgram, 469--enable-linux-caps, 494--enable-spoof-source, 74--fd-limit, 191, 218, 219, 341, 494--foreground, 494, 495--group, 494--idle-connections, 468--inet, 468--interval, 469--length, 395, 396--no-caps, 493, 494--no-framing, 470--number, 469--qdisk-dir=, 174, 185, 202, 210, 229, 236, 253, 262, 282, 290, 303--read-file, 469, 470--require-ts, 476--sdata, 470--skip-tokens, 469--stderr, 495, 503, 504--syslog-proto, 469, 470--user, 494, 495

553syslog-ng.com

--verbose, 454, 495--worker-threads, 449, 450, 495-V, 362.classifier.<message-class>, 333, 423.classifier.class, 422.classifier.rule_id, 422.classifier.system, 333, 423.classifier_class, 422.SDATA.meta, 328.USER, 402<action>, 426, 442, 444<message>, 426, 442, 444@DOUBLE@, 432@FLOAT@, 432@module, 80, 268@version, 80, 81

AAF_UNIX, 158, 159alert, 331allow-compress(), 357, 366AMPM, 381, 383api(), xviAPPLICATION_NAME, 383assume-utf8, xv, 95, 103, 114, 119, 136, 142, 159auth-password(), 271auth-username(), 271

Bbalabit.com, 549bcc(), 261body(), 259boolean, 26BSDTAG, 382, 383

Cca-dir(), 357, 359catchall, 314, 315cc(), 261cert-subject(), xiv, 357chain-hostnames(), 16, 72, 334, 384, 385CHARSETALIASDIR, 39, 43chunk-size(), 220chunk-time(), 220cipher-suite(), 358cisco-snmp(), 270cisco_snmp(), 270

class-path, 201client-host, 73client-hostname-from-the-message, 334client-hostname-resolved-on-the-relay, 334client-hostname-resolved-on-the-server, 334cluster(), 171, 172, 173, 182, 183columns(), 126, 135, 274, 281, 288Common Name, 351, 352, 355community(), 271, 274concurrent-requests, 171, 176, 182, 187concurrent-requests(), 173, 184concurrent_requests(), 173, 184condition, 427condition(), 402, 403context, 429, 443context-id, 424, 437, 438context-scope, 424, 429, 430, 437, 438, 444context-timeout, 424, 425, 430, 443CONTEXT_ID, 382create-dirs(), 190, 192, 215, 220, 221, 336, 337crit, 331crl-dir(), 359csv-parser(), 12, 407, 409curve-list(), xiv, 360

Ddatabase(), 123, 127, 228, 274, 276, 281DATE, 349, 381, 382date-column(), 123, 127datetime, 26datetime-column(), 123, 127DAY, 381, 382db-parser(), 421, 426debug, 331, 333, 494default-facility(), 85, 88, 93, 94, 128, 154default-priority(), 85, 88, 93, 95, 128delimiters(<delimiter_characters>), 410destination, 76, 82, 498dhparam-file(), xiv, 360dir(), xiv, 174, 185, 202, 210, 229, 236, 253, 262, 282, 290, 303dir-group(), 192, 221, 337dir-owner(), 192, 221, 337dir-perm(), 192, 221, 337disk-buf-size(), 176, 187, 204, 212, 230, 237, 255, 264, 283, 292, 304, 319, 320, 371, 373

554syslog-ng.com

disk-buffer(), xiv, 174, 184, 185, 202, 209, 210, 228, 229, 232, 235, 236, 252, 253, 261, 262, 281, 282, 289, 290, 302, 303dns-cache(), 338, 385, 386dns-cache-hosts(), 338, 462dont-create-tables, 284dont-store-legacy-msghdr, 95, 103, 114, 119, 136, 142, 159door(), 135, 136dot-nv-pairs, 30drop-invalid, 410drop-message, 177, 188, 205, 212, 231, 344drop-property, 177, 188, 206, 212, 231, 344, 393dropped, 446, 447

Eelasticsearch, 168, 169, 170, 171, 172elasticsearch(), 181, 320, 322, 323elasticsearch2, 167, 179, 180, 181, 182, 500elasticsearch2(), 170, 320, 322, 323emerg, 331empty-lines, 95, 103, 114, 119, 137, 142, 159Enable-FIPS, 362enc-password(), 272encoding(), 25, 95, 103, 141, 159, 393engine-id(), 267, 272, 274err, 331error, 388event-api(), xvi, 88eventlog(), 11, 87, 88exclude(), 25, 27, 28expect-hostname, 95, 104, 114, 119, 137, 142, 160explicit-commits, 284

FFACILITY, 382, 384facility, 460facility(), 329FACILITY_NUM, 382, 384fallback, 314, 315, 410fallback-to-string, 177, 188, 206, 212, 231, 344fetch-query(), 128, 130, 131file, 100, 193, 421, 450file(), 11, 76, 92, 94, 97, 113, 190, 191, 195, 219, 241, 248, 249, 257, 295, 307, 342, 343, 498FILE_MESSAGE, 384filter, 76, 82, 498filter(), 330, 402

final, 9, 314, 315, 333FIPS-mode='enabled', 362flag(syslog-protocol), 31flags(), 76, 95, 103, 114, 119, 136, 141, 159, 221, 238, 246, 255, 292, 304, 311, 312, 399, 410, 498flags(no-multi-line), 96, 99, 104, 109, 115, 120, 137, 142, 147, 160, 163, 383, 387flags(no-parse), 24, 96, 104, 115, 120, 137, 142, 160, 383, 387flow-control, 311, 312, 315, 316, 318, 319flush-limit(), 173, 184flush-lines(), 91, 107, 116, 121, 130, 138, 145, 162, 193, 239, 247, 255, 284, 293, 305, 339, 346, 365, 375, 450, 461flush-size(), 364, 456flush-timeout(), 193, 239, 247, 255, 284, 293, 305, 339, 450flush_limit, 171, 176, 181, 182, 187flush_lines, 456follow-freq(), 76, 92, 94, 96, 115, 128, 129, 138, 498force-directory-polling(), xiv, 34, 92, 96, 97format-cef-extension, xiv, 392, 393format-json, 26, 29, 393, 412, 415format-json(), 25, 214format-welf, 133format-welf(), 25, 133, 394frac-digits(), 176, 177, 187, 193, 204, 212, 222, 230, 231, 239, 247, 256, 264, 284, 285, 293, 305, 339, 348, 386from(), 259, 264, 266fsync(), 194FULLDATE, 381, 382FULLHOST, 382, 384, 402FULLHOST_FROM, 382, 385

Ggreedy, 409, 410, 411greedy(), 432grep, 395group(), 160, 194, 222, 247, 248, 339groupset(), 401, 402

Hhdfs, 198, 199, 200, 201hdfs(), 5, 80, 320, 322, 323hdfs_file(), 201hdfs_max_filename_length, 205hdfs_uri(), 201

555syslog-ng.com

HEADER, 19, 21header(), 259, 264HOST, 106, 144, 332, 340, 382, 385, 402host, 276host(), 123, 129, 130, 259, 267, 272, 285, 327, 330, 406host-override(), xiv, 91, 105, 143, 154, 161host-template(), 130HOST_FROM, 382, 386HOUR, 179, 190, 197, 207, 215, 225, 233, 245, 251, 259, 266, 273, 288, 299, 309, 347, 349, 381, 382HOUR12, 381

Iin-list, 330in-list filter, 330index(), 172, 182indexes, 285indexes(), 285info, 331inherit-properties, 426, 429, 442, 444inject-mode(), 426inlist(), xviint, 26int32, 26int64, 27internal, 426internal(), 79, 90, 91, 169, 177, 180, 188, 198, 205, 206, 207, 212, 231, 344, 426, 443ISODATE, 381, 382, 383

Jjava, 450java(), 168, 172, 179, 182, 198, 201, 207, 209journal-block-count(), 222journal-block-size(), 217, 218, 220, 223, 342json-parser(), 414

Kkafka, 207, 208, 209, 213kafka(), 320, 322, 323kafka-bootstrap-servers, 213kafka-bootstrap-servers(), 209keep-alive, 106, 144, 240, 294, 306keep-alive(), 106, 144, 161, 240, 252, 294, 306keep-hostname(), 72, 73, 106, 112, 113, 130, 144, 150, 151, 154, 156, 334, 338, 340, 343, 348, 384, 385, 386

keep-timestamp(), 13, 97, 106, 107, 115, 116, 120, 138, 145, 161, 340, 381kern, 93, 384kernel, 95, 104, 114, 119, 137, 142, 160key(), 25, 27, 28, 29, 225klogd, 94ksymoops, 94kv-parser, 413kv-parser(), 412

LLEGACY_MSGHDR, 382LEVEL, 382, 388level(), 331LEVEL_NUM, 382, 386license.txt, 41literal, 26localip(), 101, 240, 294log, 76, 82, 498log-disk-fifo-size(), 175, 185, 203, 210, 229, 236, 253, 262, 282, 290, 303, 321, 323log-fetch-limit(), 97, 107, 116, 120, 126, 130, 135, 138, 145, 155, 161, 316, 319, 320, 449log-fifo-size(), 175, 177, 186, 188, 194, 203, 205, 211, 213, 222, 229, 230, 231, 236, 237, 240, 248, 254, 256, 263, 265, 272, 282, 283, 286, 291, 294, 303, 304, 306, 316, 318, 319, 320, 321, 323, 341log-iw-size(), xiv, 91, 97, 107, 116, 121, 130, 138, 145, 161, 162, 193, 239, 247, 255, 284, 293, 305, 316, 319, 320, 365, 369, 449log-msg-size(), 19, 25, 98, 107, 116, 121, 123, 139, 145, 152, 162, 244, 276, 277, 298, 341logstore, 215, 450logstore(), 11, 34, 215, 216, 219logstore-journal-shmem-threshold(), 217, 218, 222, 341, 342log_iw_size(), 91, 107, 116, 121, 130, 138, 145, 161, 373log_msg_size(), 98, 107, 116, 121, 139, 145, 162, 341, 364

MMARK, 194, 195, 196, 240, 241, 242, 248, 249, 256, 257, 295, 296, 306, 307, 308, 342, 343mark(), 194, 240, 248, 256, 295, 306, 342mark-freq, 430mark-freq(), 194, 240, 248, 256, 294, 295, 306, 342

556syslog-ng.com

mark-mode(), 194, 195, 196, 240, 241, 242, 248, 249, 256, 257, 295, 296, 306, 307, 308, 342, 343match, 325match(), 325, 327, 331, 406max-connections(), 91, 107, 116, 121, 130, 138, 145, 146, 158, 162, 316, 317, 319, 320, 450max-field-size(), 152, 155max-uid-query(), xiv, 130mem-buf-length(), 371mem-buf-size(), 319, 320MESSAGE, 382message(), 331message-acknowledgement-timeout(), 365, 366MIN, 381, 382mod-java, 80mongodb(), 6, 11, 25, 26, 27, 225, 226, 227, 232, 233, 320, 322, 323MONTH, 381, 382MONTH_ABBREV, 381, 382MONTH_NAME, 381, 382MONTH_WEEK, 382MSEC, 381MSG, 19, 21, 331, 382MSGHDR, 331, 387MSGID, 382, 387MSGONLY, 382, 388mssql, 134, 277, 288multi-line-garbage(), 98, 99, 100, 108, 109, 146, 147, 162, 163, 164multi-line-prefix(), 98, 99, 100, 108, 109, 146, 147, 162, 163, 164myhost, 402MYSQL_UNIX_PORT, 130, 279, 285

Nname, 444netmask(), 330, 332netmask6(), 332network(), xv, 31, 96, 101, 103, 104, 105, 115, 119, 137, 142, 143, 156, 157, 160, 195, 234, 235, 241, 248, 249, 257, 295, 300, 307, 317, 320, 322, 323, 324, 342, 343, 348, 350, 351, 352, 354, 356, 364, 366, 367, 552n e t w o r k ( t r a n s p o r t ( r l t p ( t l s - r e q u i r e d ( y e s ) )flag(syslog-protocol)), 32network(transport(rltp(tls-required(yes))), 32, 34network(transport(rltp)), 32, 33network(transport(rltp)flag(syslog-protocol)), 32network(transport(tcp) flag(syslog-protocol)), 32

network(transport(tcp)), 31, 32, 33network(transport(tls) flag(syslog-protocol)), 32network(transport(tls)), 32, 33network(transport(udp) flag(syslog-protocol)), 32network(transport(udp)), 31, 32no-hostname, 95, 104, 114, 119, 137, 142, 160no-multi-line, 96, 104, 115, 119, 137, 142, 160, 192, 238, 246, 255, 293, 305no-parse, 16, 96, 104, 115, 120, 137, 142, 160nobody, 402normalize-hostnames(), xiv, 91, 343, 385, 386notice, 331NULL, 279null(), 279, 286nv-pairs, 30

Ooff, 362on, 362on-error, 393on-error(), 177, 188, 206, 213, 232, 343ORACLE_BASE, 276ORACLE_HOME, 276ORACLE_SID, 276other, 448overriding-original-program-name, 427, 442, 444overwrite-if-older(), 196owner(), 164, 196, 223, 249, 344

Ppad-size(), 100, 110, 117, 121, 122, 139, 148, 164, 165, 196, 249pair(), 26, 27, 28Parameters, xiiparser, 76, 82, 498path(), 232path.home, 171, 172, 173, 182, 183peer-verify(), 360, 367perm(), 165, 196, 197, 223, 250, 344persist_only, 112, 150, 348, 462pid, 286PID, 382, 388pipe, 114, 450pipe(), 6, 11, 113, 114, 117, 121, 139, 164, 195, 241, 246, 248, 249, 257, 295, 307, 342, 343poll(), 96, 115, 129, 138, 346port(), 131, 157, 171, 172, 173, 182, 183, 259, 272, 287, 301

557syslog-ng.com

prefix(), 89, 131, 133, 155, 412, 413, 414, 416PRI, 19, 21, 382, 388PRIORITY, 382, 388processed, 446, 447PROGRAM, 132, 382, 388program(), 6, 118, 119, 122, 195, 241, 248, 249, 251, 252, 257, 295, 307, 320, 322, 323, 327, 332, 342, 343, 406, 428program-override(), xiv, 91, 98, 100, 110, 116, 117, 121, 122, 139, 148, 162, 165properties-file, 209proto-template, 348

Qquot-size(), 321quote-pairs(), 407, 409, 411

RRCPTID, 382, 388read-old-records(no), 130read-old-records(yes), 130recursive, 100recv-time-zone(), 13, 14, 345rekey(), 29relay-hostname-resolved-on-the-server, 334reliable(), 174, 175, 176, 185, 186, 187, 202, 203, 204, 210, 211, 212, 228, 229, 230, 235, 236, 237, 253, 254, 255, 262, 263, 264, 281, 282, 283, 290, 291, 292, 302, 303, 304, 321, 322reply-to(), 265resource(), 171, 172, 182, 183response-timeout(), 366retries, 178, 189, 206, 214, 232, 287retries(), 171, 176, 181, 187, 200, 214, 267rewrite, 76, 82, 498rfc3164, 30rfc5424, 30rltp, 96, 104, 115, 119, 137, 142, 160root, 82, 83R_HOUR, 349R_UNIXTIME, 14

Ssafe-mode(), 227, 232scope(), 25, 27, 28, 29, 30SDATA, 382, 388SEC, 382

selected-macros, 30send-time-zone(), 13sender(), 264SEQNUM, 382, 389serialized, 221server(), 171, 172, 173, 182, 183, 226server-hostname, 334servers(), 226, 227, 232silently-drop-message, 177, 188, 206, 212, 231, 344silently-drop-property, 177, 188, 206, 213, 231, 344silently-fallback-to-string, 177, 188, 206, 213, 231, 344smtp(), 6, 259, 260, 320, 322, 323snmp, 267snmp(), 6, 80, 267, 268, 270, 271, 274, 320, 322, 323snmp-obj(), 267, 273, 274so-rcvbuf(), 111, 112, 141, 149, 150, 165, 242, 296, 308, 455, 460SOCK_DGRAM, 85, 86, 87, 158, 168, 301, 302, 500, 501SOCK_STREAM, 85, 86, 87, 158, 168, 301, 302, 500, 501source, 76, 82, 498SOURCE, 382, 390source(), 314, 332SOURCEIP, 382, 390SO_BROADCAST, 110, 148, 242, 296, 306spoof-interface(), 243, 244, 297, 298spoof-source, 244, 298spoof-source(), 73, 243, 298sql, 450sql(), 5, 6, 11, 36, 123, 124, 125, 168, 274, 275, 276, 280, 283, 284, 320, 322, 323, 324, 380, 501STAMP, 348, 382stamp, 448stats-level(), 345, 446, 448, 508stdin, 251, 252stored, 447STREAMS, 86, 87, 135, 500string, 27, 405strip-whitespace, 411STRUCTURED-DATA, 21subject(), 259, 266subject_alt_name, 351, 352, 355subkeys(), 27, 31sun-streams(), 6, 135, 136suppress(), 197, 244, 250, 257, 298, 308, 448suppressed, 448

558syslog-ng.com

sync_send, 214syslog, 96, 104, 105, 115, 120, 137, 142, 143, 160, 192, 238, 247, 255, 289, 293, 305, 450syslog(), 31, 70, 96, 102, 104, 105, 115, 119, 137, 140, 141, 142, 143, 160, 195, 241, 248, 249, 257, 288, 289, 295, 307, 320, 322, 323, 324, 342, 343, 348, 350, 351, 352, 354, 356, 364, 366, 552syslog(transport(rltp(tls-required(yes))), 32, 33syslog(transport(rltp)), 31, 33syslog(transport(tcp)), 31, 33syslog(transport(tls)), 31, 33syslog(transport(udp)), 31syslog-ng-relay, 73syslog-ng-server, 73syslog-protocol, 96, 104, 115, 120, 137, 142, 160, 192, 234, 238, 247, 255, 293, 305syslogd, 85, 86, 135, 158, 197, 206, 224, 244, 250, 258, 298, 308syslog_ng, 58system(), 151systemd-journal(), 152, 154systemd-syslog(), 156S_DATE, 349S_HOUR, 349S_UNIXTIME, 14

Ttable, 274table(), 123, 132, 135, 274, 287TAG, 382, 390TAGS, 382, 390tags(), 24, 91, 101, 111, 117, 122, 132, 140, 149, 165, 328, 333, 422, 423tcp, 105, 143, 288, 450tcp(), xv, 31, 156, 157, 300, 320, 322, 323tcp6(), xv, 156, 157, 300, 320, 322, 323template, 76, 498template(), 131, 133, 197, 206, 223, 244, 250, 258, 298, 308, 411, 414, 416template-escape(), 197, 244, 250, 258, 298, 308, 379test_log, 123threaded, 104, 143, 193, 451threaded(), 346, 449throttle, 455time-column(), 123time-reap(), 191, 216, 217, 220, 346time-reopen(), 133, 171, 181, 200, 238, 267, 278, 292, 346

time-sleep(), xvitime-zone(), 13, 101, 111, 118, 122, 134, 140, 149, 155, 166, 179, 190, 197, 206, 215, 225, 233, 245, 251, 258, 266, 273, 287, 299, 309, 347, 381timestamp-freq(), 224, 347timestamp-url(), 224, 347tls, 105, 143, 288tls(), 112, 150, 245, 299, 353, 354, 356, 365, 367, 368tls-required(), 367, 368to(), 259, 267topic(), 209transport(rltp(tls-required(yes)), 31transport(rltp), 31transport(tls), 31, 356trap-obj(), 267, 273, 274trigger, 443ts-format(), 21, 23, 179, 190, 198, 207, 215, 225, 245, 251, 259, 300, 309, 348, 390type(), 123, 134, 172, 182, 274, 288, 327, 399, 403, 404TZ, 382TZOFFSET, 382

Uudp, 105, 143, 288, 320, 322, 323, 450udp(), xv, 31, 156, 157, 300udp6(), xv, 156, 157, 300uid-column(), 123, 134, 135ulimit, 191, 219unicode, 405unix-dgram, 85, 117, 121, 139, 163, 164, 455unix-dgram(), 6, 96, 104, 115, 119, 137, 142, 158, 159, 160, 161, 162, 195, 241, 248, 249, 257, 295, 301, 302, 307, 320, 322, 323, 342, 343unix-stream, 85, 117, 121, 139, 164, 455unix-stream(), 6, 77, 158, 159, 195, 241, 248, 249, 257, 295, 301, 302, 307, 320, 322, 323, 342, 343, 498UNIXTIME, 382unknown, 422use-dns(), 73, 106, 112, 144, 150, 154, 340, 348, 385, 386, 462use-fqdn(), xiv, 92, 112, 150, 155, 348, 385, 386use-rcptid(), xv, 349use-uniqid(), 174, 184, 349USEC, 381usertty(), 309, 460

559syslog-ng.com

Vvalidate-utf8, 95, 96, 103, 104, 114, 115, 119, 120, 136, 137, 142, 143, 159, 160, 393value, 274, 444value(), 325, 331value-pairs, 25value-pairs(), 27, 177, 188, 206, 213, 225, 232, 233, 343, 392, 393, 394values(), 274, 288, 401varchar, 277

Wwarning, 331WEEK, 382WEEK_DAY, 382WEEK_DAY_ABBREV, 382WEEK_DAY_NAME, 382

YYEAR, 382YEAR_DAY, 382

560syslog-ng.com

Index

Symbols$(echo), 391$(indent-multi-line ${MESSAGE}), 99, 109, 147, 163$columns, 135$DATE, 30$FACILITY, 30$fetch_limit, 135$FULLHOST_FROM, 385$HOST, 30$HOST_FROM, 386$last_read_uid, 135$MESSAGE, 30$MSGID, 30$PID, 30$PRIORITY, 30$PROGRAM, 30, 479$RANDFILE, 477$SEQNUM, 30$SOURCEIP, 30$table, 134, 135$TAGS, 30$uid, 135$UNIXTIME, 26$_, 402${.SDATA.SDID.SDNAME}, 388${AMPM}, 383, 385${BSDTAG}, 383${DATE}, 381, 384${DATE}, ${R_DATE}, ${S_DATE}, 384${DAY}, 378${DAY}, ${R_DAY}, ${S_DAY}, 384${EVENT_CATEGORY}, 89${EVENT_LEVEL}, 89, 90${EVENT_MESSAGE_XML}, 89${EVENT_MSG_XML}, 89${EVENT_SOURCE}, 90${EVENT_TASK}, 89${EVENT_TYPE}, 90${EVENT_USERNAME}, 90${FACILITY_NUM}, 384${FACILITY}, 384${FULLDATE}, ${R_FULLDATE},${S_FULLDATE}, 384

${FULLHOST_FROM}, 385, 390${FULLHOST}, 384${HOST_FROM}, 386${HOST}, 12, 70, 191, 218, 219, 378, 385, 391, 392${HOUR12}, 383${HOUR12}, ${R_HOUR12}, ${S_HOUR12}, 385${HOUR}, 381${HOUR}, ${R_HOUR}, ${S_HOUR}, 385${ISODATE}, 381, 386, 391${ISODATE}, ${R_ISODATE}, ${S_ISODATE},386${LEVEL_NUM}, 386, 397${LEVEL}, 386, 388${MESSAGE}, 99, 109, 147, 163, 325, 411, 481${MIN}, ${R_MIN}, ${S_MIN}, 387${MONTH_ABBREV}, ${R_MONTH_ABBREV},${S_MONTH_ABBREV}, 387${MONTH_NAME}, ${R_MONTH_NAME},${S_MONTH_NAME}, 387${MONTH}, ${R_MONTH}, ${S_MONTH}, 387${MSEC}, ${R_MSEC}, ${S_MSEC}, 387${MSGHDR}, 379, 387${MSGID}, 387${MSGONLY}, 387, 388${MSG}, 24, 133, 387${MSG} or ${MESSAGE}, 387${ONTH_WEEK}, ${R_MONTH_WEEK},${S_MONTH_WEEK}, 387${OSUPTIME}, 388${PID}, 326, 388${PRIORITY}, 388${PRI}, 90, 388${PROGRAM}, 191, 216, 218, 269, 388, 480, 481${RCPTID}, 388${R_DATE}, 381${SDATA}, 388${SEC}, ${R_SEC}, ${S_SEC}, 389${SEQNUM}, 389, 390${SOURCEIP}, 390${SOURCE}, 390${STAMP}, ${R_STAMP}, ${S_STAMP}, 390${STRUCTURED_DATA}, 388${SYSLOGNG_OPTIONS}, 37, 46, 48${SYSUPTIME}, 390${S_DATE}, 381${TAGS}, 328, 390, 445${TAG}, 390${TZOFFSET}, 391

561syslog-ng.com

${TZOFFSET}, ${R_TZOFFSET}, ${S_TZOFFSET},391${TZ}, ${R_TZ}, ${S_TZ}, 391${UNIQID}, 391${UNIXTIME}, ${R_UNIXTIME},${S_UNIXTIME}, 391${USEC}, ${R_USEC}, ${S_USEC}, 391${WEEKDAY}, 196${WEEKDAY}, ${R_WEEKDAY},${S_WEEKDAY}, 391${WEEK_ABBREV}, ${R_WEEK_ABBREV},${S_WEEK_ABBREV}, 391$ { W E E K _ D A Y _ N A M E } ,$ { R _ W E E K _ D A Y _ N A M E } ,${S_WEEK_DAY_NAME}, 391${WEEK_DAY}, ${R_WEEK_DAY},${S_WEEK_DAY}, 391${WEEK}, ${R_WEEK}, ${S_WEEK}, 391${YEAR}, 397${YEAR}, ${R_YEAR}, ${S_YEAR}, 391--active-connections, 469--ca-dir, 476--ca-file, 476--caps, 493, 494--cfgfile, 493--chroot, 494--debug, 454, 494--debug-csv, 481--debug-pattern, 481--default-modules, 494--dgram, 469--enable-core, 494--enable-linux-caps, 494--enable-spoof-source, 74--fd-limit, 191, 218, 219, 341, 494--foreground, 494, 495--group, 494--help, 494--idle-connections, 468--inet, 468--interval, 469--length, 395, 396--module-registry, 494--no-caps, 493, 494--no-framing, 470--number, 469--persist-file, 494--pidfile, 37, 46, 48, 495

--preprocess-into, 495--process-mode, 495--qdisk-dir, 495--qdisk-dir=, 174, 185, 202, 210, 229, 236, 253, 262, 282, 290, 303--read-file, 469, 470--require-ts, 476--sdata, 470--skip-tokens, 469--stderr, 495, 503, 504--syntax-only, 495--syslog-proto, 469, 470--user, 494, 495--verbose, 454, 495--version, 495--worker-threads, 449, 450, 495-p, 37, 46, 48-V, 362.classifier.<message-class>, 333, 423.classifier.class, 422.classifier.rule_id, 422.classifier.system, 333, 423.classifier_class, 422.SDATA.meta, 328.SDATA.SDID.SDNAME, 388.USER, 402<action>, 426, 442, 444<message>, 426, 442, 444@ANYSTRING@, 432@distance, 425@DOUBLE@, 432@ESTRING@, 432@FLOAT@, 432@IPv4, 432@IPv6@, 432@IPvANY@, 432@module, 80, 268@module snmp, 268@NUMBER@, 432@QSTRING@, 433@STRING@, 433@version, 80, 81

Aactions, 426

conditional actions, 427external actions, 428message correlation, 429

562syslog-ng.com

add-prefix(), 29addition, 397add_prefix(), 29AF_UNIX, 158, 159AIX

installing syslog-ng, 45redirecting errorlog to syslog-ng, 47

alert, 331alerting, 426allow-compress(), 357, 366allow_compress(), 366AMPM, 381, 383AND, 325anonymization, 396, 400api(), xviAPPLICATION_NAME, 383APP_NAME, 383archive-query(), 126archive_query(), 126ArcSight Common Event Format, 392artificial ignorance

message classification, 431assume-utf8, xv, 95, 103, 114, 119, 136, 142, 159auth-algorithm(), 271auth-password(), 271auth-username(), 271authentication, 350, 351auth_algorithm(), 271auth_password(), 271auth_username(), 271

Bbad-hostname(), 334bad_hostname(), 334balabit.com, 549bcc(), 261block, 82block arguments, 83

dynamical, 83body(), 259boolean, 26boolean operators, 325BSDDATE, R_BSDDATE, S_BSDDATE, 383BSDTAG, 382, 383

Cca-dir(), 357, 359ca-dir-layout(), 357

caps, 493catchall, 314, 315ca_dir(), 357ca_dir_layout(), 357cc(), 261CEF, 392CentOS

installing syslog-ng, 45cert-file(), 357cert-subject(), xiv, 357certificates, 350certified packages, 6cert_file(), 357cert_subject(), 357cfgfile, 493chain-hostnames(), 16, 72, 334, 384, 385chain_hostnames(), 334channel(), 88CHARSETALIASDIR, 39, 43check-hostname(), 336check_hostname(), 336chroot, 494chroots, 462chunk-size(), 220chunk-time(), 220chunk_size(), 220chunk_time(), 220cipher(), 219cipher-suite(), 358cipher_suite(), 358Cisco sequence number, 389Cisco SNMP trap, 269Cisco timestamp, 389cisco-snmp(), 270CISCO-SYSLOG-MIB, 270cisco_snmp(), 270class-path, 201classifying messages

concepts of, 418configuration, 421creating databases, 433filtering, 422pattern matching concepts, 420

client mode, 10client-host, 73client-hostname-from-the-message, 334client-hostname-resolved-on-the-relay, 334client-hostname-resolved-on-the-server, 334

563syslog-ng.com

client-side failover, 237, 292, 324client_lib_dir(), 172, 183, 202, 209clogMessageGenerated, 269, 270cluster(), 171, 172, 173, 182, 183collection(), 227columns(), 126, 135, 274, 281, 288Common Name, 351, 352, 355community(), 271, 274comparing values, 326compress(), 220compression

on-the-wire, 357, 366concurrent-requests, 171, 176, 182, 187concurrent-requests(), 173, 184concurrent_requests(), 173, 184condition, 427condition(), 402, 403conditional rewrites, 402, 403configuration file

default configuration, 68, 69detecting changes, 79including other files, 80

configuration snippets, 82block arguments, 83dynamical block arguments, 83

connect-query(), 126connect_query(), 126context, 429, 443context of messages, 424context-id, 424, 437, 438context-scope, 424, 429, 430, 437, 438, 444context-timeout, 424, 425, 430, 443CONTEXT_ID, 382Coordinated Universal Time, 14core files, 456correlating messages, 424create-dirs(), 190, 192, 215, 220, 221, 336, 337create_dirs(), 192, 220, 336creating SDATA fields, 401crit, 331crl-dir(), 359crl_dir(), 359CSV parsers, 409csv-parser(), 12, 407, 409CSV-values, 407curve-list(), xiv, 360curve_list(), 360Custom macros, 383

custom-domain(), 337custom_domain(), 337

Ddata types, 26database(), 123, 127, 228, 274, 276, 281DATE, 349, 381, 382DATE, R_DATE, S_DATE, 384date-column(), 123, 127datetime, 26datetime-column(), 123, 127datetime_column(), 127date_column(), 127DAY, 381, 382DAY, R_DAY, S_DAY, 384daylight saving changes, 13db-parser(), 421, 426debug, 331, 333, 494default-facility(), 85, 88, 93, 94, 128, 154default-level(), 88, 154default-modules, 494default-priority(), 85, 88, 93, 95, 128default_facility(), 88, 94, 128, 154default_level(), 88, 154default_priority(), 95, 128deleting syslog-ng PE, 61delimiters(), 410delimiters(<delimiter_characters>), 410destination, 76, 82, 498destination drivers, 12, 167

cisco-snmp() driver, 270database driver, 274, 280elasticsearch , 168, 172elasticsearch2 , 179, 182file() driver, 190, 191hdfs , 198, 201java() driver, 168, 172, 179, 182, 198, 201, 207, 209kafka , 207, 209list of, 168, 501loading the snmp module, 268logstore() driver, 215, 219mongodb() driver, 225, 227network() driver, 235pipe() driver, 246program() driver, 251, 252smtp() driver, 259, 260snmp() driver, 267, 271sql() driver, 274, 280

564syslog-ng.com

syslog() driver, 288, 289tcp() driver, 300tcp6() driver, 300udp() driver, 300udp6() driver, 300unix-dgram() driver, 301, 302unix-stream() driver, 301, 302usertty() driver, 309

destinations, 8, 12, 167, 497defining, 167FreeTDS configuration, 62Microsoft SQL Server configuration, 62MSSQL configuration, 62sql() configuration, 275, 276, 277, 286

DH parameter file, 360dhparam, 360dhparam-file(), xiv, 360dhparam_file(), 360Diffie-Hellman parameter file, 360digest(), 220dir(), xiv, 174, 185, 202, 210, 229, 236, 253, 262, 282, 290, 303dir-group(), 192, 221, 337dir-owner(), 192, 221, 337dir-perm(), 192, 221, 337dir_group(), 192, 221, 337dir_owner(), 192, 221, 337dir_perm(), 192, 221, 337discarding messages, 333disk buffer, 174, 184, 185, 202, 209, 210, 228, 229, 235, 236, 252, 253, 261, 262, 281, 282, 289, 290, 302, 303, 320disk queue (see disk buffer)

disk buffer, 318disk-based buffering, 174, 184, 202, 209, 228, 235, 252, 261, 281, 289, 302, 320disk-buf-size(), 176, 187, 204, 212, 230, 237, 255, 264, 283, 292, 304, 319, 320, 371, 373disk-buffer(), xiv, 174, 184, 185, 202, 209, 210, 228, 229, 232, 235, 236, 252, 253, 261, 262, 281, 282, 289, 290, 302, 303disk_buffer(), 174, 184, 185, 202, 209, 210, 228, 229, 235, 236, 252, 253, 261, 262, 281, 282, 289, 290, 302, 303division, 397dns-cache(), 338, 385, 386dns-cache-expire(), 338dns-cache-expire-failed(), 338

dns-cache-hosts(), 338, 462dns-cache-size(), 338dns_cache(), 338dns_cache_expire(), 338dns_cache_expire_failed(), 338dns_cache_hosts(), 338dns_cache_size(), 338dont-create-tables, 284dont-store-legacy-msghdr, 95, 103, 114, 119, 136, 142, 159door(), 135, 136dot-nv-pairs, 30download

pattern databases, 423drop-invalid, 410drop-message, 177, 188, 205, 212, 231, 344drop-property, 177, 188, 206, 212, 231, 344, 393dropped, 446, 447dropping messages, 333dynamical block arguments, 83

Eecho, 392elasticsearch, 168, 169, 170, 171, 172

performance, 173, 176elasticsearch(), 181, 320, 322, 323elasticsearch2, 167, 179, 180, 181, 182, 500

performance, 184, 187elasticsearch2(), 170, 320, 322, 323embedded log statements, 312emerg, 331empty-lines, 95, 103, 114, 119, 137, 142, 159enable-core, 494Enable-FIPS, 362enc-algorithm(), 272enc-password(), 272encoding(), 25, 95, 103, 141, 159, 393encrypt-certificate(), 221encrypting log files, 215encrypting log messages, 350, 351

on the hard disk, 215encrypt_certificate(), 221enc_algorithm(), 272enc_password(), 272engine-id(), 267, 272, 274engine_id(), 272environmental variables, 78eq, 326

565syslog-ng.com

err, 331error, 388error solving, 454escape-backslash, 411escape-double-char, 411escape-none, 411escaping special characters, 404event-api(), xvi, 88eventlog sources, 87eventlog(), 11, 87, 88event_api(), 88EVT API

limitations on Windows Vista or newer, 89exclude(), 25, 27, 28expect-hostname, 95, 104, 114, 119, 137, 142, 160explicit-commits, 284extended timestamp format, 389extract-prefix(), 416extract_prefix(), 416

Ffacilities, 20, 22, 330, 460FACILITY, 382, 384facility, 460facility(), 329FACILITY_NUM, 382, 384fail-over, 18, 237, 292, 324fail-over servers, 237, 292, 324failover

in mongodb, 226failover servers, 237, 292, 324failover-servers(), 237, 292FailoverSyslogServer, 237, 292, 324failover_servers(), 237, 292failure script, 457fallback, 314, 315, 410fallback-to-string, 177, 188, 206, 212, 231, 344fast-follow-mode(), 128fast_follow_mode(), 128fd limit, 191, 219fd-limit, 494feature releases, 14Federal Information Processing Standards, 362fetch-query(), 128, 130, 131fetch_query(), 128file, 100, 193, 421, 450file descriptors, 191, 219file encryption, 215

file informationfile@18372.4, 94

file(), 11, 76, 92, 94, 97, 113, 190, 191, 195, 219, 241, 248, 249, 257, 295, 307, 342, 343, 498file-template(), 338file@18372.4, 94FILE_FACILITY, 384FILE_LEVEL, 384FILE_MESSAGE, 384FILE_MSG, 384FILE_NAME, 384file_template(), 338filter, 76, 82, 498filter functions

list of, 329, 502filter(), 330, 402filtering

.classifier_class, 422on message class, 422

filtering rewrites, 402, 403filters, 8, 12, 324, 406, 460, 497

AND, OR, NOT, 325blacklisting, 330boolean operators, 325comparing values, 326control characters, 328defining, 325facilities, , 329facility and priority (level) ranges, 331in-list(), 330priorities, 331reference, 328tags, 328whitelisting, 330wildcards, 327

final, 9, 314, 315, 333FIPS, 362FIPS-mode='enabled', 362flag(syslog-protocol), 31flags, 311, 315

assume-utf8, 95, 103, 114, 119, 136, 141, 159empty-lines, 95, 103, 114, 119, 136, 141, 159expect-hostnames, 95, 103, 114, 119, 136, 141, 159kernel, 95, 103, 114, 119, 136, 141, 159no-hostname, 95, 103, 114, 119, 136, 141, 159no-multi-line, 95, 103, 114, 119, 136, 141, 159no-parse, 95, 103, 114, 119, 136, 141, 159no_multi_line, 238, 246, 255, 292, 304

566syslog-ng.com

store-legacy-msghdr, 95, 103, 114, 119, 136, 141, 159syslog-protocol, 238, 246, 255, 292, 304validate-utf8, 95, 103, 114, 119, 136, 141, 159

flags(), 76, 95, 103, 114, 119, 136, 141, 159, 221, 238, 246, 255, 292, 304, 311, 312, 399, 410, 498

for SQL destinations, 283flags(no-multi-line), 96, 99, 104, 109, 115, 120, 137, 142, 147, 160, 163, 383, 387flags(no-parse), 24, 96, 104, 115, 120, 137, 142, 160, 383, 387flow-control, 311, 312, 315, 316, 318, 319

example, 320hard, 318multiple destinations, 318soft, 318

flush-limit(), 173, 184flush-lines(), 91, 107, 116, 121, 130, 138, 145, 162, 193, 239, 247, 255, 284, 293, 305, 339, 346, 365, 375, 450, 461flush-size(), 364, 456flush-timeout(), 193, 239, 247, 255, 284, 293, 305, 339, 450flush_limit, 171, 176, 181, 182, 187flush_lines, 456flush_lines(), 193, 239, 247, 255, 284, 293, 305, 339flush_timeout(), 193, 239, 247, 255, 284, 293, 305, 339follow-freq(), 76, 92, 94, 96, 115, 128, 129, 138, 498follow_freq(), 96, 115, 129, 138force-directory-polling(), xiv, 34, 92, 96, 97foreground, 494format-cef-extension, xiv, 392, 393format-cef-extension(), 392format-json, 26, 29, 393, 412, 415format-json(), 25, 214format-welf, 133format-welf(), 25, 133, 394formatting messages, 378formatting multi-line messages, 99, 109, 147, 163frac-digits(), 176, 177, 187, 193, 204, 212, 222, 230, 231, 239, 247, 256, 264, 284, 285, 293, 305, 339, 348, 386frac_digits(), 176, 187, 193, 204, 212, 222, 230, 239, 247, 256, 264, 284, 293, 305, 339from(), 259, 264, 266fsync(), 194FULLDATE, 381, 382

FULLDATE, R_FULLDATE, S_FULLDATE, 384FULLHOST, 382, 384, 402FULLHOST_FROM, 382, 385

Gge, 326generating alerts, 426glob, 405glob patterns, 405global, 404, 405global objects, 12global options, 334

reference, 334global variables, 78greedy, 409, 410, 411greedy(), 432grep, 395group, 494group(), 160, 194, 222, 247, 248, 339groupset(), 401, 402grsecurity, 37gt, 326

Hhard macros, 24, 382hdfs, 198, 199, 200, 201hdfs(), 5, 80, 320, 322, 323hdfs_file(), 201hdfs_max_filename_length, 205hdfs_uri(), 201HEADER, 19, 21header(), 259, 264help, 494HOST, 106, 144, 332, 340, 382, 385, 402host, 276host(), 123, 129, 130, 259, 267, 272, 285, 327, 330, 406host-override(), xiv, 91, 105, 143, 154, 161host-template(), 130HOST_FROM, 382, 386host_override(), 91, 105, 143, 154, 161host_template(), 130HOUR, 179, 190, 197, 207, 215, 225, 233, 245, 251, 259, 266, 273, 288, 299, 309, 347, 349, 381, 382HOUR, R_HOUR, S_HOUR, 385HOUR12, 381HOUR12, R_HOUR12, S_HOUR12, 385

567syslog-ng.com

Iif, 396ignore-case, 404, 405in-list, 330in-list filter, 330including filename in messages, 94indent-multi-line, 99, 109, 147, 163, 397indenting multi-line messages, 99, 109, 147, 163index(), 172, 182indexes, 285indexes(), 285info, 331inherit-properties, 426, 429, 442, 444inject-mode(), 426inlist(), xviinstallation path, 36installing syslog-ng, 36, 37

from a transformed PKG package, 51from DEB package, 48from PKG package, 49from PKG package in silent mode, 50from RPM package, 45in silent mode, 45on AIX, 45on CentOS, 45on clients and relays, 38on log servers, 41, 51on Microsoft Windows, 51on Red Hat Enterprise Server, 45on SUSE Linux Enterprise Server, 45prerequisites, 36using Puppet, 58

installing syslog-ng on Windows, 51without user interaction, 57

int, 26int32, 26int64, 27internal, 426internal(), 79, 90, 91, 169, 177, 180, 188, 198, 205, 206, 207, 212, 231, 344, 426, 443ip() or localip(), 105, 143ip-protocol(), 105, 143ip-tos(), 105, 143, 239, 294ip-ttl(), 105, 144, 239, 294ipv4-to-int, 397IPv6

filtering, 332ip_protocol(), 105, 143

ip_tos(), 105, 143, 239, 294ip_ttl(), 105, 144, 239, 294ISODATE, 381, 382, 383ISODATE, R_ISODATE, S_ISODATE, 386

Jjava, 450java(), 168, 172, 179, 182, 198, 201, 207, 209JavaScript Object Notation, 393journal files, 217journal-block-count(), 222journal-block-size(), 217, 218, 220, 223, 342journal_block_count(), 222journal_block_size(), 223JSON, 393JSON parsers, 414json-parser(), 414

Kkafka, 207, 208, 209, 213kafka(), 320, 322, 323kafka-bootstrap-servers, 213kafka-bootstrap-servers(), 209keep-alive, 106, 144, 240, 294, 306keep-alive(), 106, 144, 161, 240, 252, 294, 306keep-hostname(), 72, 73, 106, 112, 113, 130, 144, 150, 151, 154, 156, 334, 338, 340, 343, 348, 384, 385, 386keep-timestamp(), 13, 97, 106, 107, 115, 116, 120, 138, 145, 161, 340, 381keep_alive(), 106, 144, 161, 240, 294, 306keep_hostname(), 106, 144, 154, 340keep_timestamp(), 97, 106, 115, 120, 138, 145, 161, 340kern, 93, 384kernel, 95, 104, 114, 119, 137, 142, 160key(), 25, 27, 28, 29, 225key-file(), 360key=value pairs, 412key_file(), 360klogd, 94kmsg, 93ksymoops, 94kv-parser, 413kv-parser(), 412

Lle, 326

568syslog-ng.com

LEGACY_MSGHDR, 382LEVEL, 382, 388level(), 331level() or priority(), 331LEVEL_NUM, 382, 386lgstool, 217license, 11, 15license.txt, 41line-breaks

on Windows, 191literal, 26loading modules, 268local time, 20, 23local-time-zone(), 194, 231, 285localip(), 101, 240, 294localport(), 240, 294local_time_zone(), 194, 231, 285log, 76, 82, 498log messages, representation, 24log messages, structure, 18

BSD-syslog protocol, 19IETF-syslog protocol, 21legacy-syslog protocol, 19RFC 3164, 19RFC 5424, 21

log paths, 8, 311, 497defining, 311flags, 311, 315flow-control, 316, 319, 320

log pipesembedded log statements, 312

log statements, 12embedded, 312log paths, 8, 497

log statistics, 446on unix-socket, 446

log transfer protocolreliable, 364, 365

log-disk-fifo-size(), 175, 185, 203, 210, 229, 236, 253, 262, 282, 290, 303, 321, 323log-fetch-limit(), 97, 107, 116, 120, 126, 130, 135, 138, 145, 155, 161, 316, 319, 320, 449log-fifo-size(), 175, 177, 186, 188, 194, 203, 205, 211, 213, 222, 229, 230, 231, 236, 237, 240, 248, 254, 256, 263, 265, 272, 282, 283, 286, 291, 294, 303, 304, 306, 316, 318, 319, 320, 321, 323, 341

log-iw-size(), xiv, 91, 97, 107, 116, 121, 130, 138, 145, 161, 162, 193, 239, 247, 255, 284, 293, 305, 316, 319, 320, 365, 369, 449log-msg-size(), 19, 25, 98, 107, 116, 121, 123, 139, 145, 152, 162, 244, 276, 277, 298, 341log-prefix() (DEPRECATED), 98, 116, 121, 139, 162logcat, 217logchksign, 79logging procedure, 8logrotate, 191, 219logstore, 215, 450logstore(), 11, 34, 215, 216, 219logstore-journal-shmem-threshold(), 217, 218, 222, 341, 342logstore_journal_shmem_threshold(), 341log_fetch_limit(), 97, 107, 116, 120, 130, 138, 145, 155, 161log_fifo_size(), 177, 188, 194, 205, 213, 222, 231, 240, 248, 256, 265, 272, 286, 294, 306, 341log_iw_size(), 91, 107, 116, 121, 130, 138, 145, 161, 373log_msg_size(), 98, 107, 116, 121, 139, 145, 162, 341, 364log_prefix() (DEPRECATED), 98, 116, 121, 139, 162Long Term Supported releases, 14losing messages, 454lt, 326LTS releases, 14

Mmacros, 12, 378

date-related, 381default value, 380hard, 24hard and soft macros, 382in filenames, 380patterndb tags, 390read-only, 24reference, 383rewritable, 24SDATA, 388soft, 24

Mandatory Access Control, 37MapR, 200MapR File System, 200MapR-FS, 200MARK, 194, 195, 196, 240, 241, 242, 248, 249, 256, 257, 295, 296, 306, 307, 308, 342, 343

569syslog-ng.com

mark(), 194, 240, 248, 256, 295, 306, 342mark-freq, 430mark-freq(), 194, 240, 248, 256, 294, 295, 306, 342mark-mode(), 194, 195, 196, 240, 241, 242, 248, 249, 256, 257, 295, 296, 306, 307, 308, 342, 343marker(), 416mark_freq(), 194, 240, 248, 256, 294, 306, 342mark_mode(), 195, 241, 248, 256, 295, 307, 342match, 325match(), 325, 327, 331, 406max-connections(), 91, 107, 116, 121, 130, 138, 145, 146, 158, 162, 316, 317, 319, 320, 450max-field-size(), 152, 155max-uid-query(), xiv, 130maximal message size, 341max_connections(), 107, 146, 162max_field_size(), 155max_uid_query(), 130mem-buf-length(), 371mem-buf-size(), 319, 320memory use of

logstore journal files, 218message

classification, 421, 422, 433context, 424correlation, 424facilities, 20, 22, 330ID, 389parsing, 421, 422statistics, 446templates, 378triggers, 426

MESSAGE, 382message counters, 446message encoding, 25message filtering

using parsers, 422message loss, 454message parsing, 407message statistics, 446message(), 331message-acknowledgement-timeout(), 365, 366message-template(), 131message_acknowledgement_timeout(), 366message_template(), 131Microsoft SQL

sql() configuration, 277Microsoft SQL Server configuration, 62

Microsoft Windowslimitations, 5

MIN, 381, 382MIN, R_MIN, S_MIN, 387mod-java, 80modes of operation, 10

client mode, 10relay mode, 11server mode, 11

modifying SDATA, 401module-registry, 494modules, 79, 80modulus, 397mongodb (see type-casting)

failover, 226replicasets, 226

mongodb(), 6, 11, 25, 26, 27, 225, 226, 227, 232, 233, 320, 322, 323MONTH, 381, 382MONTH, R_MONTH, S_MONTH, 387MONTHNAME, R_MONTHNAME,S_MONTHNAME, 387MONTH_ABBREV, 381, 382MONTH_ABBREV, R_MONTH_ABBREV,S_MONTH_ABBREV, 387MONTH_NAME, 381, 382MONTH_NAME, R_MONTH_NAME,S_MONTH_NAME, 387MONTH_WEEK, 382MONTH_WEEK, R_MONTH_WEEK,S_MONTH_WEEK, 387MSEC, 381MSEC, R_MSEC, S_MSEC, 387MSG, 19, 21, 331, 382MSG or MESSAGE, 387MSGHDR, 331, 387MSGID, 382, 387MSGONLY, 382, 388mssql, 134, 277, 288MSSQL

sql() configuration, 277multi-line messages, 98, 99, 108, 109, 146, 147, 162, 163multi-line-garbage(), 98, 99, 100, 108, 109, 146, 147, 162, 163, 164multi-line-prefix(), 98, 99, 100, 108, 109, 146, 147, 162, 163, 164multiline messages (see multi-line messages)

570syslog-ng.com

multiplication, 397multithreading in syslog-ng PE, 449multi_line_garbage(), 98, 108, 146, 162multi_line_prefix(), 98, 108, 146, 163mutual authentication, 350, 353myhost, 402MYSQL_UNIX_PORT, 130, 279, 285

Nname, 444name resolution, 460, 461

local, 461ne, 326netmask(), 330, 332netmask6(), 332network(), xv, 31, 96, 101, 103, 104, 105, 115, 119, 137, 142, 143, 156, 157, 160, 195, 234, 235, 241, 248, 249, 257, 295, 300, 307, 317, 320, 322, 323, 324, 342, 343, 348, 350, 351, 352, 354, 356, 364, 366, 367, 552n e t w o r k ( t r a n s p o r t ( r l t p ( t l s - r e q u i r e d ( y e s ) )flag(syslog-protocol)), 32network(transport(rltp(tls-required(yes))), 32, 34network(transport(rltp)), 32, 33network(transport(rltp)flag(syslog-protocol)), 32network(transport(tcp) flag(syslog-protocol)), 32network(transport(tcp)), 31, 32, 33network(transport(tls) flag(syslog-protocol)), 32network(transport(tls)), 32, 33network(transport(udp) flag(syslog-protocol)), 32network(transport(udp)), 31, 32no-caps, 494no-hostname, 95, 104, 114, 119, 137, 142, 160no-multi-line, 96, 104, 115, 119, 137, 142, 160, 192, 238, 246, 255, 293, 305no-parse, 16, 96, 104, 115, 120, 137, 142, 160nobody, 402normalize-hostnames(), xiv, 91, 343, 385, 386normalize_hostnames(), 91, 343NOT, 325not available on Windows, 113, 114, 118, 119, 123, 125, 136, 158, 159, 225, 227, 246, 251, 252, 274, 280, 301, 302notice, 331Novell Ready, 6NULL, 279null(), 279, 286number of open files, 191, 219numeric operations, 397

nv-pairs, 30

Ooff, 362on, 362on-error, 393on-error(), 177, 188, 206, 213, 232, 343on-the-wire compression, 357, 366on_error(), 343optimizing regular expressions, 406optimizing syslog-ng performance, 460

regular expressions, 406optional(), 117, 121, 139, 164options, 12

reference, 334OR, 325Oracle

sql() configuration, 276ORACLE_BASE, 276ORACLE_HOME, 276ORACLE_SID, 276OSUPTIME, 388other, 448output buffer, 316, 319output queue, 318, 321overflow queue (see output buffer)

output buffer, 318overriding facility, 85overriding-original-program-name, 427, 442, 444overwrite-if-older(), 196overwrite_if_older(), 196owner(), 164, 196, 223, 249, 344

Ppad-size(), 100, 110, 117, 121, 122, 139, 148, 164, 165, 196, 249pad_size(), 100, 110, 117, 121, 139, 148, 164, 196, 249pair(), 26, 27, 28Parameters, xiiparameters

disk-buffer(), 174, 184, 202, 209, 228, 235, 252, 261, 281, 289, 302log-disk-fifo-size(), 320log-fetch-limit() , 316, 319log-fifo-size() , 316, 319log-iw-size() , 316, 319max-connections() , 316, 319

571syslog-ng.com

parser, 76, 82, 498parsers, 8, 12, 407, 421, 422, 497

kv-parser, 412parsing messages, 407, 421, 422, 431

concepts of, 407filtering parsed messages, 422

password(), 131, 232, 286path(), 232path.home, 171, 172, 173, 182, 183pattern database, 421, 422, 433

concepts of, 418correlating messages, 424creating parsers, 431pattern matching precedence, 420schema, 433structure of, 419using the results, 422

pattern matchingprocedure of, 420

patterndbdownload, 423

pcre, 405peer-verify(), 360, 367peer_verify(), 360performance

optimizing multithreading, 451using multithreading, 449

perm(), 165, 196, 197, 223, 250, 344persist-file, 494persist_only, 112, 150, 348, 462pid, 286PID, 382, 388pid file, 37, 46, 48pidfile, 495pipe, 114, 450pipe(), 6, 11, 113, 114, 117, 121, 139, 164, 195, 241, 246, 248, 249, 257, 295, 307, 342, 343plugins (see modules)poll(), 96, 115, 129, 138, 346port(), 131, 157, 171, 172, 173, 182, 183, 259, 272, 287, 301port() or destport(), 242, 296port() or localport(), 110, 148posix, 404PostgreSQL

sql() configuration, 275prefix(), 89, 131, 133, 155, 412, 413, 414, 416preprocess-into, 495

prerequisites, 36preventing message loss

flow-control, 316, 319PRI, 19, 21, 382, 388PRIORITY, 382, 388process-mode, 495processed, 446, 447processing multi-line messages, 98, 108, 146, 162, 163PROCESS_ID, 388PROGRAM, 132, 382, 388program(), 6, 118, 119, 122, 195, 241, 248, 249, 251, 252, 257, 295, 307, 320, 322, 323, 327, 332, 342, 343, 406, 428program-override(), xiv, 91, 98, 100, 110, 116, 117, 121, 122, 139, 148, 162, 165program-template(), 132program_override(), 91, 100, 110, 117, 122, 139, 148, 165program_template(), 132properties-file, 209proto-template, 348proto-template(), 344proto_template(), 344pseudonymization, 396, 400Puppet

installing and managing syslog-ng PE, 58

Qqdisk-dir, 495quot-size(), 321quote-pairs(), 407, 409, 411quote_pairs(), 411

RRCPTID, 382, 388read-old-records(), 89, 100, 132read-old-records(no), 130read-old-records(yes), 130read-only macros, 24reading messages

from database tables, 123from external applications, 118

read_old_records(), 89, 100, 132receiving Cisco syslog messages, 269recursive, 100recv-time-zone(), 13, 14, 345recv_time_zone(), 345Red Hat Enterprise Server

572syslog-ng.com

installing syslog-ng, 45RedHat Ready, 6regular expressions, 324, 403, 406, 460

case-insensitive, 404escaping, 404pcre, 405posix, 327

rekey(), 29relay destinations, 11relay mode, 11

destinations, 11relay-hostname-resolved-on-the-server, 334releases, 14reliable(), 174, 175, 176, 185, 186, 187, 202, 203, 204, 210, 211, 212, 228, 229, 230, 235, 236, 237, 253, 254, 255, 262, 263, 264, 281, 282, 283, 290, 291, 292, 302, 303, 304, 321, 322removing syslog-ng PE, 61replace-prefix(), 29replace_prefix(), 29replacing message text, 398reply-to(), 265resource(), 171, 172, 182, 183response-timeout(), 366response_timeout(), 366retries, 178, 189, 206, 214, 232, 287retries(), 171, 176, 181, 187, 200, 214, 267retry_sql_inserts, 287reusing snippets, 82rewritable macros, 24rewrite, 76, 82, 498rewrite if, 402, 403rewrite rules, 8, 12, 398, 497rewrite(), 399rewriting

IP addresses, 396, 400rewriting messages, 398

concepts of, 398conditional rewrites, 402, 403

rfc3164, 30rfc5424, 30rltp, 96, 104, 115, 119, 137, 142, 160RLTP , 364, 365root, 82, 83rotating log files, 191, 219R_HOUR, 349R_UNIXTIME, 14

Ssafe-mode(), 227, 232safe_mode(), 232scaling to multiple CPUs, 449scl

system() , 151scope(), 25, 27, 28, 29, 30SDATA, 382, 388

file@18372.4, 94SEC, 382SEC, R_SEC, S_SEC, 389secondary messages, 426secondary servers, 237, 292, 324sedding messages, 398segmenting messages, 407, 409, 412, 414selected-macros, 30SELinux, 37, 47send-time-zone(), 13sender(), 264SEQNUM, 382, 389sequence ID, 389sequence number, 389

Cisco, 389serialized, 221server license, 15server mode, 11, 168, 179, 198, 207, 225, 227, 274, 280server(), 171, 172, 173, 182, 183, 226server-hostname, 334servers(), 226, 227, 232session-statements(), 287session_statements(), 287set(), 400setting facility, 85setting message fields, 400, 401

setting multiple fields, 401shift(), 29signing log files, 215silently-drop-message, 177, 188, 206, 212, 231, 344silently-drop-property, 177, 188, 206, 213, 231, 344silently-fallback-to-string, 177, 188, 206, 213, 231, 344skipping messages, 333smtp(), 6, 259, 260, 320, 322, 323snmp, 267SNMP traps, 267snmp(), 6, 80, 267, 268, 270, 271, 274, 320, 322, 323snmp-obj(), 267, 273, 274

573syslog-ng.com

snmp_obj(), 273so-broadcast(), 110, 148, 242, 296, 306so-keepalive(), 110, 148, 165, 242, 296, 306so-rcvbuf(), 111, 112, 141, 149, 150, 165, 242, 296, 308, 455, 460so-sndbuf(), 111, 149, 242, 297, 308SOCK_DGRAM, 85, 86, 87, 158, 168, 301, 302, 500, 501SOCK_STREAM, 85, 86, 87, 158, 168, 301, 302, 500, 501soft macros, 24, 382source, 76, 82, 498SOURCE, 382, 390source drivers, 12, 85

eventlog() driver, 87, 88file() driver, 92, 94internal() driver, 90, 91list of, 87, 500network() driver, 103pipe() driver, 113, 114program() driver, 118reference, 85sql() driver, 123, 125sun-streams() driver, 135, 136syslog() driver, 140, 141system() driver, 151systemd-journal() driver, 152systemd-syslog() driver, 156tcp() driver, 156tcp6() driver, 156udp() driver, 156udp6() driver, 156unix-dgram() driver, 159unix-stream() driver, 159

source(), 314, 332SOURCEIP, 382, 390sources, 8, 12, 85

defining, 85eventlog, 87on different platforms, 86

SO_BROADCAST, 110, 148, 242, 296, 306so_broadcast(), 110, 148, 242, 296, 306so_keepalive(), 110, 148, 165, 242, 296, 306so_rcvbuf(), 111, 149, 165, 242, 296, 308so_sndbuf(), 111, 149, 242, 297, 308splitting messages, 407, 409, 412, 414spoof source

on Microsoft Windows, 243, 297

spoof-interface(), 243, 244, 297, 298spoof-source, 244, 298spoof-source(), 73, 243, 298spoofing

on Microsoft Windows, 243, 297spoof_interface(), 243, 297spoof_source(), 243, 298sql, 450sql destinations, 274SQL NULL values, 286sql sources, 123

customized queries, 134default queries, 129limitations of, 123variables, 135

sql(), 5, 6, 11, 36, 123, 124, 125, 168, 274, 275, 276, 280, 283, 284, 320, 322, 323, 324, 380, 501stable releases, 14STAMP, 348, 382stamp, 448STAMP, R_STAMP, S_STAMP, 390statistics, 446stats-freq(), 345stats-level(), 345, 446, 448, 508stats_freq(), 345stats_level(), 345stderr, 495stdin, 251, 252store-matches, 405stored, 447strace, 457STREAMS, 86, 87, 135, 500string, 27, 405string comparison, 326strip-whitespace, 411STRUCTURED-DATA, 21STRUCTURED_DATA, 388subject(), 259, 266subject_alt_name, 351, 352, 355subkeys(), 27, 31subst(), 399subtraction, 397sun-streams(), 6, 135, 136supported architectures, 4supported operating systems, 4supported sql destinations, 279supported sql sources, 124suppress(), 197, 244, 250, 257, 298, 308, 448

574syslog-ng.com

suppressed, 448SUSE Linux Enterprise Server

installing syslog-ng, 45sync() or sync-freq() (DEPRECATED), 346sync() or sync_freq() (DEPRECATED), 346sync_send, 214syntax-only, 495syslog, 96, 104, 105, 115, 120, 137, 142, 143, 160, 192, 238, 247, 255, 289, 293, 305, 450syslog(), 31, 70, 96, 102, 104, 105, 115, 119, 137, 140, 141, 142, 143, 160, 195, 241, 248, 249, 257, 288, 289, 295, 307, 320, 322, 323, 324, 342, 343, 348, 350, 351, 352, 354, 356, 364, 366, 552syslog(transport(rltp(tls-required(yes))), 32, 33syslog(transport(rltp)), 31, 33syslog(transport(tcp)), 31, 33syslog(transport(tls)), 31, 33syslog(transport(udp)), 31syslog-ng

troubleshooting, 454syslog-ng binaries

compact, 36location of, 36

syslog-ng clientsconfiguring, 68

syslog-ng PE certifications, 6syslog-ng relays

configuring, 72syslog-ng servers

configuring, 70syslog-ng-relay, 73syslog-ng-server, 73syslog-ng.conf, 75

environmental variables, 78fingerprint, 79global variables, 78includes, 80

syslog-protocol, 96, 104, 115, 120, 137, 142, 160, 192, 234, 238, 247, 255, 293, 305syslogd, 85, 86, 135, 158, 197, 206, 224, 244, 250, 258, 298, 308syslog_ng, 58system(), 151systemd-journal(), 152, 154systemd-syslog(), 156SYSUPTIME, 390S_DATE, 349S_HOUR, 349

S_UNIXTIME, 14

Ttable, 274table(), 123, 132, 135, 274, 287table-init-query(), 132table_init_query(), 132TAG, 382, 390tagging messages, 328tags, 328

as macro, 390TAGS, 382, 390tags(), 24, 91, 101, 111, 117, 122, 132, 140, 149, 165, 328, 333, 422, 423tcp, 105, 143, 288, 450tcp failover, 237, 292, 324tcp(), xv, 31, 156, 157, 300, 320, 322, 323tcp6(), xv, 156, 157, 300, 320, 322, 323template, 76, 498template functions, 391

embedding, 396numeric operations, 397

template(), 131, 133, 197, 206, 223, 244, 250, 258, 298, 308, 411, 414, 416template-escape(), 197, 244, 250, 258, 298, 308, 379templates, 12, 378, 379

defining, 379, 380escaping, 379example, 380literal $, 379template functions, 391

test_log, 123threaded, 104, 143, 193, 451threaded(), 346, 449threading, 449throttle, 455throttle(), 178, 189, 206, 214, 224, 245, 250, 258, 266, 299, 309time-column(), 123time-column(col_name, [format]), 133time-reap(), 191, 216, 217, 220, 346time-reopen(), 133, 171, 181, 200, 238, 267, 278, 292, 346time-sleep(), xvitime-sleep() (DEPRECATED), 346time-zone(), 13, 101, 111, 118, 122, 134, 140, 149, 155, 166, 179, 190, 197, 206, 215, 225, 233, 245, 251, 258, 266, 273, 287, 299, 309, 347, 381

575syslog-ng.com

timestamp, 14, 20, 23, 460timestamp-freq(), 224, 347timestamp-policy(), 224, 347timestamp-url(), 224, 347timestamping

Microsoft Authenticode Timestamping, 224, 347OID, 224, 347policies, 224, 347RFC3161, 224, 347URL, 347

Timestamping Authority, 215timestamp_freq(), 224, 347timestamp_policy(), 224, 347timestamp_url(), 224, 347timezone

in chroots, 463timezones, 13, 14time_column(col_name, [format]), 133time_reap(), 346time_reopen(), 133, 346time_sleep() (DEPRECATED), 346time_zone(), 101, 111, 118, 122, 134, 140, 149, 155, 166, 179, 190, 197, 206, 215, 225, 233, 245, 251, 258, 266, 273, 287, 299, 309, 347TLS, 103, 140, 141, 350

configuring, 351, 353reference, 356

tls, 105, 143, 288tls(), 112, 150, 245, 299, 353, 354, 356, 365, 367, 368tls-required(), 367, 368tls_required(), 367to(), 259, 267Tomcat logs, 99, 109, 147, 163topic(), 209transport

TCP, 364, 365transport layer security

TLS, 350transport(), 111, 150, 245, 299transport(rltp(tls-required(yes)), 31transport(rltp), 31transport(tls), 31, 356trap-obj(), 267, 273, 274trap_obj(), 273trigger, 443triggered messages, 426triggers, 426troubleshooting, 454

core files, 456failure script, 457strace, 457syslog-ng, 456, 457truss, 457tusc, 457

truncating messages, 25truss, 457trusted-dn(), 361trusted-keys(), 361trusted_dn(), 361trusted_keys(), 361ts-format(), 21, 23, 179, 190, 198, 207, 215, 225, 245, 251, 259, 300, 309, 348, 390TSA, 215ts_format(), 179, 190, 198, 207, 215, 225, 245, 251, 259, 300, 309, 348tusc, 457type(), 123, 134, 172, 182, 274, 288, 327, 399, 403, 404type-casting, 26, 177, 188, 205, 212, 231, 343type-hinting, 26typecasting (see type-casting)TZ, 382TZ, R_TZ, S_TZ, 391TZOFFSET, 382TZOFFSET, R_TZOFFSET, S_TZOFFSET, 391

Uudp, 105, 143, 288, 320, 322, 323, 450udp(), xv, 31, 156, 157, 300udp6(), xv, 156, 157, 300uid-column(), 123, 134, 135uid_column(), 134ulimit, 191, 219unicode, 405uninstalling syslog-ng PE, 61UNIQID, 391unix-dgram, 85, 117, 121, 139, 163, 164, 455unix-dgram(), 6, 96, 104, 115, 119, 137, 142, 158, 159, 160, 161, 162, 195, 241, 248, 249, 257, 295, 301, 302, 307, 320, 322, 323, 342, 343unix-stream, 85, 117, 121, 139, 164, 455unix-stream(), 6, 77, 158, 159, 195, 241, 248, 249, 257, 295, 301, 302, 307, 320, 322, 323, 342, 343, 498UNIXTIME, 382UNIXTIME, R_UNIXTIME, S_UNIXTIME, 391unknown, 422

576syslog-ng.com

unsupported features on Windows, 5use-dns(), 73, 106, 112, 144, 150, 154, 340, 348, 385, 386, 462use-fqdn(), xiv, 92, 112, 150, 155, 348, 385, 386use-rcptid(), xv, 349use-syslogng-pid(), 92, 101, 113, 118, 123, 134, 140, 151, 166use-time-recvd() (DEPRECATED), 349use-uniqid(), 174, 184, 349USEC, 381USEC, R_USEC, S_USEC, 391user, 495username(), 134, 233, 288usertty(), 309, 460use_dns(), 106, 112, 144, 150, 154, 340, 348use_fqdn(), 92, 112, 150, 155, 348use_rcptid(), 349use_syslogng_pid(), 92, 101, 113, 118, 123, 134, 140, 151, 166use_time_recvd() (DEPRECATED), 349use_uniqid(), 349UTC, 14utf8, 405

Vvalidate-utf8, 95, 96, 103, 104, 114, 115, 119, 120, 136, 137, 142, 143, 159, 160, 393value, 274, 444value comparison, 326value(), 325, 331value-pairs, 25value-pairs(), 27, 177, 188, 206, 213, 225, 232, 233, 343, 392, 393, 394value-separator(), 414values(), 274, 288, 401value_pairs(), 27, 233varchar, 277variables

in sql queries, 135verbose, 495version, 495version(), 274

Wwarning, 331WebTrends Enhanced Log file Format, 133, 394WEEK, 382WEEK, R_WEEK, S_WEEK, 391

WEEKDAY, R_WEEKDAY, S_WEEKDAY, 391WEEK_ABBREV, R_WEEK_ABBREV,S_WEEK_ABBREV, 391WEEK_DAY, 382WEEK_DAY, R_WEEK_DAY, S_WEEK_DAY, 391WEEK_DAY_ABBREV, 382WEEK_DAY_NAME, 382WEEK_DAY_NAME, R_WEEK_DAY_NAME,S_WEEK_DAY_NAME, 391WELF, 133, 394wildcards

in file sources, 92Windows

installing syslog-ng server on, 51silent installation, 57

worker-threads, 495

YYEAR, 382YEAR, R_YEAR, S_YEAR, 391YEAR_DAY, 382

577syslog-ng.com