TIBCO FTL Monitoring · 2017. 3. 7. · Prometheus is an open-source systems monitoring and...

TIBCO FTL®

MonitoringSoftware Release 5.2March 2017

Two-Second Advantage®

Important Information

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCHEMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (ORPROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THEEMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANYOTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS ANDCONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTEDSOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THECLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOADOR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE)OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USERLICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THESOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, ANDYOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BEBOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright lawsand treaties. No part of this document may be reproduced in any form without the writtenauthorization of TIBCO Software Inc.

TIBCO, Two-Second Advantage, TIB, Information Bus, eFTL and Rendezvous are either registeredtrademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

Enterprise Java Beans (EJB), Java Platform Enterprise Edition (Java EE), Java 2 Platform EnterpriseEdition (J2EE), and all Java-based trademarks and logos are trademarks or registered trademarks ofOracle Corporation in the U.S. and other countries.

All other product and company names and marks mentioned in this document are the property of theirrespective owners and are mentioned for identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOTALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASEDAT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWAREVERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSOR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICALERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESECHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCOSOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S)AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY ORINDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE,INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

Copyright © 2010–2017 TIBCO Software Inc. ALL RIGHTS RESERVED.

TIBCO Software Inc. Confidential Information

2

TIBCO FTL® Monitoring

Contents

About this Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

TIBCO Documentation and Support Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Client Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Realm Server Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Data Flow of Monitoring Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

The Monitoring Endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Subscribing to the Monitoring Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Structure of Monitoring Data Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Fields of Monitoring Data Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Catalogs of Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Catalog of Application Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Catalog of Persistence Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

Catalog of eFTL Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Catalog of Realm Server and Service Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Prometheus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Adapter (tibpromgateway) Command Line Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Docker Files for Prometheus and Grafana . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Prometheus Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Security of Monitoring Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Monitoring Data: Timing and Interruptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Estimating the Volume of Metric Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3


About this Product

TIBCO® is proud to announce the latest release of TIBCO FTL® software.

This release is the latest in a long history of TIBCO products that leverage the power of InformationBus® technology to enable truly event-driven IT environments. To find out more about how TIBCO FTLsoftware and other TIBCO products are powered by TIB® technology, please visit us at www.tibco.com.

4


HTTP://WWW.TIBCO.COM/

TIBCO Documentation and Support Services

Documentation for this and other TIBCO products is available on the TIBCO Documentation site. Thissite is updated more frequently than any documentation that might be included with the product. Toensure that you are accessing the latest available help topics, visit:

https://docs.tibco.com

Product-Specific Documentation

Documentation for TIBCO products is not bundled with the software. Instead, it is available on theTIBCO Documentation site. To access the documentation web page for this product from a localsoftware installation, open the following file:

TIBCO_HOME/release_notes/TIB_ftl_5.2.0_docinfo.html

TIBCO_HOME is the top-level directory in which TIBCO products are installed.

● On Windows platforms, the default TIBCO_HOME is C:\tibco.

● On UNIX platforms, the default TIBCO_HOME is /opt/tibco.

The following documents for this product can be found on the TIBCO Documentation site.

TIBCO FTL® Documentation Set

● TIBCO FTL Concepts Everyone reads this booklet first, for an intuitive introduction to thefundamental concepts of FTL software.

● TIBCO FTL Development Application developers and architects read this manual to understandconcepts relevant in any supported programming language.

● TIBCO FTL API Reference Application developers use this HTML documentation to learn the detailsof the FTL API in specific programming languages.

● TIBCO FTL Administration Administrators read this manual to learn how to use the realm serverand its interfaces, and how to define a realm. Developers can also benefit from understanding FTLsoftware from an administrator’s perspective.

● TIBCO FTL Monitoring Administrators read this manual to learn about monitoring and metrics.Developers read this manual to learn how an application can subscribe to the stream of monitoringdata.

● TIBCO FTL Installation Read this manual before installing or uninstalling the product.

● TIBCO FTL Glossary The glossary contains brief definitions of key terms used in all other parts ofthe documentation set.

● TIBCO FTL Release Notes Read the release notes for a list of new and changed features. Thisdocument also contains lists of known issues and closed issues for this release.

TIBCO eFTL™ Documentation Set

TIBCO eFTL software is documented separately. Administrators use the FTL realm server GUI toconfigure and monitor the eFTL service. For information about these GUI pages, see the documentationset for TIBCO eFTL software.

How to Contact TIBCO Support

For comments or problems with this manual or the software it addresses, contact TIBCO Support:

● For an overview of TIBCO Support, and information about getting started with TIBCO Support,visit this site:

http://www.tibco.com/services/support

5


https://docs.tibco.com

http://www.tibco.com/services/support

● If you already have a valid maintenance or support contract, visit this site:

https://support.tibco.com

Entry to this site requires a user name and password. If you do not have a user name, you canrequest one.

How to Join TIBCOmmunity

TIBCOmmunity is an online destination for TIBCO customers, partners, and resident experts. It is aplace to share and access the collective experience of the TIBCO community. TIBCOmmunity offersforums, blogs, and access to a variety of resources. To register, go to the following web address:

https://www.tibcommunity.com

6


https://support.tibco.com

https://www.tibcommunity.com

Client Monitoring

The TIBCO FTL client library monitors each client process, sampling several values that revealoperating behavior and performance. The client library sends these monitoring samples to the realmserver, which publishes them, and displays the most recent values for administrators.

Scope

The TIBCO FTL library gathers monitoring data from the following processes:

● Application program processes

● TIBCO FTL component processes, such as bridge processes, persistence store servers, and TIBCOeFTL servers

● The realm server process

Sampling

Throughout the realm, every client process samples data at a regular interval. The realm propertyClient Monitoring Sample Interval specifies that interval. Zero is a special value, instructing allclients to disable the monitoring feature.

Each client sends all its pending sample data to the realm server in the next client heartbeat message.The first batch of data becomes available with the first heartbeat after the first sample interval. If theheartbeat interval is large, then the realm server receives a batch of accumulated samples in eachheartbeat).

Realm Server MonitoringThe TIBCO FTL realm server also monitors its own operation, sampling several values that measureoperating behavior and performance. The realm server publishes these metrics, and displays the mostrecent values for administrators.

7


Data Flow of Monitoring Metrics

The following diagram illustrates the flow of FTL monitoring data.

Explanation of the Diagram

● Metric data begins in client processes. The TIBCO FTL client library monitors the metrics relevant toeach client, and includes monitoring data values in the heartbeat data that it sends to the TIBCOFTL realm server.

The realm server receives the monitoring data from its client processes.

● When the realm server receives monitoring data from any of its clients, the realm serverimmediately publishes that data in an FTL message. The stream of those messages is called themonitoring stream.

Any FTL application can subscribe to the built-in monitoring endpoint to receive the monitoringstream.

● You can develop custom monitoring applications that subscribe to the monitoring stream and usethe metrics data.

● TIBCO FTL software includes an executable application, tibpromgateway, that subscribes to themonitoring stream and makes the monitoring data available to Prometheus.

Prometheus is an open-source systems monitoring and alerting toolkit. You can access the datadirectly from Prometheus, or using other tools that manipulate and display the data, for example,Grafana.

For more information about this data pathway, see Prometheus, and the topics that follow it.

● The realm server retains only the most recent value of each metric from each client.

These values are available from the realm server through its web API and browser GUI.

Key to Shape and Colors in the Diagram

RectangleExecutable program process

EllipseFTL application

8


Small Circle (inside another shape)Monitoring endpoint

Rounded RectangleAPI, GUI, or HTTP request

BlueComponent of TIBCO FTL software

RedYour FTL application

GreenThird-party tool

Change in Functionality

In releases prior to 5.0, the realm server retained a set of historical monitoring data. Starting in Release5.0, the realm server immediately publishes the data and discards all but the most recent sample in eachdata series. External tools can store and process the data.

This arrangement reduces memory usage in the realm server, and makes the data available to analyticaltools in a more timely and useful manner.

9


The Monitoring Endpoint

The stream of monitoring metric data is available to any client program through the monitoringendpoint.

The realm server publishes a stream of messages that contain monitoring metrics data.

Application programs can subscribe to this stream on the monitoring endpoint. The name of thisendpoint is available as a constant value in the TIBCO FTL API library.

Subscribing to the Monitoring StreamTo receive monitoring data messages, application programs can subscribe to the built-in monitoringendpoint. An application can use a content matcher to narrow the message stream.

Procedure

1. Optional. Compose a content matcher.

● To receive the entire stream, omit any content matcher.

● To receive a sub-stream, construct an appropriate content matcher.

For example, an application could subscribe to all monitoring messages, or to a sub-streamsuch as the following:

● Only event messages.

● Only client metric messages with a specific client label.

● Only messages that report a specific client status.

● Only messages about a specific host computer.

See Fields of Monitoring Data Messages.

2. Create a subscriber object.a) Specify the API constant that denotes the monitoring endpoint.b) Optional. Specify the content matcher from step 1.

3. Ensure that the application program includes the other tasks required in any subscribing program.For example, the program must define appropriate callbacks, create an event queue, add thesubscriber to the event queue, and start a dispatch loop.See "Structuring Programs" in TIBCO FTL Development.

10


Structure of Monitoring Data Messages

The realm server publishes a stream of monitoring data messages. Each message can contain a group ofsamples, that is, data values. Each monitoring message also contains fields that identify and describethose samples.

Types of Monitoring Messages

A monitoring message can contain metric samples from a client, metric samples from the realm server,or data about an event. The msg_type field distinguishes among these three types of message.

Descriptive Fields

Many of the top level fields of a monitoring message contain descriptive information about the samplesor event. For example, they describe the application program, client process, and the sampling time.

This descriptive information applies to all of the data samples in the message. For example, all thesamples in the message come from the same client, and report data from the same sampling interval.

Samples

A monitoring message can contain data samples of multiple metrics. The metrics field contains anarray of data sample submessages.

Data sample submessages can include the following fields:

ContextThe object that this sample measures. For example, the context value could be the human-readablename of a specific endpoint.

Type of the metricThe type of counter, that is, the kind of measurement. For example, type 11 denotes the bytes sentmetric, which counts the number of bytes sent outbound through a transport. For descriptions of themetric types, see Catalogs of Metrics.

ValueThe numeric value of the counter at the end of the sampling interval.

IDThe unique internal identifier of the counter. This ID unambiguously identifies the specific counter,even though the context and type together are usually sufficient to describe the counter.

11


Fields of Monitoring Data MessagesThe format of monitoring data messages includes many fields that are common to all monitoringmessages, and some fields that are present only in certain types of monitoring messages. The followingtables describe the fields and their values.

Top Level Fields

Field Name Data Type Description

msg_type long The type of a monitoring message indicates the variableportion of its contents.

For a table of possible values, see Message Type Values.

Each monitoring message type number is defined as aconstant value in the TIBCO FTL API library.

client_label string When connecting to the realm server, a client programcan supply a label that denotes that particular client.Unlike a client ID, which denotes a specific clientprocess, this label remains unchanged even when theclient restarts as a new process.

You can use this label to identify clients by their rolewithin your enterprise.

client_status string This string denotes the status of the client.

For details, see the Status field in "Client StatusReference" in TIBCO FTL Administration.

metrics messagearray

Each message in this array contains one data sample ofone metric.

For a table of possible fields of these sample messages,see Data Sample Submessage.

This field contains a value only if the msg_type is clientmetric or server metric.

event_type long If the msg_type is event, this value indicates the type ofevent.

For a table of possible values, see Event Type Values.

Each event type number is defined as a constant valuein the TIBCO FTL API library.

timestamp long The sampling time or event time (in microseconds afterthe epoch) for the metrics in this message.

application string The application name of the client.

This field contains a value only if the msg_type is clientmetric.

12



application_instance string The application instance name of the client.


client_id string The client ID of the client process.

The realm server assigns each client process a uniqueclient ID.


server_id string The UUID of the realm server.

This field contains a value only if the msg_type is servermetric.

pid long The process ID of the client process, as assigned by theoperating system of the client's host computer.


ftl_user string The client supplied this value for the username propertyin its realm connect call.

effective_user string The user name that started the client process, accordingits host computer's operating system. The client processinherits permissions from this user.

version string The version of the TIBCO FTL client library that theclient or realm server process uses.

host string The host name of the computer where the client processis running.

ip string For a client process, this IP address denotes the networkinterface through which the client communicates withthe realm server.

For a realm server, this IP address denotes the primaryinterface of the host computer.

server_uptime long Time (in seconds) that the realm server process has beenrunning.

This field contains a value only if the msg_type is servermetric.

Message Type Values

These values occur in the msg_type field of monitoring messages.

Each monitoring message type number is defined as a constant value in the TIBCO FTL API library.

13


Value Description

1 Client Metric

The message contains client metrics in an array of data sample submessages.

2 Server Metric

The message contains realm server metrics in an array of data samplesubmessages.

3 Event

The message contains information about an event, such as a new clientconnection or a state change.

The value of event_type field denotes the event.

Data Sample Submessage

Each sample submessage contains all of these fields.


context string This human-readable string identifies the context of themetric within the client application. For example, if themetric measures some aspect of a specific endpoint,transport, or queue, then the context is the endpointname, transport name, or queue name.

type long This number identifies the type of metric.

For a table of possible metrics, see Catalog of ClientMetrics.

Each type number is defined as a constant value in theTIBCO FTL API library.

value long This field contains the data value of the metric.

id long This internal ID number uniquely identifies the contextof the metric.

Event Type Values

If the msg_type is event, these values can occur in the event_type field to indicate the type of event.

Value Description

1 Connected

A client process has connected to the realm server.

2 Disconnected

A client process has disconnected from the realm server.

14


Value Description

3 Status Change

The status of a client process has changed.

The client_status field indicates the new status.

15


Catalogs of Metrics

The catalogs that follow present the available monitoring metrics.

IDA unique ID denotes each type of counter. When subscribing to the monitoring stream, contentmatchers can refer to these ID values using API constants.

Metric TypeA human-readable string represents each type of counter. These strings appear in Prometheus userinterfaces.

DescriptionThis column explains the meaning of each type counter and the quantity it measures.

TrackingThis column describes the object that each counter tracks. For example, a counter could collect dataabout the operation of an endpoint, a transport, an event queue, a client process, or a persistence store.

Catalog of Application MetricsThese tables describe metrics that you can use to assess the operation of any client application or serviceprocess.

Standard Application Metrics

TIBCO FTL software gathers these metrics for processes, event queues, transports, and endpoints.

Unless otherwise specified, a metric reports a counter's value at the end of each sample interval, and thecounter resets to zero for the next interval.

ID Metric Type Description Tracking

1 Messages Sent Outbound data messages through an endpoint in theclient process.

Per endpoint

2 MessagesReceived

Inbound data messages through an endpoint in the clientprocess.

Per endpoint

12 Bytes Received Inbound data bytes on a transport in the client process. Per transport

11 Bytes Sent Outbound data bytes on a transport in the client process. Per transport

31 Data Lost Inbound data loss events on a transport in the clientprocess.

It is not possible to measure the magnitude of the eventsin bytes.

Per transport

32 FormatUnavailable

Messages with an unrecognized format carried on atransport in the client process.

Per transport

41 Queue Backlog Maximum number of messages in an event queue in theclient process during the sample interval.

Per queue

16



42 Queue Discards Inbound messages discarded by a queue in the clientprocess.

Per queue

51 Dynamic Formats Number of distinct dynamic formats registered withinthe client.

Creating a message with a dynamic format canincrement this counter. Receiving an inbound messagewith a dynamic format can increment this counter.

This counter can decrease as the client libraryautomatically unregisters unused formats.

Per client

67 Store MismatchMessages

Number of message flows that result from storemismatch.

Any non-zero value indicates a store mismatchmisconfiguration.

Store mismatch is a misconfiguration in which a directpath transport connects two endpoints that areassociated with two different persistence stores.

Per endpoint

70 Resident MemorySet Size

Current size in kilobytes of an application's workingmemory set.

Per client

71 Peak ResidentMemory Set Size

Maximum size in kilobytes, since the process started, ofan application's working memory set.

Per client

72 Virtual MemorySize

Current size in kilobytes of an application's virtualmemory set.

Per client

73 User CPU Time Total user CPU time in microseconds, that is, timeexecuting the process' object code.

Per client

Since theprocessstarted

74 System CPU Time Total system CPU time in microseconds, that is, timeexecuting operating system calls from the process.

Per client

Since theprocessstarted

Bytes sent and received need not correlate exactly with messages sent and received on correspondingendpoints. For example, filtering and optimizations can decouple these metrics.

Metrics for Multicast and RUDP Transports

TIBCO FTL software gathers additional metrics for each transport that uses UDP-based protocols, suchas multicast and RUDP transports.

For all metrics in the following table, tracking is per transport.

Unless otherwise specified, a metric reports a counter's value at the end of each sample interval. Thesemetrics are cumulative since the transport established its bus.

17


ID Metric Type Description

61 Packets Sent Outbound data packets on a multicast or RUDP transport.

This counter includes all outbound data packets, including original datapackets and retransmitted data packets.

62 Packets Received Inbound data packets on a multicast or RUDP transport.

This counter includes all inbound data packets, including original datapackets and retransmitted data packets.

63 PacketsRetransmitted

Outbound packets retransmitted on a multicast or RUDP transport.

If the transport retransmits a packet several times, it increments thiscounter for each retransmission.

64 Packets Missed Number of missed inbound packets on a multicast or RUDP transport.

65 Packets LostOutbound

Number of outbound packets discarded on a multicast or RUDPtransport.

66 Packets LostInbound

Number of inbound packets deemed lost on a multicast or RUDPtransport.

The transport accounts a packet as lost after its reliability constraintsexpire.

Catalog of Persistence MetricsThis table describes metrics that you can use to assess the operation of persistence stores.

Each persistence server is itself an application process, and tracks application metrics; see ClientMetrics.

The sampling interval for persistence metrics is fixed at 2 seconds. You cannot change this interval.

Store Metrics


1000 MessageCount

Number of messages in a store at the end of the sampleinterval.

Per store

1001 Message Size Storage (in bytes) that messages occupy in a store at theend of the sample interval.

Per store

2000 Durable Count Number of durables in a store at the end of the sampleinterval.

Per store

Catalog of eFTL MetricsThis table describes metrics that you can use to assess the operation of eFTL channels.

Each TIBCO eFTL server is itself a application process, and tracks application metrics: for example,metrics for the server's endpoints, and metrics for the transports that implement those endpoints. See Client Metrics.

The sampling interval for eFTL metrics is fixed at 2 seconds. You cannot change this interval.

18


eFTL Metrics


4000 ConnectionCount

Number of eFTL client connections on a channel at theend of the sample interval.

Per channel

4001 SubscriptionCount

Open subscriptions from eFTL clients on a channel atthe end of the sample interval.

Per channel

4002 Inbound eFTLMessage Count

Messages inbound from eFTL clients on a channel. Per channel

4003 Outbound eFTLMessage Count

Messages outbound to eFTL clients on a channel. Per channel

4010 DiscardedMessage Count

Data loss from publishers: the number of messages thatthe server discarded on a channel.

Discards can occur because a publisher sent a messagethat exceeded the maximum message size.

Per channel

4011 Dataloss MessageCount

Data loss to subscribers: the number of messages thatthe server discarded on a channel.

Discards can occur because a subscriber’s queueexceeded the maximum outbound backlog.

Per channel

4012 ConnectionPending Count

Number of new eFTL client connections on a channelthat are not yet fully initialized at the end of the sampleinterval.

Per channel

4013 ConnectionSuspended Count

Number of eFTL client connections disconnected by atemporary network outage that have not yetautomatically reconnected nor fully disconnected at theend of the sample interval.

Per channel

Catalog of Realm Server and Service MetricsThis table describes metrics that you can use to assess the operation of the realm server and FTLservices.

Realm Server Metrics

The sampling interval for realm server metrics is fixed at 2 seconds. You cannot change this interval.


5009 Client Connect Count Cumulative number of client connections since the realm serverprocess started.

5010 Client ReconnectCount

Number of clients that reconnected to the realm server during thesample interval.

A spike in this metric could indicate a network connectivity issuebetween the realm server and its clients.

19



5011 Send to InboxFailures

Number of undelivered protocol messages that the realm serversent to a client.

A spike in this metric usually indicates that several clients abruptlyexited.

Transport Bridge Metrics

Each bridge service is itself an application process, and tracks application metrics; see Client Metrics.

The sampling interval for bridge service metrics is fixed at 2 seconds. You cannot change this interval.


6000 Bridge Active 1 indicates that the bridge process actively forwardsmessages.

0 indicates that the bridge process is a backup instandby mode.

Per bridgeprocess

20


Prometheus

You can use a variety of other third-party tools, such as Prometheus and Grafana, to store, analyze, andvisualize monitoring data.

Prometheus is an open source toolkit for systems monitoring and alerting.

Grafana is an open source application for visualizing large-scale measurement data.

You can run Prometheus and Grafana processes in Docker containers. TIBCO FTL software includessample Docker files to configure and build Docker images for this purpose. Grafana dashboards in thesample are tailored to display FTL monitoring data. (See Docker Files for Prometheus and Grafana.)

Pushgateway and Adapter

Prometheus scrapes data from monitoring sources into its data repository.

Prometheus cannot receive data directly from FTL applications or the realm server. Instead,Prometheus provides an intermediate component, called Pushgateway, that receives data and caches itso that Prometheus can scrape it at a regular interval.

Pushgateway is not an FTL application, so it cannot subscribe to the monitoring message stream.TIBCO FTL software includes an adapter application, called tibpromgateway, that subscribes to themonitoring stream and posts the data to Prometheus Pushgateway.

To understand the context within which these processes cooperate, see the diagram in Data Flow ofMonitoring Metrics.

For technical details, see Monitoring Data: Timing and Interruptions.

Grafana Dashboard Example

You can use Grafana to display monitoring data stored in a Prometheus server. Grafana can displaydashboards, such as the following example, in a web browser.

21


Adapter (tibpromgateway) Command Line ReferenceTIBCO FTL Prometheus adapter (tibpromgateway) acts as an adapter between the realm server and aPrometheus Pushgateway process.

Location

The tibpromgateway executable file is located in the bin directory of the product installation tree.

22


Help

Parameter Arguments Description

--help

-h

Display a help message describing the commandline parameters and options.

Communications


--push-gateway

-g

IP:port Required.

The adapter application sends its data output tothe Prometheus Pushgateway at this location.

Supply an IP address and port number,separated by a colon character (:).

--realmserver

-rs

URL Required.

URL of the realm server. The adapter applicationreceives both its realm definition and monitoringdata from the realm server.

The adapter application contacts the realmserver at this location. Supply the location of theprimary realm server or a satellite server (see"Connect Port" in TIBCO FTL Administration).

--secondary-

realmserver

-s

URL Optional.

URL of the backup realm server.

If the regular realm server is unavailable, theadapter application contacts the backup realmserver at this location to receive its realmdefinition. Supply the location of the backuprealm server (see "Connect Port" in TIBCO FTLAdministration).

23


Data


--exclude-timestamp

-x

Optional.

When present, the adapter application removesthe client's sampling interval time stamp fromthe data samples before sending it to thePrometheus Pushgateway.

When the clocks of the clients' host computersare not approximately synchronized, thenPrometheus queries that refer to the time stampfield can yield unexpected results. Excluding thetime stamp causes the Prometheus Pushgatewayto supply a new time stamp value, representingthe time that the adapter posted the data sampleto the Pushgateway process. (This posting timeis necessarily later than the time the clientcollected the monitoring sample.)

--source-name name Optional.

Supply a human-readable string that denotes thesource of the monitoring data. The adapterprocess tags data with this string.

This name becomes the value in the Prometheusexported_job label. You can use this value inPrometheus queries to distinguish data fromdifferent sources, for example, primary andsatellite realm servers.

When absent, the default value is the URL of theream server, that is, the value of the--realmserver command line parameter.

--max-age

-a

age Optional.

When the adapter stops receiving monitoringdata about a client, the adapter waits for thistime (in seconds) before instructing thePushgateway to reclaim the memory that helddata about that client.

When absent, the default value is 1800 seconds(30 minutes).

24


JAAS Security


--password-file

-pf

path Optional. (Required for JAAS authentication.)

When present, the adapter application reads a username and password from the file at path, andauthenticates itself to the realm server using thosecredentials. For details, see "Password File" inTIBCO FTL Admistration.

TLS Security


--trust-file path Optional. (Required for TLS communication with asecure realm server.)

When present, the adapter application reads a trustfile from path, and uses that trust data incommunications with the secure realm server. See"Running a Secure Realm Server" in TIBCO FTLAdmistration.

--trust-everyone Optional.

The adapter application trusts any realm serverwithout verifying trust in the server's certificate.

Do not use this parameter except forconvenience in development and testing.It is not secure.

Tracing and Logging


--trace

-t

level Optional.

When present, the adapter application outputs tracemessages to stderr.

You may specify any of the standard log levelstrings (see “Tuning the Log Level” in TIBCO FTLDevelopment).

When absent, the default trace level is info.

25



--logfile logfile_prefix Optional.

When present, the adapter application logs to arolling set of log files instead of the console. Thelogfile_prefix argument may denote a path. All of thedirectories in the path must already exist.

For more information about rotating log files, see“Log Output Targets” in TIBCO FTL Development.

When absent, the adapter application sends logoutput to the console, ignoring the parameters --max-log-size and --max-logs.

--max-log-size size Optional.

Limits the maximum size, in bytes, of log files. Thevalue must be greater than 100 kilobytes (102400).The default value is 2 megabytes (2*1024*1024).

--max-logs Optional.

Limits the maximum number of rolling log files.The default is 50.

Docker Files for Prometheus and GrafanaYou can build sample Docker images to quickly start working with Prometheus and Grafana.

Disclaimer

For your convenience, TIBCO distributes sample Docker files that build images that includePrometheus and Grafana dashboards configured for TIBCO FTL monitoring data. However, the TIBCOFTL distribution does not include these third-party components, and TIBCO does not provide supportfor any of these third-party components.

Prerequisites for Building the Docker Files

● TIBCO FTL software must be installed using a Linux 64-bit installation package.

● Docker must already be installed on the host computer.

● The Docker build process must have access the Internet to download Prometheus and Grafana.

Location

The Docker files are located in the samples/docker directory of the product installation tree.

The file README.txt contains instructions for building and using the images.

HTTP Ports

When using the Docker images, you can connect to the browser interfaces using these default HTTPports.

26


Process Default HTTP Port

Prometheus Pushgateway 9091

Prometheus Server 9090

Grafana Server 3000

Grafana

To access the Grafana server, supply Grafana's default login credentials: user name admin, passwordadmin.

Prometheus LabelsPrometheus identifies each datum with a collection of labels, each of which has the form key=value.

The key and value of a label can originate at any stage of the monitoring data pipeline: client, realmserver, tibpromgateway adapter, or Pushgateway.

A label that originates in either a client or a realm server is a straightforward transformation of a field inthe monitoring message that carried the datum (see Fields of Monitoring Data Messages). However, notevery field of the monitoring message becomes a label.

The following table presents additional label keys, which originate either in the tibpromgatewayadapter or in the Pushgateway.

Key Description

exported_job The realm server that published the datum.

See the --source-name parameter in tibpromgateway Command LineReference.

job The intermediary source that scraped the datum.

The value is the job_name in the Prometheus scrape configuration.

instance The scraped target address.

The default value is the address of the Pushgateway, in the form host:port.

27


Security of Monitoring Data

TIBCO FTL components and clients can secure monitoring data using TLS.

If your enterprise uses secure realm servers, then monitoring data is secure as it travels from client torealm server, and then to subscribers.

However, TIBCO makes no assertions about the security of third-party components, such asPrometheus or Grafana.

Realm Server

A secure realm server uses a secure TCP transport to communicate with its clients. TLS secures theheartbeat messages that carry monitoring data from clients to the realm server.

A secure realm server can secure the monitoring message stream using TLS.

See "Secure Realm Servers" in TIBCO FTL Administration.

Adapter

The adapter (tibpromgateway) supports JAAS and TLS to secure its client connection to a secure realmserver. Administrators must supply appropriate command line arguments when starting the adapter.

Prometheus

The data paths that link the tibpromgateway adapter, Pushgateway, the Prometheus server, andGrafana are not secure.

28


Monitoring Data: Timing and Interruptions

Monitoring data flows through several segments on its path from a client to Prometheus. Timinginteractions along this path can affect the data. Some segments of this path retain data acrossinterruptions, while others do not retain data.

Best Practice for Intervals

Default values for the monitoring interval, heartbeat interval, and scraping interval are a good choicefor many applications. If your enterprise needs a different set of values, follow these guidelines for bestresults:

● Set the client monitoring sample interval and the client-server heartbeat interval to identical values.

● Set the Prometheus scrape interval to approximately half of the client-server heartbeat interval.

● Remember that the monitoring sample interval for persistence servers and TIBCO eFTL servers isfixed at 2 seconds. If it is critical to your enterprise that Prometheus capture all monitoring datafrom these processes, set the Prometheus scrape interval to 1 second. If your enterprise can acceptcoarser granularity, discarding some of the data samples from these processes, then use a largerscrape interval.

Client

Most monitoring data begins in a client process. Each client gathers data and transfers it to the realmserver in its regular heartbeat messages.

For the most complete results, ensure that the client monitoring sample interval is identical to theclient-server heartbeat interval. (You can configure these intervals as global realm properties of therealm server. However, the monitoring sample interval for persistence servers and TIBCO eFTL serversis fixed.)

If the heartbeat interval is longer than the sample interval, the client accumulates data from severalsample intervals, and sends the accumulated samples to the realm server in the next heartbeat.

If the client cannot communicate with the realm server, the client retains its monitoring data until itsuccessfully reconnects to the server, then it transfers the accumulated samples. Memory availabilitycould limit the amount of data that a client can accumulate.

When a client stops or restarts, it does not preserve monitoring data it has not yet sent to the realmserver.

Realm Server

The realm server receives data from clients, and also generates monitoring data that measures its ownoperations. The realm server publishes a stream of monitoring data messages on the built-inmonitoring endpoint.

As each monitoring sample set arrives from a client, the realm server immediately publishes it on themonitoring stream. The realm server also stores the new data samples, overwriting previous datasamples for that client.

The realm server publishes each sample in a separate message.

The monitoring message stream is available only to current subscribers. Subscribers cannot receive anymessages that the realm server sends while a subscribing process is not running, or while thecommunications link is interrupted. Monitoring data in those messages is no longer available to thesubscriber.

You cannot associate a persistence store with the monitoring endpoint.

29


Adapter

The adapter (tibpromgateway) subscribes to the monitoring message stream.

As each message arrives, the adapter immediately posts the data to the Prometheus Pushgateway andthen discards the message.

The adapter cannot receive any messages that the realm server sends while the adapter is not running,or while the communications link from the realm server is interrupted. Monitoring data in thosemessages is no longer available to the adapter.

If the adapter process stops, or if communications from the realm server are interrupted, then it cannotreceive any messages that the realm server sent in the interim. Monitoring data in those messages islost.

Pushgateway

The adapter transfers data to the Prometheus Pushgateway. The Pushgateway behaves as a single-record last value cache. Upon receiving a monitoring sample, it stores the inbound data for scraping.For each client and metric, Pushgateway retains only the most recent data sample.

Monitoring data is lost if the Pushgateway process stops or communications from the adapter areinterrupted.

Prometheus Server

Prometheus scrapes data from the Pushgateway at regular intervals, and stores it for analysis andnotification.

If the scrape interval is too long, the Pushgateway could receive data from several sampling intervalsbefore the next scrape. Each sample posted to Pushgateway overwrites the previous sample for thesame client and metric, so Prometheus scrapes only the newest sample for each metric, missing oldersamples.

For the most complete data sets, ensure that the Prometheus scrape interval is approximately half of therealm server's client-server heartbeat interval. In this way, Prometheus scrapes about twice as fast as themonitoring messages arrive, so it scrapes each sample at least once before the next sample overwrites it.Set the parameter scrape_interval in the Prometheus configuration file.

30


Estimating the Volume of Metric Data

You can estimate the size of client monitoring data to gauge its impact on network bandwidth andother resources. The number of counters depends on the number of application processes, and on thenumber of endpoints, transports, and event queues in each application process.

Procedure

1. For each application instance definition, determine the number of counters per client process of thatapplication instance. Combine information from the application instance definitions withinformation from the following table.

Client Monitoring Resource Usage

11 counters per client process

3 counters per endpoint

2 counters per event queue

3 counters per transport (other than multicast transports)

9 counters per multicast transport (6 specific to multicast, in addition to the 3 counters commonto all transports)

2. For each application instance definition, multiply the number of counters per client process (asdetermined in the previous step) by the expected number of application process instances.

3. Sum that number of counters over all the application instance definitions to obtain the enterprisetotal number of data samples generated in each sample interval.When estimating bandwidth requirements, remember that each data sample travels betweenmultiple components. Depending on the network architecture, the contents of a specific data samplemay traverse the network several times, possibly across the same network segments or links. See Data Flow of Monitoring Metrics.

4. Multiply the enterprise total by the number of samples you plan to retain in your monitoring datarepository (such as Prometheus).This product estimates the total volume of metric data.

31


Date post:	01-Jan-2021
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

TIBCO FTL Monitoring · 2017. 3. 7. · Prometheus is an open-source systems monitoring and...

Documents